I'm in a similar boat, but I am a bit more optimistic. They had a transperson on the podcast a while back because they respected her work and were acquaintances (I forget her name, but I think she does a podcast with Jim Sterling?) and caught heat from some of their fanbase for it. Liam has infamously used the n-word, but from all the streams I've seen of his he's kept out the garbage. Matt I feel is more clueless than anything else. Woolie and Pat seem like all around good guys who tend towards self-depreciating comedy more than anything else, and although they oftentimes skirt the line in with some of their jokes, they're self-conscious enough to know when to back off. Based on the way they talk about their personal lives and beliefs on their podcast I'd be very surprised if any of them turned out to be JonTron levels of awful.
Now Pat's girlfriend Paige is another story, because checking her twitter feed she tends towards "lol triggered people/microaggresssions amirite?" and I couldn't find it but she has retweeted stuff from some questionable people. Didn't know there was something up with Plague, what'd he do?
I don't know anything about any Paiges (or Plague - my only exposure to him is the guest spots on the SBF podcast), but I think there are a lot of cynical Gen Xers who are just kind of sick of hearing words like "triggered". They get triggered by them. Sometimes people with awesome ethics can be Lisa Simpson, and sometimes they can be Homer. Both are obnoxious, and neither should be tossed onto the garbage pile with Bill Cosby, Trump supporters, gamergaters, Roman Polanski, or Amir0x.
I think aggressively policing casual racism and misogyny amongst public figures has merit, but people who roll their eyes at social justice buzzwords, or dismiss a few more outspoken advocates as "SJWs" now and again, aren't necessarily bad people. You can be irritated by vocal proponents of morality without disagreeing with them: Russell Brand is living proof of this.
I don't need to hear anything more about Earthworm Jim guy to know he's a scumbag after a single deliberate misgendering, but it'll take more than being told someone rolled their eyes at an "SJW" to convince me they're anything worse than a cynic.
I'm not remotely worried about the SBF. I doubt they'd pass any purity tests, but if any one of them is outed as racist, sexist, or homophobic (including Liam), I'll eat my Official Matt McMuscles copyright-skirting Punisher-inspired touque.