I would argue that the joke which inspired #CancelColbert was similarly about racist people and yet it sparked a controversy all the same.
My take is that when you go to a comedy show, especially comedians known for covering controversial topics, you are entering into an agreement of sorts. You agree to the assumption that what the comedian is saying shouldn't be taken at face value and to interpret it in a charitable way (e.g. as a clear joke). Even if it's a bad joke, is delivered poorly, or in some other way falls flat, it should still be taken as a joke. So, from the comedian's perspective, if you get upset at what is clearly a joke, you are breaking that agreement and the whole exercise of going to a comedy show is then futile.
And that's why I think comedian's get so riled up over the issue. Without that assumption by the audience, they literally can't perform their act. It threatens their livelihood.
If a joke is truly in poor taste, the crowd will let the comedian know by not laughing. A good comedian will either axe the joke from his/her routine or adjust it to make it funny. However, if almost the entire crowd is laughing but there is a Twitter controversy after, I think the comedian is in the right to ignore it as manufactured outrage. Because when it's on Twitter, the audience no longer paid to see a comedy show knowing full-well what they were getting into. Twitterers have not entered into an agreement to receive the statement in a charitable light. And stripped of its context, a joke will be much more likely seen as offensive.
So, I do err on the side of the comedian. They should be free to make the jokes they want to make. The crowd will decide if it's a good joke or not. And by "crowd" I mean the ones actually at the show--the people who paid specifically to have their taboos prodded and mocked--not the people on the Internet who never agreed to that sort of thing.