I don't think the word "thug" is nearly as tainted as the N word, for instance. I agree with you on that.
I would compare it instead to something like the name Marcus. Are there white guys named Marcus? Yes, there are. Even famous ones. But when I hear the name "Marcus," the image that forms in my head tends to be of a black guy. It's what I'd refer to as a soft association.
I'd say the same of thug. There are certainly white people who are lowlifes, and sometimes those people are referred to as "thugs," but I more often hear them referred to as "trash" or just "criminals." When I hear someone is a "thug," I tend to associate that with an image of a black man in my head. It's not always true -- again, a soft association.
And those soft associations are sometimes the most pernicious. The N word, by contrast, is completely unambiguous: yes, you are referring to a black person, and yes, you are trying to be hurtful. In some ways, that makes it much easier to deal with, because the intentions behind the word are so clear.