A few months back I was watching TV with some friends and that Susan Boyle came on TV (the singer from the British talent show). Now my mate starts rambling on about how ugly she is, what a dog she looks like etc. I said to him 'why say it'? 'Why does it even need to be said for'?. You can see it, I can see it but why does it need to be said? I have a fundamental problem with this sort of thing not because there's any danger that somehow the singer on TV is going to hear it, of course not, but because it conflicts with my basic principles of showing respect and decency toward others. I guess I know first hand how these sorts of insults built up over time can come back and adversely affect the quality of life of the victim in a very sad way.
What may be a joke to one person, a momentary piece of amusement, can contribute greatly in causing low self confidence and low self esteem, depression, anxiety, phobia and mental and psychological stress in another. And this is wrong. I know those are simple words to you, but trust me, it isn't any fun to experience such things. It can lead to all sorts of issues of social integration in society.
I do think think there is no worse and more destructive insult to a hetrosexual man these days than to call him gay. It's a major issue when in comes to bullying at school and has been identified as such in studies on bullying; a time where adolescent males are trying to establish themselves in society only to have the core of their manhood publicly questioned through such slurs.
Here was one recent and sad example. I think it's sick and insidious myself, and this sort of crap has sadly become so endemic and accepted in modern society as to be seen as the norm.
So while to you it may be a silly throwaway comment, don't blindly assume that what may be a joke to you will always invariably be a joke to another. The OP made clear he doesn't like the assumption. I certainly doubt you've ever suffered issues in regards to low confidence, low esteem and low mood as a result of the acts of others. But honestly, have the respect and decency to try see how others may be adversely affected by the things you may see as a joke. I also don't mean to single you out either, this is a problem rife in society and I think it needs to be better appreciated what this sort of thing can actually do to someone over a long period of time.
Not that I expect any of this to fall on anything but deaf ears. Unless you've been there you have no idea.