Sucks, OP. Gotta own it, though. Can't let people intimidate you. Sometimes I feel a little weird when I'm the only black guy in the room, but I make my presence known (saying something, asking a question, etc.) just to let everyone know, "Yes, I'm here bitches."
But I'm a little more subtle in my approach. My brother is super direct; subtly is not in his vocabulary. I remember when we were in a clothing store a few years ago and my brother noticed that an employee was following us. Right before we left, he went right up to the guy and was like, "Are you following us because we're black?" I thought it was kinda funny because my brother caught him completely off guard and he was stammering. My brother was just like, "Uh huh. OK." And we left.
So, OP....be proud of who you are. People are going to stare and people are going to talk. But don't let them intimidate you. You're here, and they either can deal with it or fuck off.