You know, I grew up in downtown Toronto and there were all kinds of different people. We all had to follow some unwritten rules to get along and enjoy each others company. The rules were the rules and following them not only kept you out of trouble but earned you some good friends and fun times. It was simple and it was real. It was not even or symetrical and that's because human relationships are not like math and they are not a legal proceeding. Culture and history don't have an affect on math or the law but they do have a profound effect on human beings.
This concept of "well they say this in music so what's the big deal if I say that?" isn't grounded in reality. You are dealing with black people as a theoretical rather then face to face. You are coming up with some kind of fairness doctrine that in no way fosters a healthy relationship with your fellow man. The importance of this perceived fairness trumps actual human interaction and there is a simple reason why this happens: people who do this don't hang around with those other races who will be hurt, offended, and ostracized. They feel there is no victim in their usage of the word. Everybody likes to think of themselves as fair and logical, no one considers themselves a racist, and this is how people explain away uttering that word.