Well one, the people answering this question, how would a normal person watching the news or seeing them in action in person be able to tell who is a mutant and who isn't? To me everyone is a filthy mutant.
If we were to think about living in the Marvel universe, the difference between a Spiderman/FF and X-Men is that both all five of the former are much more recognizable than the X-Men. There have been so many mutants in and out of the team how could you remember them all? Further, there are so many 'mutants' running around being dickheads and fucking shit up, how do i know they aren't part of the X-Men vigilante group? That's how I'd think of it.
Cosmic Spiderman punched the Hulk into Orbit. That's pretty assholish.