Well, actually Xmen is pretty well known, as most of them used multiple villains per movie. You also had a very popular animated series with a slew of villains.
Spiderman had the same benefit: multiple movies, and TV shows.
No one knew the Hellfire Club before FC, and no one gives a fuck about them now. Maybe Emma Frost MAYBE, but not Shaw, Mastermind, Pierce, or Leland. Apoc, Magneto, William Stryker (in name only and without the Purifiers), and very specific Brotherhood members (Sabretooth, Mystique, Blob, Juggernaut, Toad) are really all anyone knows. Cassandra Nova? Mr. Sinister and the Marauders? Omega Red? The Shi'ar (depending)? Phalanx? Bastion? Purifiers? And those are only some of the biggest. That's not bringing in Vulcan, Exodus, Emplate, Kavita Rao, Ord and the Breakworld, Danger, Dark Beast, Belasco, Stryfe, Shadow King, Selene, Genocide, and there's a few others that I know I'm missing.
Like, maybe they know some of them from the cartoon and maybe a couple stuck from the movies (again, Stryker or the Hellfire Club or Shadow King or Bolivar Trask), but by and large the X-Men villains gallery is not well known.