I think it basically has a lot to do with the quality of the games.
People seem to want more characters in mainline Mario platformers because they don't associate any SMB side-characters (Wario, Daisy, Rosilina, Yoshi, etc.) with bad franchise ruining games like the Sonic Adventure Series, Sonic Heroes, or Shadow The Hedgehog.
The only Super Mario character that comes close to being as loathed as any non-Genesis STH character is Bowser Jr. because he's generally associated with Super Mario Sunshine (which, despite being critically acclaimed, is probably the most unpopular mainline Mario game among fans and the general gaming audience), and he also replaced the fan favorite Koopalings until late-2009.
Funny thing is...many casual and hardcore Mario fans have started to warm up to the little guy after the Galaxy series came about.