It's not even that the Mario world is overused (that's a main reason, though), it's the fact that it has gotten to the point where you can predict what they're going to do with each system.
Bare Basic release amount for Mazza on any Ninty console (Combining all add-ons/Digital shop storefronts.):
-"Mainline" game (platformers)
-Puzzle game (Various. Including: Dr., Cookie, Picross, )
From Snes on, you can add in the following for consoles:
-RPG
-Kart Racer (Both of these start on GBA for portables.)
-Character spinoff (starts on GB for portables.)
From N64 on:
-Minigame compilation (Portable starts at GBA
-At least two sports titles (also for portables, starting from GBC)
From GCN on:
-Miscellaneous (DDR, Fortune Street)
From Wii on:
-Series crossover (MASATOG, Sports mix)
So therefore following that and using some fridge logic, these are the rest of the Mario titles you can expect from WiiU and 3DS
WiiU:
-At least two more sports (Let's just say Jai-alai and American Football)
-An RPG (Nothing original mind you. Probably Paper Mario)
-A Mario Party
-Puzzle game (Dr. Mario Operation gamepad)
-Specific character spinoffs (Non platformer DK game, Wario Platformer)
-Miscellaneous (Mario Visual Novel?)
3DS:
-Character Spinoffs (Wario Ware/Platformer, Non Platformer DK game, Non Platformer Yoshi game.)
Forget new IP's for a second, the most recent new genre that's been attempted with the Mario world was some AR photo apps. 2nd most recent new genre? (that's not a different sport. Different sport would be sports mix) Itadaki Street DS. Out in 2007.
It's not as if there are no new genres that can be attempted by Nintendo for Mario. Off the top of my head: Tower Defense, Simulation, DOTA, FPS, Endless Runner, WRPG, Visual Novel... With a lot of these, Ninty already has done these genres better with other IP's, but that never stopped this series before; heck we've had 4 Ninty IP's that have had pinball titles. 4!
For example: There was a lot of buzz about PvZ: GW when it was revealed at E3 from GAF. People were interested in a FPS that didn't bathe itself in tropes, ParentGAF was interested in a kid friendly-FPS (That was the impression I got, anyway.) People wanted a change.
Now imagine if there was a Mario Laser tag FPS with striker's art style. Would the reaction to PvZ: GW mean that people want something like this? No one will ever know, because Ninty plays it safe with the IP that's a safe bet.