Are you implying that Mario Kart would have sold 10x better by being on a base 10x the size?
It just doesn't work like that. Check out sales of the Sonic series. Outside of the first two games and the mobile port, the games sell about 1-2 million copies each, regardless of whether they're on Sega hardware or not, regardless of being on one platform or three.
There are certain games that will sell hardware. Mario Kart and Smash are among them. Putting them on PS4 and XB1 probably wouldn't have a significantly increase in sales. It's not likely to increase software sales enough to make up for having to pay 30% of the revenue to Sony and Microsoft, plus make up for losing revenue on hardware as well. So yes, I would say it's probable, perhaps even likely, that Nintendo made more money on Mario Kart 8 as a Wii U exclusive than they would have if it were on PS4/XB1.
Downsizing a company is not a good move for a company as established and cash-rich as Nintendo. Shareholders should be wanting Nintendo to grow when they have the money to spend, not shrink.