I've thought of a level progression system that I think would balance open and linear gameplay. It's not boring stretches of nothingness like a lot of people associate the term open world with, but you still have freedom.
There are no level select screens, just open levels (64/Sunshine) connected by linear levels (Galaxy/2D Mario). You start each world by beating a linear level, grab the star, hop off a cliff (or whatever level transition would be best) and land in an open level. Once you meet a requirement of the open level - whether it be collecting a certain amount of stars, beating a boss, triggering something in the environment, or simply finding a hidden area - different paths open up leading to more linear levels. Rinse, repeat, and add an area boss and you have the main level progression system. In addition to the basic system, collecting a certain amount of stars or activating something in the environment could cause a boss to appear in an area you've been to. Beating it would open another path branch. Hidden areas throughout the world contain more focused platforming challenges (2D and 3D) to get you extra stars and unlock more sections of the map.
"Hub" area is Toad Town where you can fast travel to any open level that you have completed in order to unlock other path branches. This area will also serve as a means of progressing the plot (which should still be nonexistent) in a similar way to TTYD and Super Paper Mario. Challenges return from NSMBU and are interconnected with the main world. Different characters in Toad Town will give you these challenges and the area you have to complete them in. Stars, secrets and bosses could also appear here.
Yes, I've thought way too much about this for how bad my ideas probably are. >_>