Nail the foundation of their unified hardware architecture and development tools. The more scalable it is, the more flexibility Nintendo will have when it comes to releasing all kinds of different hardware configurations and form factors, which would help their products reach more market segments. If it were up to me, the lineup of NintendOS gaming devices would consist of at least four pillars at all times; a low cost, entry-level handheld, a low-cost, entry-level console, a high-end handheld and a high-end console, all following a shorter update cycle than the usual 5-6 year hardware generation.
A modern account system and better online features. Pretty self-explanatory.
A truly unified software library. No separate versions, no download codes and not even cross-buy. Every game should be compatible with every NintendOS device that is capable of running it at the lowest settings, and a single purchase should give you access to the game on any compatible Nintendo device. This would not only free up Nintendo's development resources, but also encourage consumers to buy into and remain in the Nintendo ecosystem.
Focus on delivering an attractive product for everyone instead of aiming it specifically at the "hardcore" or "casuals". Contrary to popular belief, the PS1 didn't win the 32-bit generation because of a focus on "mature" content (the PS1 had more kiddy shovelware than the N64 had games), but because it was the first console that successfully marketed itself towards both younger and older audiences. Even the Wii had a pretty good balance between blue ocean software and more traditional stuff during its first two years on the market, which greatly contributed to its early success. Apple doesn't build most of its products with a specific demographic in mind - they simply try to make them desirable to as many people as possible while following an overarching vision and staying within a certain budget.
Ditch the heavy focus on child-friendly content and advertising. Seriously. Most kids over 8 years old want to play what the older siblings and/or adults they look up to are playing. You can probably count the Apple ads aimed at younger audiences on one hand, and yet iDevices and Macbooks are some of the most popular Christmas gifts for children. This does not mean Nintendo needs to shelve Mario and Pokemon or shy away from creating whimsical new IPs, but their first-party lineup needs a better balance between child-friendly content and games with a more mature presentation. Same goes for their advertising.
Beef up their western studios and invest in a handful of high-profile IPs with a mature presentation. This doesn't mean pumping out dozens of mature games every year at the expense of everything else, but they need one or two high-profile mature IPs for variety's sake. The N64 only really needed Goldeneye to stay relevant among an older crowd in NA, even with Sony's advertising machine firing on all cylinders. And for fuck's sake, stop trying to shoehorn cel-shading into Zelda when the market has clearly shown a preference towards a darker, more realistic artstyle.
Make sure the 3DS successor starts out strong and retains its momentum. A well designed and successful platform usually attracts third parties. With a unified software library, any support for a Nintendo handheld would automatically bolster the home console, which in turn makes it a more enticing purchase.
Aggressively court Japanese third parties. Japan shuns consoles in favor of handhelds. The West does the exact opposite. Being able to deploy a piece of software on both form factors in a single development cycle is a very attractive proposition for Japanese developers who want their games to reach the biggest audience possible without gambling on the volatile mobile market. Nintendo should try to use this opportunity to get most of the big name IPs and developers back on board.
Extend an olive branch to western third parties. They likely won't be back in full capacity, but making it easier to port their most popular games over to NintendOS devices wouldn't hurt anyone.