As long as longer gameplay sessions suck down a phone's battery quickly and brutally, there will be a need for a second gaming device.
Smartphones have gored other single purpose devices once they got good enough at that particular purpose (cameras, e-readers, PDAs, mp3 players, etc). The key for Nintendo to keep their handheld viable is to create good, exciting software only found on their system, at an affordable price ($30 should be the max) and good, quality, well priced hardware with a long battery life. (The 3DS was their first handheld to fail in this regard).
Mobile tech is increasing to the point that by the time the next Nintendo handheld comes out, their hope would be that they could release a handheld a little more powerful than Vita, charge $150 (thus maximizing profits), with a chipset efficient enough to get back to 10+ hours of battery life. A device that would be far superior to any smartphone for gaming.
The smartphone market won't grow forever. For example, is there any real reason (other than from gadget geeks) to upgrade from a Galaxy S4 to a Galaxy S5? These things are going to hit the wall very soon. Mobile gaming? How sustainable are these F2P whale hunting ventures? Why have we accepted that smart phone gaming is an ever increasing, unstoppable curve? (Sorry, tangent)
On the software side, put more effort into old school Mario and Zelda. Lower the barrier of entry for indies. However, allowing Android to run on the device would kill it - any devs at that point would just target Android.
Nintendo in the console market is a little more tricky. The biggest problem they have there is that the big Western third parties hate their guts and wouldn't support them even if their device was more powerful than PS4/Xbone - they'd just invent some new excuse for not supporting them. Nintendo built the Wii U stupidly thinking that companies like EA were on board (remember the "unprecedented partnership"?)
They have to go into their next console first with the idea of something that they can support by themselves if they have to. Profitability has to be their focus. "At the end of the day, even if we're third, did we make money?" Wii U did not have that focus with its expensive gamepad and other features the market decided were useless. Power is not nearly as important here as ease of development. For controllers? The N64 and Gamecube had complicated controllers and sold like garbage. The Wii had a simple controller and sold like crazy. The Wii U has an even more complicated controller and is selling worse than the Vita at times. The path is clear there - go back to the Wii's controller, improve the motion tracking, and keep things simple. You can sell the Wii U pad separately for your new box for BC.
Make sure their development teams are ready and able to crank out software for this machine. But the software again is the key. Look back over the last 3 generations at the games that actually SOLD and why. This means more Metroid Prime and less Metroid: Other M. More Link Between Worlds and less Skyward Sword. Separating 2D from 3D Mario and advancing both franchises instead of trying to fuse them into something nobody wants. Treating their best franchise (Wii Sports) as a top franchise instead of breaking it up and selling it over the e-Shop.
This will take time. They should ride out both the 3DS and Wii U for the next few years while they get all of this in place behind the scenes. Both platforms suffered from being a year or two too early. (Wii U would likely have been profitable if Nintendo had used their market leader position to wait a bit longer until AMD came knocking on their door like they allegedly did for Sony and Microsoft). Use the Wii U to get used to HD development, and the 3DS as a cash cow. Get back to profitability and build an ecosystem of indies and Japanese third parties that will support your box.