It started to shift in the 90s, when kids were growing up with PS1 instead of N64, and it only got worse later.
Nope. The Game Boy became more popular with a younger audience worldwide and as an entry level games platform for that younger audience that generation thanks to Pokémon, and Game Boy Advance carried on that torch (it was the fastest selling system in Japan).
Meanwhile, while Game Boy was making up for N64's weaknesses in Europe and Japan the N64 wasn't a loss in North America. It cultivated the Xbox audience a generation before Microsoft entered the race, thanks to a host of great FPSes and solid support for other genres, like sports titles, from western third parties. And, er, Pokémon spinoffs. That's the reason why Nintendo managed to almost retain the SNES's install base numbers in North America.
And, well, there was then the DS and Wii, which sold through 250 million units combined across a single console generation, and a heap of software to go with it. Games like Mario Kart Wii sold 40 million units, and would have been an influential point of contact for many younger players.
The shift has definitely started this generation, not back with the PS1. 3DS has basically done a reverse N64, with sales holding up in Japan but falling off hugely elsewhere compared with the last generation. We're seeing the effects of the shift to smart devices right now, as Japanese publishers traditionally invested heavily in Nintendo handhelds are putting more of their eggs in the mobile basket. Just look at Level-5's new mobile-centric lineup.
The short story is: Handhelds have always been the most accessible way for Nintendo to reach the widest audiences, let alone younger audiences. And this generation has seen Nintendo's dominance of the handheld market - as an entry level device for portable games - diminish.