To answer your question: yes it does matter. The fact that they’re making games, that you and many others consider “as good or better”, despite hardware limitations isn’t a good thing. It’s like someone would paint a picture blindfolded. It might look OK, or even “as good”, but what can he do when he’s on level ground with other painters?
If visuals in video game doesn’t matter, than why didn’t they support WiiU, or Wii for that matter? Put BOTW on it, Mario Odessy, Metroid and everything else. And if they make you buy a new device, than why didn’t they put modern hardware in it? Why use 3y.o. mobile SoC? $300 is more than PS4 costs, than why do we see games like Bayonetta looking almost like on Xbox360? Does this “almost portable, but not really” gimmick worth it?
They didn't support the WiiU because it was a failed concept that no one wanted with truly horrible marketing. A console designed to catch up with the PS3 and 360 mere months after their next gen was about to be launched.
Because unlike the Switch, and arguably the Wii, it had weak hardware but gained nothing of value from it aside from an ugly tablet controller you couldn't even take outside of the room where the console was placed . A controller that was so useless that not even Nintendo knew how to implement it in most games.
They didn't put "modern hardware on it" because it would have limited the portability of the Switch. You can't make a PS4 (much less a pro) with the size and battery life of the Switch, specially not for $300.
It's not all just about the prize , it's about size, power consumption and the intended use for the device in the same way that an iPad pro is weaker than a PC you can build for a similar price (or at least could before GPU's skyrocketed in price) or the fact you can get a gtx1060 gaming laptop for the same price as a Surface Pro with Intel HD graphics.
And while we are talking about portability, I have no idea what you mean by "portable but not really". It is 100% portable. I take it to work
every day, I took it on a longer work trip late last year and I will be taking in on another one next month.
Not only does the portability allow me to take it with me but it will also allow Nintendo to focus on a single console for the first time since...the NES?. It's going to be the first time we have a Nintendo console that's going to have the main 3D Zelda and Mario game but also the next main Pokémon and Fire Emblem. And I find that way more exciting than a more powerful console, specially since I already have a non portable and power focused PS4 Pro under my TV.