Software is only as good as the hardware allows, and the PSP was severely compromised in this area whereas Vita expands the possibilities available on consoles.
You're right, that Vita will inevitably live or die by its software library, but part of enabling good software is the strength of the hardware that allows such experiences to be available to begin with. PSP was a bit doomed from the start in the west.
So from my perspective, Vita at least has a shot at gaining significant traction by selling software and not necessarily hardware. I'd argue that Vita may very well sell less than the original PSP, but it could go on to do much more business thanks to higher attach rate and a more energized and active userbase.