They made a plethora of fundamental design mistakes that needed far more time, effort and money than Sony was willing to spend.
In no particular order:
* initially trying to make the Vita into a micro-tablet:
-- they moneyhatted a bunch of apps, which, however, didn't get much support, because they weren't willing to keep bribing the devs into releasing updates (the devs themselves didn't see much value in the Vita versions of their apps);
-- they earmarked half the system RAM for tablet functions: at launch, only 256 MB of RAM were available for games; they only cleared another 128 MB circa 2015, not-coincidentally, when they finally disabled several of the original launch apps;
-- the useless 3G version: another piece of evidence that the Vita had initially been conceived as a micro-tablet; handhelds aren't released with 3G functionality, however tablets usually come in two flavors. Its inclusion, however, caused the device's max clocks to be dropped, because running both the SoC and the 3G modem at full power caused the system to overheat (micro-HDMI was disabled on the non-TV versions for the same reason);
* the bad control scheme: back touch was an awful idea; N3DS-style dual shoulder buttons should have been the way to go;
* the pointlessly overpriced components: back in 2011, the only mass-market mobile device using an OLED was the Galaxy S2; at the time mobile OLEDs were rare, expensive, and, frankly, not that great: they offered vibrance, but had a lot of other flaws (including short subpixel lifespans, notably for the blue one);
* the idiotically priced proprietary memory card: in a bid to both recycle their Memory Card Micro tech (grudgingly dropped from the new Xperia generation in favor fo MicroSDXC), and spread out the total price for their system, they made what was the most visible bungle of their strategy; the main problem wasn't necessarily that the Vita memory card was expensive, but rather that it was way more expensive than an equivalent microSDXC.