The vita, though? People aren't buying these things for tech.
The 3DS had major issues
Hardware reasons:
a) It was too expensive for its target market by far
b) The hardware had a confusing name that meant it wasn't immediately apparent that it was a 'new generation' of handhelds
c) A terrible screen right when cheap phones were starting to make poor resolution screens a thing of the past
d) An ugly clamshell design that made it look like a toy in comparison to other tech products, whereas the DS Lite was pretty damn slick for its time
e) Poor battery life for a gaming product
f) A design flaw that scratched up the top screen (happened to me, pretty damn poor show)
g) Poor multimedia functions - can't really watch movies, web browsing isn't great - these are really a fault of the screen, but still
h) Needed a 'not for young kids 5 and under due to 3D' warning - from a fucking Nintendo handheld!
i) Glasses free 3D was a gimmick no one cared about that added costs
Non-hardware reasons:
j) Marketing was poor
k) Launch software was poor
l) Limited app selection
As for the Vita, it was also too expensive and had little software support and no killer apps, that was its problem as much as anything. As for the 3DS, it DID end up getting a whole bunch of killer apps, which is why they managed to push 60m of a really mediocre machine.
Reasons a) through l) can be fixed by Nintendo next time around with their new hardware. It can only help.