Owning both a Vita and a 3DS i think i can comment on this with a good understanding of both consoles.
Note: i'm on my 4th 3DS, i had a launch model, sold it to buy the Zelda 3DS with Ocarina, i won a pikachu 3DS XL and i've recently sold both of those to buy the Zelda 3DS XL which is the best of the 4
The 3DS has good build quality, but has niggling issues like screen rub, the 3DSXL has slightly better build quality, but out of the two i've owned one felt flexible and not as sturdy as it could be, the Zelda one i have now is perfect and feels close to Vita build quality.
OS wise it does the job, you can navigate it using buttons so i don't know what the issue there is, unless you are refering to using a DS game in it, when its emulating DS mode....The issue with the OS is the account system (which seems to be finally coming) and the eshop which is revised frequently enough, and is better than it was at launch...oh wait months after launch even
The Vita on the other hand feels like i could use most of it as a brick, the only issue is the triggers feel far far too loose, allowing them to rattle a little, and obviously OLED bleed/black blobs (i forget the correct term)
OS wise.......god its ugly sorry but it is, the icons are too large and pointless, folders are stupidly small meaning you have to have multiple pages of icons, i think visually its much worse than the 3DS, it feels like the 3DS OS made for people with poor eyesight....
I hope they switch to a PS4 OS, or even something more tiled like
BUT it works great, the PSN store is better than eshop, and downloads work better in that you can game while they are in opperation, but they take so long most of the time it might as well be the same as the 3DS' sleep mode download, which isnt bad really.
Online infrastructure it might be better, it s a shame i have virtually no games that are worth playing online on it! Killzone might be the only one worth bothering and its a stable as Resident evil on 3DS was, so ive personally seen no difference.
BUT all that said, my vita is a brick, it sits there unused and unloved until i want to replay a PSone game or PSP game on it, or in a rare occurance (such as with tareaway and Killzone) an actual Vita game, where as i can't keep up with my 3DS' games, my backlog of games is ever increasing, so much so its meant i've yet to finish the Last of Us for my PS3 and loads on my WiiU...and then theres my steam library...
In a way my 3DS is the best and worst console, it takes so much of my time on its awesome games library that it stops me playing on other consoles, even other nintendo ones!
TL:DR Vita is better hardware, OS's both have their own faults, GAMES libraries are what counts and the 3DS's decimates not just handheld but console competition