I feel the same way as the OP.
I've had my 3DS for a couple of years and my Vita for about a year. The Vita made it very hard for me to go back to the 3DS; it made the 3DS seem clunky and outdated.
I'm aware that the 3DS supposedly has better games, but there's nothing on it which I've enjoyed as much as, say, Sine Mora or Super Stardust Delta.
It's probably just a matter of taste though. I think if the games on the 3DS 'get you' you won't notice the hardware and OS so much. The games didn't 'get me' so I find it a really outdated experience.