Short Answer
If you enjoy driving and/or want practicality, the Fit. If you want comfort, Yaris.
Long Answer
Honda Fit
-------------
The automotive press unanimously loves this little car. It has sporty dynamics with sharp steering, little body roll and good road feedback. If you like driving for the sake of driving at all, the Fit will give you a way more involving ride than the Yaris (or any Toyota really).
In addition, it's very efficient with its space. It's a tiny car, but it offers a lot of storage configurations with its "Magic Seat". You can fold the rear seat vertically so the bench goes up. You can fold the rear seat horizontally flat. You can split it so that you have part of the rear seat completely flat, and part of it upright for seating. It just gives you a lot of options to carry stuff.
Downsides? My brother owns one and these are his complaints:
- No arm rest.
- Stiff suspension, so road irregularities (potholes, uneven pavement) will be felt.
- Like most Hondas, it lacks low-end torque (peaks at 4800 RPM), so in order to get throttle response, you kinda have to drive it like you're angry - but keep in mind that's also why some people love it.
Toyota Yaris
-------------
People definitely think it's a better car than its predecessor, the Echo, but there's some peculiarities about it. First of all, the sedan and hatchback models were designed by different teams. The actual mechanicals are the same, but exterior and interior design are specific to either model so if you were checking one out, I'd suggest looking at both.
If you just want an A-to-B car, the Yaris will provide comfortable transportation. It's no-frills, and the floaty suspension with soak up bumps pretty well. It's also a bit more affordable than the Fit.
Drawbacks though...
- Center mounted gauges. They stuck the speedometer and tachometer in the middle of the car so there's nothing in front of the driver. I guess one can get used to it, but for many of us who are used to having all the gauges on the driver's side, it is a negative and perhaps worse for safety?
- The floaty suspension means that you are not going to be thrilled to drive it for driving's sake. It is strictly an A-to-B car like I said, and if you have any sporty pretentions, the vague steering and body roll will discourage you from taking sharp turns or playing around in the twisties.
- The dimensions are comparable or maybe even larger than the Fit, but you actually have less room for people/stuff.
--------------------------------------------------------
With both cars though, I would definitely recommend getting a manual transmission. With really small engines like these (barely over 100hp), you really need to be able to wring what you can out of it and automatics just don't give you as much flexibility. Plus... you'd be losing a gear (4 vs 5 speed) with an auto box.
I'd also add... that perhaps you hate driving because you've only driven a Tercel

I may seem anti-Toyota, but that's only because that's what I own. I never really cared about driving with my trusty Corolla either. It works, it's comfy, it's practical. But it wasn't ever exciting. Inheriting a Miata changed all that, and now my Corolla just sits in the garage.