Skyward Sword is my least favorite. I don't know if I can truly qualify it as "worst" since a lot of my dislike for it is built on a growing boredom with the series structure, but it was definitely the least magical one I've experienced going into for the first time.
That being said, if we want to make a case for actual worst - I think the original might kind of fit the bill. I don't mean it's a bad game - it was absolutely groundbreaking, spellbinding, etc etc - but in terms of design it's by far the most obtuse and unintuitive. That's understandable since it's the first in a now well-established series. They were trying new things and the series got refined as it went on, but seriously, the original Zelda is rough if you strip it of historical context. Which, to be fair - is how it should be, considering it's the very first one.