Call me biased, but physics has to be at the top or close to it. Incredibly conceptually challenging, and an insane workload. Basically, during my last two years in physics, if I wasn't sleeping or eating, I was working. Just a brutal, brutal degree. And sure, you can say "everyone is different, what's difficult for some might be easy for others, etc." but the fact is, pretty much no one finds physics easy - even the people who are good at it.
As for other degrees: math and various engineerings have to be up there.