So, even though my original choice was QoS, I just re-watched it since seeing it in theaters and it actually made Skyfall worse. QoS really isn't that bad, and sets up what we all know as Bond - but I feel they for some reason decided to try and further set up Bond in a third film, which just dragged it along even further, punching us in the face with more 'what he becomes'.
So, as much as I love the fight scene in the skyscraper, I'm putting QoS above SF.