The script wasn't nearly as crisp as the Avengers. The villain wasn't as appealing as Loki. To be more frank, Spader was a great character but his character didn't work as Ultron. The banter between the Avengers was not as good. The editing/pacing was uneven. The new characters weren't nearly developed enough and motivations of said characters were flimsy at best.
AOU had a strong opening set piece and and strong close but as a whole, it fell short of the natural progression of the Avengers brand, IMO.
Also, avengers 1 was the first time all big heroes came together in one frame for the first time.
The novelty wears off the second time around