Like others have said, Incredible Hulk is the only Avengers movie that isn't important to the Avengers canon. You should definitely watch IM2, Thor and Capt America to be prepared. And watch Thor before Capt America...at least Thor's end credit clip.
eh, don't let the trolls get to you. At this point, you can predict what their responses will be in this thread, since they've repeated them over & over.
And I agree with you. I've enjoyed the Marvel movies so far. I do think IM2 is the weakest one of them all, and I've still watched 3x so far, and will watch it one more time before the Avengers.
Expect a good time.