I think the other problem is that Mitt and Ann Romney and the campaign pushed the narrative that Mitt is actually still a fun loving, prank doing guy underneath it all. That was in response to the notion that he's really some sort of unfeeling robot. So they tried to convince people that, hey, underneath it all he's still a wild and crazy guy like when he was young. Then it comes out that his wild and crazy youth was really just not that "fun" and a bit more questionable.
So I think it's fair to really bring this up and condemn it a little bit, because they pushed that he was still this sort of guy. Then when people questioned it he wasn't very apologetic, further symbolizing that he really has no problem with the behavior, or at least just doesn't fully understand why people would be upset at him for it.