They grew up and realized that they weren't going to inherit the earth. They needed to be well rounded individuals, which they never became. All the nerdy things they were into were great and all, but they needed to be able to be social with others and play by established rules to succeed in life.
They never had to before, because instead of learning how to interact with people not like them, they formed groups consisting of people just like them.
All of a sudden it hit them like a ton of bricks, but instead of working up the social skills they didn't develop growing up, they blame everyone around them, tricking themselves into believing that the world should work for them and value them for what they have.
The same thing happened to the super popular who peaked in high school.
This, of course, is not all nerds. Most normal nerds grew up, learned social skills, developed additional, more varied interests, and grew as people to be able to function in the real world.