I think it was due to the novelty wearing off. Guitar Hero through Rock Band 2 are amazing games, but after Rock Band 2, there wasn't really anywhere that they could logically take it. They tried new instrumnets, more realistic plastic instruments, but you couldn't beat the awe factor of playing a plastic guitar for the first time, or playing in a plastic instrument band with your friends for the first time.
I know the oversaturation of games helped speed the process of falling out of relevance a lot quicker, but I always thought that Rock Band, like the Wiimote, had a shelf life due to what kind of people it was bringing in. That's not to say there couldn't be awesome Wii games, or that people wouldn't still enjoy playing those instruments, but the idea of it being a cultural phenomenom couldn't sustain itself for very long. Unlike game IP's that peter out due to lack of innovation or people moving on, the fact that the game(s) were built around getting friends together and playing really kind of made it a rolling stone of irrelevancy that would only get worse once people started to get bored with it.
Such a shame though, because Guitar Hero through Rock Band 2 are some of the best experiences in gaming. The problem is that you had to be alive and playing those games at the time because it's likely this is one of those things in gaming that won't be able to be recreated, or at least recreated in a natural way.
Who knows? Maybe we'll see a revolution of the plastic instument genre some day, and we'll all be reliving the awesome fun we had with these games.