It's an unfortunate state of affairs, to be sure, but what do you want? These games have simply gone out of style, to the point that these big publishers really aren't interested in following up with the legacy anymore. Sure, they "could" and they have done so to different degrees over the years (Igavanias, Hardcore Uprising, Contra 4) but I am going to suspect that the numbers these games were doing were diminishing to the point where it simply was not worth the effort. The sad truth is the mainstream gaming public as a whole has no interest in such franchises, they are niche. And of course so much of what has been said earlier in this thread - the people who have originally made those franchises into what they were, are long since departed from either the company or the industry or what have you.
It's not all bad news, the spirit of those older games lives on through devs who they have touched. It sucks because people here (myself included) would love to see a modern update for Mega Man, Castlevania, Contra, Gradius, etc which are true to their source material and not just in name only: but since that's really not happening, dig around in the other scenes (downloadable, indie) and see that there's a fair representation of spiritual follow-ups in the mix. They are not in short supply - they may be a little unbalanced (how many more indie metroidvanias do we really need?) but there is an active enough scene that produces such output.
And the knee-jerk to that answer, sadly, is that those who do keep a keen eye on the indie scene will often jump on such games "this is a ripoff of [classic game], I am tired of so many indies making throwback games.. indie games are junk, not 'real games,' etc" This is not the overpowering response, fortunately, but it is a strong one.
Anyway, the takeaway is that although Konami/Capcom/etc have long since abandoned these beloved decades-old franchises, that doesn't mean these styles of gaming are dead and buried. If not in name, they do continue to survive in spirit, and the best of them will still get a chance to thrive. All it requires of the audience is to do some work and dig around, participate, and support with your $$ and your word of mouth.