There are four main reasons that I see.
1. Their development methods are woefully outdated.
You keep seeing retarded statements from Okamoto and whoever saying things like "You can only make a Western game on a Western engine," which is... probably the dumbest thing I've ever heard.
The core reality is that you can't make competitive games these days without technology sharing, which is something that is apparently antithetical to Japanese development.
Historically, Japanese would tell you their games were "handcrafted." To a Western developer like myself, that means "hacked-together." They start from scratch every time, and there are no real systems to speak of, let alone reuse.
I've heard Western publishers hiring Japanese studios, and being very frustrated by this. The Japanese studios resist the engines they are given, and don't believe it will save them time and money in the longrun, which will let them make better games. Maybe after 3 games they'll figure out why it's good, maybe not...
2. Even if they had the will to, they lack the technical know-how to modernize their development methods.
The reason Japanese games were comparitively awesomer back in the day is because they have always been driven by the creative forces in their companies - designers and artists. Programmers were considered lower and mere implementors of their vision. At the time, the West had the opposite problem - putting process and programming before creativity.
Eventually Western developers figured out that fun and quality are good, and you started to see very creative-driven studios bolsterred by the West's technical- and process-driven background.
Japan, on the other hand, doesn't seem to have the technical background to catch up quickly, having not historically valued it.
I'll use a recent example...
A friend of mine is going to work at IREM. He was looking at their site, and there was a Q&A in the jobs section...
Q: I want to be a programmer. Do I need to know how to program?
A: No, as long as you have a love of programming.
Meanwhile, in the West you have people with computer science degrees and a lot of extracurricular game-specific training applying for jobs.
This also makes me think that maybe Japanese computer science courses just aren't very good? Maybe it's because to the Japanese, all the Western-created programming languages adhere to an alien logic that is incompatible with their grammar.
I really don't know, but you'd think this problem could sort itself out with new people coming into the industry. But...
3. The Japanese game industry does not seem to be attracting new people.
The West is very much engaged with indie development now. Japan, on the other hand, is not. Not only is this fueling creativity in Western games, it's also a fantastic and productive in-road to the industry.
You don't have to look at the gaming media very hard to see this. There don't seem to be any young, ascendent Japanese producers or directors... they're all 40- or 50-something men, who have been doing this for 20 years or more. On the other hand, you see quite the opposite in the West, between indies and the big publishers alike.
My sense is that the Japanese industry is an old boys' club, and no longer taking applications. Their handling of the indie scene in Japan reminds me of the way they handle internet media - they are actively disdainful of it, because it threatens their top-down control of things.
Cave Story has been lauded for years by gamers, yet it took a Western indie company to bring it a paid platform. Likewise, how many downloadable games do you see from independent Japanese studios? They're almost all made by internal studios.
4. They don't know how to expand their audience, so they're desperately trying to retain it.
For years we've seen reports that the Japanese gaming audience is shrinking. To counter this, Japanese publishers have focused on more and more "easy-sell" games and genres.
I kind of see the Monster Hunter-ification of the Japanese games industry as being akin to the Moe-ificiation/Endless series-ification of the anime industry - focusing in on the genres that hardcore want most, to the detriment of everyone else. And eventually even the hardest-core Monster Hunter fans will stop buying those.
Western publishers are guilty of this as well, of course, but our industry is still growing so they are somewhat less risk-averse. And thanks to the indie scene, publishers can identify the success of new ideas without breaking the bank on them.
But outside of Nintendo the best selling games are mostly non-Japanese. Nintendo is 1 company in a massive sea of developers, they are the outlier not the norm. 
