Many reasons.
Pure arcade experiences were kinda stagnating. There’s only so much you can do with shooters, fighting games, racing games and rhythm games. Yeah, people will always enjoy a new one, but for any really good one there were literal dozens of clones saturating the market. You don’t get a new audience out of that. There’s a reason the video games industry only really exploded when consoles became fully 3D-capable. The arcade experience was always going to be limited, expensive, and repetitive.
Then there’s the unquestionable draw people have towards stories, and moving pictures especially. A lot of games had stories and context beyond the few actions permitted by the gameplay even before the mid-90s, but the stylized, symbolic nature of most of those games’ graphics were a turn-off for many, and the stories were often too simple. Most people apparently need a good balance of narrative and purposeful action to get interested in a video game that’s more sophisticated than the simplest arcade experiences. Mindlessly shooting spaceships or moving Pac-Man through a maze to gobble pills is fine for most (which is why the early 80s’ arcades were loved by adults too), but the moment you introduce something more complex, a lot of people can’t cope with 2D stylized graphics. If the story is “real”, the images must look “real”, too.
We saw this with the slew of Full Motion Video games that were all the rage for a couple of seasons. The gameplay was often abysmal, yet just having actual video-recorded scenes was enough to get people to buy CD-ROM drives for their PCs and buy games they’d probably never even get close to finishing. This looked exactly what the CD-ROM had been made for. There was basically no other reason for games to come on CD at the time. But suddenly, CDs became necessary for video games to “advance”.
Then when CD-based console came, the most basic way of using all that storage was video. So devs did their best to fill those CDs with video, and as crude as those videos were, they were exactly what most people needed to finally “believe” the stories that games offered. So in turn stories had to become more complex, darker, grittier, more “mature”. And a new audience was conquered.
I think it boils down to this. Most people enjoy pure arcade experiences as simple timewasters, but when a game gets more complex, they need an element of reality to get involved. It’s the same reason why adults never cared for superhero cartoons, but they’re now flocking to cinemas to watch the umpteenth MCU movie. It’s also the reason why GTA was moderately successful in 2D and a top-down perspective, but took the world by storm when it made the jump to 3D. Just making a pure arcade game with realistic graphics isn’t enough. There needs to be a synergy of narrative and realistic-looking images. Like, if it looks real, the game has to give you a purpose, otherwise it’s not interesting. It’s no wonder games like Candy Crush Saga and the various endless runners still rely on simple, colored graphics - people associate that with something that doesn’t have to “make sense” beyond the simple actions that take place onscreen.
Also, when games started to become bigger, simpler games started to look like they weren’t worth the money; unfortunately, those simpler games kept asking the same price as the biggest releases for a long time. On the Game Boy, the shortest platformer would cost as much as Zelda. On the PS2, Fantavision and Katamari Damacy would cost as much as Onimusha and Kingdom Hearts. So the purest arcade experiences were slowly but surely pushed off the mainstream market because you had to choose what to invest your money in. It wasn’t until digital games became a thing that an adjustment in pricing was finally made, but by then it was too late for what is now the indie market, and now it’s not hard to understand why most people only care for the big cinematic AAA releases.
