The problem with the world of Metroid is that whenever they move beyond moody environmental storytelling it becomes goofy and terrible. Even the basic premise of the series, that there's these weird Jellyfish aliens that are somehow the ultimate bioweapon, is more than a little bit silly.
I'd be more interested to have Retro develop a brand new Science Fiction IP as a clean break while preserving preserving similar design priorities, but that's a bit of a pipe dream and Retro now isn't Retro then anyway.
That's because there was no true effort to expand the series' lore since 2002
Prime 2 tried to be like Zelda but the Luminoth and such and up feeling irrelevant af, they could have done so much more(having better realize areas for starters) to motivate the player to help them
Other M was just unnecessary, you had shounen filler tier writing to support cinematics and such for a game that was pretty much outdone by Fusion in every aspect
As for Prime 3 and the multiplayer games, they are basically kneejerk reactions to the media/gamers in general that saw in Metroid Nintendo's answer to Halo or whatever. The other characters and such were very half-assed, none were remotely as memorable as Samus
I think Prime 1 did a great job at providing context for the game, like you can say way more about Tallon IV than you would about Zebes. Fusion did the same in regards to Samus' role in that universe. Both are definitely good examples of the series going beyond the basic premise. Sadly they never really lived up to that in the other entries. Star Fox and F-Zero kind of had the same issue, they showed the potential to basically have a much more interesting space-opera like setting with Adventures and GP Legend yet they basically died right after, Star Fox settling on just rehashing 64.
Come to think of it I can't expect modern Nintendo to deliver on those, their games from the last 5 years were extremely light on story. Right now they are too prideful of having extremely simple premises to justify that they are the only developer that delivers "pure gameplay". Star Fox Zero would have sold way more if it looked more interesting/wasn't just a rehash of past games. Outsourced anime shorts are not enough