During the late 80's and early 90's Nintendo had almost complete control over the industry, they were by far the largest game company and their hardware sold by far the most, there were all kinds of people purchasing Nintendo hardware, fans of all sorts of genres, demographics, ages and such, essentially everyone who was playing videogames was playing on Nintendo.
Nintendo was such a big deal in 1986 that essentially everything the put out was a major success only because of their branding, with Metroid being happily affected by it, the game sold over a million units in Japan being one of the best selling games on the Famicom Disk System.
The series was popular enough that its sequel, released 5 years later, sold a few hundred thousand units as well, despite being released over 5 months after it's NA release, Metroid didn't became as popular as another Disk System franchise, The Legend of Zelda, but wasn't doing poorly either.
8 years after the release of the original title, Super Metroid saw release on the Super Famicom, by 1994 it was a different industry, most of the people who played the original Metroid were kids at the time and likely got it as a gift from their parents, and well the ones who brought it out of their own money probably weren't playing anymore, whatever the case was, this older audience was looking for more mature experiences and a dark moody game like Super Metroid was perfect for that, the title sold over half a million units, more than Metroid 2 and was Nintendo's 2nd best selling game of that year, behind Donkey Kong Country.
Then the PlayStation and Saturn came out and the people who were interested Metroid like titles jumped ship, the Nintendo 64 was the worst selling console of it's generation and it's unlike Metroid would've helped on that.
More importantly, the people who had grown up playing Super Famicom were now old enough to buy their games themselves, as they were now investing their own money they were picky as to what games to buy, so mostly long single player games were the most popular back then, this is why Final Fantasy hit it's popularity peak during these days and why Resident Evil became so big in the first place.
At this time too, the idea of selling games after you were done with them became more and more popular, good idea right? If I buy a game for 9.000 yen, it's over in 6 hours and has no replay value I could just resell it and get back some of the money I put up first. This is why shorter games starting to sell poorly in Japan and ultimate killed Castlevania and Mega Man's popularity in the region if I can beat the game in a day I just sell it and get some money back for other games, then other people instead of buying an expensive new copy buy a cheap used one and the game stops selling.
Metroid would've died in popularity at this time, unless it became a huge hit like Dragon Quest or Final Fantasy, or a moderate like Ocarina of Time, the people who played that kind of games had moved on to other consolas and changed the way they bought games. SOTN was essentially a Super Metroid 2 but being a short game with not much replayability it sold less than half of Super Metroid, and other Castlevania or Metroid like games didn't sell much better.
By the time Metroid finally returned it was too little to late, the little fanbase that survived was on GBA and now could've just resell the game, Fusion is over in 5 hours, it never had a chance. For Prime things were even worse, it released at a flop of a console that at the time had only family friendly games, more importantly the game was seen as part of a genre that was unpopular at the time and this time the quality of the game couldn't save it.
Unless you're Pokémon and Dragon Quest remakes don't sell well in Japan and so Zero Mission was ignored (it's 2 hour lenght didn't helped either), NGC was dead by the time Prime 2 came out so it sold worse than it's poorly selling predecesor.
Prime 3 had the same problems the first one had, except the install base, demographics weren't there and the game sold poorly, same problem with Hunters. However Nintendo did gave Prime and Prime 2 a shot in Japan again by releasing them under the New Play Control! label, and guess what, they both sold poorly.
There's nothing to say about Other M, it sold as poorly in Japan as it did everywhere else, it wasn't well recieved either.
The series had a string of bad luck regarding releases and market changes, if Metroid was popular today in Japan it'd be a very different series.
