It doesn't help that Other M was a fairly high budget game that sold very poorly and that Prime 3 was already showing a series decline in sales before it.
This goes well beyond where games sell better. It's a simple business decision. Pikmin is relatively cheap to make with a small team. Metroid games are higher budget with bigger teams. It's a numbers game. Pikmin may sell "less", but it doesn't need to sell as much to be profitable.
Nintendo did it to themselves. They killed Metroid. They're content with that fact, clearly.