It wasn't that smart financially IMO.
Aside from Rare's original IP's like Perfect Dark and Banjo, they were great at utilizing Nintendo's existing franchises. Starfox Adventures was one of the best-selling Gamecube games and Diddy Kong Racing would've been a guaranteed multi-million seller too. Rare had like five teams at that point and they would've been great at filling gaps in the lineup.
I personally think Nintendo would've made far more money by keeping Rare.
Both the quality and the pace of their output was already on the decline, and seeing the development hell they immediately sunk into after the buyout I'm not sure they would've been the Golden Goose you imagine they might have been. $100 million was huge for Nintendo at that time. With that they could literally have bought Retro Studios 100 times over.