my god there's so much misinformation here, it's almost as if we're witnessing the rewrite of a pivotal moment in gaming history.
every generation this question is asked by 'newer' gamers, and every generation the reasons and stories become even more uninformed, distorted, and outright fictional.
there's about a handful of people in this thread so far that have mentioned something that is specifically 'accurate'.
i've taken part in addressing this question in the distant past on other forums with a slew of links, newspaper clippings, interviews with square executives involved, etc.
the problem now is, because that was so long ago i no longer have those links at my disposal, unfortunately.
anybody who says 'cartridges' as the full and definite answer is wrong. that is NOT why square left nintendo. that wasn't even the straw that broke the camels back.
nintendo, up until the gcn, was very terrible with their treatment of most 3rd parties. i say most, because there were some developers like enix, who were treated like royalty in comparison.
from what i remember, square, too, had a very close relationship with nintendo. they enjoyed a fruitful partnership and made a lot of money off each other.
square's dissatisfaction started during the 16 bit era. again i don't have all the details anymore, but people mentioning Nintendo being 'cheap' to their close partner with examples of disallowing square the extra megs they needed for larger cartridges is absolutely CORRECT.
Square made plenty of requests of a similar vein; all of which were denied by nintendo. in an act of complete douchebag-ery, when enix asked for the same thing, nintendo gave it to them without question.
this didn't happen all at once. multiple requests throughout the generation were denied and it turned square off, ever so slightly
Square and Nintendo had their fights, mostly because Nintendo was a dick, but Square kept coming back to Nintendo because Nintendo was generally less of a dick to Square than they were to other publishers. Also, Nintendo was money, so most people couldn't afford to walk away from Nintendo.
Nintendo had this thing going on the SNES where you're supposed to ask them for (as an example) 2 million cartridges and give them a massive money order for that amount, and then Nintendo decides that you only "need" 500k units, so they arbitrarily sell you 500k and give you back your change (you have no say in the matter, Nintendo decides how big your game is going to be). It was crazy because you had to have enough cash in your bank account to buy a money order for 2 million units, even though you only want 500k. And if you tell Nintendo that you only want 500k (and put forward that much money), Nintendo will decide that you get 100k, and then your game bombs. If you really do want 2 million units, you need to ask (and pay for) 10 million units. At expensive cartridge prices.
Well, Square was apparently supposed to develop AND publish Mario RPG, but they decided to send Nintendo a message about this stupid policy, so they asked Nintendo for an amount of carts that Nintendo typically gave Square, not the amount that Square typically asks for. Nintendo got insulted and took the publishing duties away from Square. That might have something to do with why Square owns the name and Nintendo can't use it, because Square (not Nintendo) was originally set to publish it.
But that particular fight didn't end Nintendo and Square's relationship, nor did FF7's FMV necessarily demand that FF7 be made for PSX. FF7 got FMV added to the formula after Square moved. FF7 was originally slated to be made on SNES after FF6, but then it got cancelled so the developers could work on Chrono Trigger, and after Chrono Trigger they got pencilled in for FF7 on the N64, and started to experiment on how to use the analog stick. They never got past playing around with the analog stick before higher-ups at Square decided to give Nintendo the boot and move to Sony.
Sony had flirted with Square over the original Play Station (the Sony-made CD add-on for the SNES). This resulted in some fights with Nintendo and canned/shoehorned games, and a few business cards being traded back and forth.
When carts were announced for the N64, half the industry objected (the other half cheered for the death of Nintendo), but RPGs were singled out as the genre which would be hurt by carts the most. So Nintendo quickly put together their "64DD" idea, which was good enough to get Square and Enix to hesitate and say "I'm listening". Nintendo apparently asked Square and Enix for a wishlist of things they wanted for the 64DD (anything except CD), and they apparently asked for premade text fonts to be built into the system, to make it cheaper and easier for text-heavy games to be made for the 64DD. I remember at one point I read in an old magazine that Square and Enix were being listed as "design partners" with Nintendo on the 64DD. Nintendo really wanted to make them happy. At one point, I believe the 64DD was pencilled in for about four games, Zelda 64, FF7, DQ7, and Mother 3.
But Sony had an actual CD-based system and Square's business card, so they asked Square "Seriously, what are you doing bending over backwards to support these assholes who have proven time and time again that they don't give a damn about you. Nintendo's trying to meet you halfway? Fuck halfway. If you were our girl, we would give you everything you wanted and more."
So yeah, Square dumped Nintendo and moved to Sony. And apparently when Square told Yamauchi about it, he was okay with it, because everyone else dumped Nintendo a lot quicker, and it was the right move for Square.
But then Sony treated Square like rock stars, and there were interviews about how awesome Sony and PSX were, compared to the hell Square had to endure with Nintendo and LOL carts, and Sony produced and paid for ads for FF7 in America which specifically took shots against Nintendo and the N64 (ads made by Sony, in Square's name). Another thing which was rumored to be a major factor in pissing off Yamauchi was that Square's president apparently met with Enix's president for coffee, and Square convinced Enix to dump Nintendo too. Now Nintendo was left with no RPG support, and the 64DD which was built almost exclusively for Square and Enix looked pointless. Zelda 64 got shoehorned onto carts, kind of like what happened to Square's games on the original Play Station. Square gave Nintendo a second taste of what it was like to be one of Nintendo's "partners".
So Yamauchi lashed out against Square, and held a grudge for years. He only let them come back (pretty much tried to bribe them, even) after Square's president was forced to step down.
some of this is correct.
a few things though, from what i remember researching…
**your point about the cartridge orders is spot on. from my memory nintendo did indeed do this. ridiculous, really.
**by the time the SMRPG project rolled around, nintendo already felt a strain on their relationship with square. that's why they offered them the mario franchise; as a show of faith of some sorts.
**FF7 was never an active n64 project. that tech demo was nothing more than that. FF7 did start out a snes game though, and it was originally set in NYC, and the main character was a detective called 'joe' or something.
**square indeed flirted with sony a little during the snes cd rom debacle. SoM2 was originally planned for that platform but development was switched, then butchered in an attempt to fit into your typical snes cartridge.
** the point about Yamauchi being ok with square's departure is also true. his exact response was indeed "it couldn't be helped". the insult square 'felt' at yamauchi being ok with them leaving is also true. i used to have the article that linked to an executive confirming this. again that was years ago and i no longer have it.
** the point about most developers already leaving by the time square decided to jump ship is a half truth as far as i remember. true, most developers had already signed up to make games for playstation, not too many of them though, aside from namco and square, committed to the psx platform exclusively. they still had plans to develop n64 titles. it was only after squaresoft actively sought them all out and convinced them to drop nintendo (which happened very early in the generation), did they finally commit to playstation exclusivity. Onimusha for example was rumoured to have begun as a n64 project, which moved to psx. when capcom realised they couldn't fullfill their vision on that platform, they moved it to ps2 where it released.