I don't think this is nearly as "simple" for Nintendo as it sounds because Project M did not exist in a void next to Melee. It was the "next best thing" to past Melee players that got burned on Brawl. A hypothetical situation in which Brawl, as released by Nintendo, is basically what we know of Project M, or more realistically, is something closer to Project M, could have resulted in Melee getting abandoned, or it could have resulted with someone developing Project Double M, or it could have led to Melee fans collectively deciding to stick to Melee faster than it happened in our reality. To paraphrase Chronicles of Narnia, you never know what would have happened, what you may know is what did or will. Exodus of PM players towards Melee did start before the project takedown and tournament denials being things.
By Smash 4 release the Melee community has grown cautious. Things that would have probably been just accepted in pre-Brawl world are way more prone to detract Melee players from supporting a new game, because by now their default way of thinking is that Nintendo can't top Melee, as supported by previous games not doing it.
An interestingly attractive element of proprietary games as opposed to "open" ones is the rule arbitration. In chess, there have been people playing with mods of sorts to make the game interesting once it started to get boring to them, and these mods eventually became recognized as official game rules of some organizations, composed of players. This is not the case with, say, Street Fighter V, where whatever Capcom says are the rules of SFV, is the rules of SFV... when you play online, at least, since that encourages version updates. The negative side to this is that you can't try out changes to the rules even if they feel blatantly obvious to you. The positive side to this is that players are not tasked with determining whether the rules of the game they play are broken by some new strategy, or is that a new and exciting consequence of the rules that does not invalidate them. The community decision is limited to playing SFV or not playing SFV. By sponsoring SFV and not older games, Capcom pushes the earlier option.
If you wanted to make a Smash game that would bring the Melee crowd over, you would have to make a game that is interesting to that crowd, but there is no obvious criterion for determining whether it is or it isn't, besides the "is like Melee" test, at which Melee wins by default, as it is the arbiter. Of course each individual member of Melee crowd has some sort of preferences that could be fulfilled in some sort of opposition to that test, but for crowd as a whole to move on, you would probably need something more arbitrary. Now, Nintendo could throw in some extras for the tournament scene in general in order to make Melee less attractive, the tricky part is, they didn't really do this before, so it could meet a pretty cynical reaction.
I think what will eventually break Melee will come from the inside instead of the outside. Mass hardware failures, huge community scandal, a strategy that makes 90% of matches boring, something along these lines.