Has a non-Nintendo Smash Brothers genre (Smasher? Brawler? Competitive platformer?) game really ever caught on?
That Ubisoft TMNT game didn't make a splash.
A traditional fighting game would potentially have a built in audience in the FGC for the gameplay if it's decent, disregarding the arguments of Sony character brand recognition.
What does everyone have against a SF/MK game?
Is it that Smash Bros and that TMNT thing are the only ones in the genre?
There have been a bunch. Jump Super Stars, that TMNT one, a Cartoon Network one, an Onimusha one, a Viewtiful Joe one, a Japan-only one with Solid Snake, Optimus Prime, and Bomberman (which sounds way more awesome than it actually was).
The only one that ever caught on in any way was Jump Super Stars, which was successful enough to get a sequel, but never got out of Japan thanks to the mess it would've been to license all the Jump characters in other countries.
The rest didn't catch on because they were bad games. They were made by people who saw the success of Smash Bros. and copied it without understanding why Smash was popular and fun in the first place. They copied the superficial flashy and chaotic nature of Smash without capturing the underlying reason the flashy chaos in Smash is actually fun.
As far as Sony goes, though, I don't think they should just straight up copy Smash. If I were doing it, I'd probably take something like God of War's gameplay and adapt it into a multiplayer brawler. Same button layout: jump, weak and strong attacks, and grab on the face buttons; evade with the right stick; L1 to block, and hold L1 and press a face button to do power attacks; and hold R1 and press a face button to do special attacks. Relatively simple, but open enough to provide each character with a good variety of different attacks.
It would basically be a 3D multiplayer brawler like Power Stone, except with some semblance of complexity and depth.