Yeah but people can learn football rules in three minutes. You need a lot more time to understand what is going on in a fighting game. How do those casual viewers enjoy it?
Fighting games can be learned in less time. Just like Soccer/Football you can easily learn from just watching the game be played. Both people have a health bar, getting hit depletes the health bar, depleting the opponents health bar wins you the round, 2/3 rounds win you the game, 2/3 games win you the set. That's the basics of practically every fighter. Then you have to get into things like meter/enhanced moves, crossups/mixups, and the meta of the game you're watching. But that's also the same for that sport as you have to learn about penalties, what gives you different cards, how substitutions work, the roles of the players, etc.
Edit: Here's a game that many people don't watch, Arcana Hearts 3.
https://www.youtube.com/watch?v=SSUvy3PCFJg
It follows the rules I posted. As the action is going on the commentators are explaining what's going on with the characters. They speak on mechanics like burst and, even without explaining what it is, you see it's a move that knocks the opponent away. Fighting games are very self-explainatory and a lot of time the commentators explain the mechanics for newcomers.