Not all games are for everyone, but if you don't make all games for everyone, they won't sell nearly as well as games that are explicitly for everyone. Ergo, the people who like difficult games that were made to be difficult, even if it's just because tradition dating back to quarter-sucking arcades, are going to be afraid that people who can't make the cut are going to demand games LIKE it be made, but accessible. In the process, they'll be made easier because accessibility is hard, and making a game piss easy is easy.
And those games that are made are going to be more popular than their previous incarnation. The free market speaks, and it says "fuck the games you like, yes, you."
So on one hand, you have people who want to enjoy games that are relegated mostly to the people who are willing and able to put in the time and effort to get good at them, but because they are not willing and/or able to get to that level of play, are excluded from ever really experiencing them. And on the other hand, you have people who take joy in the suffering when they try and fail and try and fail and try and finally succeed, who fear they will LOSE something when games no longer do that in an even half-way organic way.
And because there are more people in the former than there are in the latter, the industry will move toward that, instead. Just like the industry moved from skill based shooters with variable movement, weapons, mods, and maps, to more drudgery like Call of Duty, shoot first win first, etc.
Making a game more accessible doesn't always mean making it easier, but most of the time, it's "making it easier to fake accessibility," which actively harms MY enjoyment, for instance.
So then it turns people against each other. Why should I care what the casual scum wants when they need gimmick waggle, verticality and speed scares them, and if they count more than two weapons at a time they get panic attacks? Why should they care about the neckbeard who sweats cheeto dust, when what they enjoy is anathema to what the rest enjoy?
Difficulty levels lessens animosity, allows people to enjoy a hard game, without sacrificing what makes the "real gamers" attracted to it. ...When done right. But even then, there's certain elitist views.
And this is present across multiple cultures, too.
Two characters from the Touhou series, a series primarily consisting of "bullet hell" games. Easy mode allows more people to enjoy the game. It has fewer bullets than Normal, or Hard, or Lunatic. But then, the question arises; WHY are you playing Touhou if you don't want a challenge? WHY would you play in Easy mode when there's harder difficulties? "Why are you playing a game that is not for you?" "What right do you have to try to tell us what we should enjoy in our own genres?"
Everyone enjoys something, and I would contend that it's not that hard to please everyone, but the divisions by skill and derision because of them are going to exist regardless. Hell, it's the ONE thing humans are good at.
I enjoy FPS games tremendously. But I'm almost 30 and the dexterity is reduced from what it used to be. I used to be pretty goddamn good, but now arthritis and nerve injuries are real risks, and I can't play 14 hour marathon sessions in the games that I used to against other people. I'll never get out of diamond league in Overwatch now. But 10 years younger...definitely. So there's a good argument, on both sides, about difficulty and accessibility. There's a good argument for putting it in games, as long as you don't sacrifice what made it enjoyable to do so.
There is no argument for NOT doing so. There's no argument for no easy mode. There's no argument for why we treat others derisively because t hey can't play at the same level (or anymore).