It is a downgrade, in my view you have just been fooled into thinking that more restrictive features are somehow better.
First of all the Notch is just antithetical to Apple's design aesthetic IMO. That thing is disgusting. Like every time I watch someone use an iPhone X it actually bothers me that it's there. You don't "just have a big screen". You have a screen with a fairly large portion of it cut out by this rectangle thing. It gives this weird asymmetrical look to the phone when it's turned longways and it just gives the sense that there is more screen under it that it's covering. It's very inelegant and destroys the idea of the iPhone X being some premium device because it just looks like shit. Sorry.
FaceID is in no way an improvement over TouchID. The idea that it somehow should be a replacement for it is ridiculous. With TouchID (and a freaking home button), my SE is open before my iPhone camera is even pointed at my face. And of course, I had to make a conscious decision to open it - it's not like I looked at my phone to check the time, it opened because of FaceID, and then I had to lock it again for no reason. TouchID is just better. It's faster and less restrictive. Even if someone preferred FaceID, Apple should give people the choice.
As for screen size, I have never felt I needed more screen size out of my SE. If I want to watch something on a bigger screen, that's not something I'm going to do on my phone. But, are there times when I want to be able to access the entire screen space with one hand? Absolutely. This is because of the nature of a phone. Suppose I'm carrying something with one hand and I want to zoom in with the other...well, I can do with that. Carrying a textbook and reaching for the link on the far side of the screen is no issue. The SE is the perfect size.
Headphone jacks are extremely useful, and the idea that they shouldn't be on a modern phone just this weird, pseudo-futurist crap that makes someone sound like they are trying to get a job at Apple because of how sycophantic they can be towards the company. Blutooth headphones simply do not sound as good as good wired ones. They are annoying - you have to sync them up over and over again and something often goes wrong. They have to be charged. If you're at the gym and some hot girl tries to talk to you you can't quickly pull the wire near your neck with a finger and let one of them hang over your shirt while you keep lifting weights. With wireless headphones you have to put the weights down and fumble with your overpriced AirPods like a dumbass to put them in your pocket. The girl's probably gone by this point and then you have to put them back in your ears. Man, if only you had been using an SE. Hopefully they don't die during your workout.
Before I go to bed every night I do some reading on my phone while listening to music and charging it at the same time. Wow it's amazing what we figured out 5 years ago. With iPhone X I won't be doing that because of some wack ass design choices. Wow wireless charging! Instead of my phone being tethered to a wire while I charge it now it has to lie flat on a surface - so innovative! I mean come on seriously this is just a bunch of bullshit...
You mention the specs of the phone...the specs of the phone give no credibility to these regressive design decisions. If there was an SE2 right now it would have the same speed and camera quality.
Well there's my rant. Now there have been many articles expressing similar opinions (ex.
https://techcrunch.com/2018/09/14/the-iphone-se-was-the-best-phone-apple-ever-made-and-now-its-dead/), hopefully Apple makes an SE2 if not I will have to find something new when the time comes.