Refusing to comply with identity isn't the same as restricting a person because of their born sex, race, or hell even religion. Trans gender people enjoy the same rights as everyone else, but that doesn't mean everybody else has to go along with their perceived gender because it's not as set in stone as a persons race or sex.
Sex refers to biological characteristics. Gender is behavioral.
And this is exactly why it's a complicated issue. When in discourse a gap is created between gender and sex, what follows naturally is a disconnect between behavior and biology where non exists. We are biological creatures and our behavior, our thoughts and feelings are linked to what we are. Sure we can make decisions, and for example we can choose when to eat or when not to eat, but in general when we are hungry we eat.
Now, being intersex is a purely biological condition, to put it crudely to be part male, part female. And your sex does not merely determine the dangly bits, but also hormones, brain structures, thought patterns etc. In short, your sex also affects your behavior.
We've had our first working head-transplant this month, on a corpse, but what if next year you were kidnapped and your head would be transplanted on the body of the opposite sex, what would that mean for you? At the very least new hormones would alter your behavior. I think I would still see myself as a man, just stuck on a woman's body. My brain would still be that of a male. If you'd give me a brain-scan you would see that my brain is structured like the brain of other males. That's no social construct, no feeling, that's actually there.
Now, I've been a male my whole life, so I know what this is like. Now consider people being born with a brain like mine, but either with a mixed set of genitals or purely female genitals. I can imagine - like I would - they would like a male body instead.
But it gets more complicated,... being intersex is never clean-cut. Sure you could have an all male brain and all female genitalia, but usually when intersex it's not as clear cut. Brain development could be a mix from both sexes, looking unlike either one, just like is often the case on the development on the genitalia.
I can imagine that is very confusing, how can you identify as either male or female being unable to experience being either? Some people might easily feel like either one, pick one and be happy with the choice and the world would be non the wiser that their brain has characteristics of the other sex. Some people might feel that they are neither, and might or might not later still decide for convenience sake pick a gender.
What they do here, and how they decide to handle their condition might not be set in stone, but the brain you are born with you are very much stuck with. It's easier to change the color of your skin or the genitalia you have than it is to restructure the brain that is developed in the womb and throughout your childhood.
TLDR: This is an actual biological condition, and academics discussing it like a soft-science are making it sound like people just feel stuff and feel whatever they want. Sex and gender cannot be separated and that doesn't mean everybody is either a boy or a girl, some people are born in between.