It's both. The brain can be (and often is) physically different between genders. We aren't just talking sexual organs and physical appearance. Different parts of the brain can be different sizes, more/less active and so forth. It's all measurable by science and modern technology.
To try and be hard-line on either side of the fence, physical OR mental is to deny biology. The answer is more complex than JUST one or the other. It's a combination of both. We've learned a lot of this from the animal world, let alone studying ourselves, and unless you want to argue against evolution the science is correct, and we are all "animals".
Gender is a social construct is some nonsense you'd expect to read on a Tumblr blog. It's biological, both physical and mental, and YES it is possible with all the complications of making babies, genes and development for a male/female brain to unfortunately end up with the opposite body. Why wouldn't it be? Things can go wrong during pregnancy. We know that from tons of ailments our babies end up with and can die from. From down syndrome due to unfortunate issues with chromosomes, to heart/breathing problems, to skin problems and so on. So why would it be oh so insane to believe male or female brains could end up in the "wrong" body? There's a laundry list of other shit that can and does go wrong in pregnancy.
Heck in a world where
conjoined twins are a thing, and babies can be born with both a
penis and a vagina, what the fuck is the problem believing in
wrong brains in
wrong bodies?
Science doesn't give a shit for your bigotry and inability to use your critical thinking faculties. It deals with logic, reason and reality. Not "feelings". So when the Pope "feels" something is "against nature", Science comes back and says
fuck you Pope, here is the reality.