I feel the same way, but it's not really because of the puzzles for me I think, it's more of an environment problem.
Like for example, if the game is open world and I'm looking for some secret / collectibles I missed, going back to an open area and searching for these secrets is fine, but if it's in an interior like cave, dungeon, maze or whatever, I absolutely hate it.
There's just something about "dungeons", dark interiors, that kind of repulse me if I have to stay too long in them, even more if they are labyrinthine. When I'm done with an area like this, I immediatly want to get out, and when I'm out I feel a lot better and never want to go back in, they kind of give me anxiety.
Puzzle in outsides, open environments are fine though, so it's really the dull repetitive design of dungeons in general, and their claustrophobic feeling, that doesn't work with me as far as I'm concerned.
I also hate how games never allow you to fast travel out of interiors like this.
Like I've been playing Biomutant recently, and as always, if you are in the undergrounds and open the map, it will say "you can't fast travel from here", so you have to walk all the way back to the surface manually. I don't understand why, it's not even like if it added a challenge or anything, I already killed all the ennemies, so it's just tedious, you can fast travel anywhere from anywhere, except from "dungeons" (or sewers in this case, same thing), it's annoying, tedious, and makes me hate these type of areas even more. And it's been like this in every games since forever.
