You're probably right, but this doesn't explain why some games sound like shit with VSS enabled while others do sound great.If any game outputs a standard 5.1 soundfield it will be converted to a psychoaccoustic simulation over 2 channels.
The game audio engine doesn't need to produce the psychoaccoustics.
That's how Turtle Beach can do it using Dolby Headphone over an optical cable too.
If anything, TLOU is perfectly aware of what it's outputting its sound to:
http://giant.gfycat.com/FocusedWarmAlleycat.webm
(showing the game knows that you're using a headset, and also that it switches automatically from a 2.0 to a 7.1 audio mix, and the other way around, whenever you enable/disable VSS)
Could be the reason why it sounds good with that game.