I don't think it's going to be going upwards.
When you're inside the VHS andre walks down that corridor, he could have either gone into the TV room or through that door. If he had gone into the TV room then family man would need to knock him out/kill him, open the secret door, get him through the secret door, close the secret door, get him down the ladder and then position him before leaving via the corridor to the left of Andre. To me it sounds more reasonable that Andre went through the door we can't open which takes him downstairs and links up with the corridor to the left of where we find his body. To corroborate this after you get the back door key the family man walks past from the direction of the door we can't open, either he's been stood there waiting to creep us out or he's coming through that door from another area, perhaps he's just finished removing the ladder and is on his way back up?
Ah, the door in the pantry likely goes downwards towards the cellar, as per my maps there is a void between accessible rooms which could very well contain the route down to the cellar, at the same time the door at the end of the hallway with the broken doorknob, from outside that section of building shows a window on the external face, in between floors, suggesting that it contains a stairway that turns either 90 degrees as it goes up or a full 180.
It's unlikely to go downwards immediately, unless of course it goes downwards in a 90/180 degree angle and the floor above repeats that for the next floor up.
In which case the pantry would just be an accessible room between the livingroom and kitchen.
Regardless, the map I had seen prior to making my own assumed that there was no space between the kitchen, livingroom and first hallway, that's incorrect, while my updated maps wont be 100% accurate (give or take a few ft here and there) it's about as accurate as we're going to get without a pc version to rip apart!
Will investigate outside a little more and see if i can find evidence of the stairways direction
Random note: i've been trying to glitch past Andre in VHS, if you hold left on both sticks and rapidly tap L3/R3 the camera twitches like crazy as if its trying to break free of the forced movement.
[Edit]
Lending credence to the theory that the cellar can be entered from the end doorway, if you listen to audio as Andre walks off, after the exact length of time it takes to walk from the bottom of the stairs to the door, you hear it open then close behind him, at which point you can pass through the invisible wall in the kitchen.
I'd imagine that Andre definitely passes through that door under his own accord, and the loud crashing noise you hear while outside of the livingroom is the sound of him being attacked in the cellar.