I agree about the PSX but the Saturn? I also had one and it's overall far worse than the N64 performance wise if you exclude 2D games. Both Quake and Duke Nukem 3D are sub-20fps fests, Daytona USA is 20fps, the Wipeout port is 20fps (the N64 port is 30fps). And let's not even talk about Saturn DOOM vs DOOM 64. There is nothing on either PSX or Saturn that can be remotely compared to DOOM 64 visual and performance wise.
I guess, it's a matter of what games you ended up with. I get that a lot of games by RARE had unstable performance on the N64, and these games are some of the most popular. But a game in the scope of OOT, running at 20fps isn't that bad (for the time). It's not comparable to Tomb Raider either. That one is a very simple puzzle-platformer with no analog movement, no camera control, etc and very simple, boxy environments with short draw distances. It was surpassed by Mario 64 (bigger environments, proper analog 3D movement, unlimited draw distances, stable 30fps).
OOT is way more advanced and ambitious than that. And Tomb Raider also had frame rate issues, especially on the Saturn, OOT at least was locked 20fps.