http://www.libretro.com/index.php/category/pgxp/
...what made the difference in terms of the N64 having more stable rendering compared to the PSX is the lack of perspective correct texturing, which the N64 does have. Calculating the coordinates so that a texture would look correct from any angle would be computationally expensive... By only applying perspective-correct sample coordinates at certain intervals, rendering could be done much quicker on the PSX, but at the cost of the warping/dancing polygons that you can see in so many PSX games. To combat these issues, youd see developers like Psygnosis using many tricks in games like Wipeout to mask these issues (for instance by subdividing textures into many parts)...
PGXP attempts to kill two birds with one stone. First, it introduces subpixel precision to get rid of the wobbling polygon issues. Second, it adds perspective correct texturing to stop the textue warping/dancing issues. iCatButler first started integrating PGXP into the emulator
PCSX-R (by injecting it into Petes OGL2 plugin, a closed-source plugin). It seems this attempt has been successful and was the first one out of the starting gates, but there are some geometry issues which seems partly can be attributed to the ageing fixed-function Petes OGL2 renderer. iCatButler now has backported this functionality to
Mednafen/Beetle PSXs GL renderer. According to iCatButler, there are less geometry issues with Mednafen/Beetle PSXs GL renderer...