That still doesn't quite explain it - for it to manifest in one game but not the other it would mean bandwidth contention with something else is a problem - not usage from FSR alone.
But when base-rendering completes the frame 5ms early (and frames are not CPU limited either) - for FSR to spend all that time to bring it back to the cost of native, it would be contending with... what exactly?
It seems more like a type of API issue that creates hard-sync points where there shouldn't be any, in some scenarios. Might be misuse of the API on that particular game - or driver problem, but it doesn't have the signs of a direct hardware bottleneck as such.