Well it's totally possible that the P4 cast people will be like 'No thanks, I'm kinda burn out on this series right now'
I mean, 5 games is a LOT to be doing 1-2 characters for.
A situation like that almost certainly wouldn't arise because if an actor doesn't want to play a character, they won't audition for it. And actors like long-haul roles because that's guaranteed work for them. Troy Baker isn't not playing Kanji because he's tired of it, he presumably just has more and probably better paying offers.
The only time an actor doesn't want to reprise a role is if the working environment is unfavorable of they feel they can do no more with the character they play. The latter is pretty rare, too.