1) For certain PS3 PSN titles this is not true. If PSN goes down, you can't play at all.
2) For other titles, this is only true for the existing piece of hardware. You can't move it to another piece of hardware.
3) Sony has already shut down PSN servers, making games unplayable on anything but the hardware they happened to be installed on at the time.
1) The only time that is true that I'm aware of for the most part (on PS3/PS4/PSP/Vita) is online games.
In fact, even if a single-player offline game is removed from PSN the game can still be played if it's already downloaded, even if it's never been launched, and there are apparently I think there are some cases where you can re-download from your download list even if the game is removed, though not all like P.T, which can't be re-downloaded.
Another example: People who bought and downloaded the digital version of Brave Story, a JRPG for PSP, are still able to play the game (and even on their Vita) even though it had to be removed from the store due to licensing issues, probably due to the developer going out of business. There are conflicting reports about whether people who bought it can re-download this specific game from the download list or if it's like P.T, but there is no check on the game ever, so it will continue to work indefinitely as long as the system works regardless of PSN even if PSN were to not exist.
For offline games, the PS4 just needs to be set as "primary". Once you do this, you do not need to be online to play offline games you've bought on PSN. PS+ games are the only ones that stop working, and only when you unsubscribe.
2&3) Agreed, the one big shortcoming is that you have to be able to download the game from the store in order put it on a new system if yours dies since you can't save the game installer to a disk to install later to another machine like you can with a PC DRM-free copy from GOG/Humble Bundle.
Some people understandably wait for sales or discounts due to this shortcoming.