I don't think that's the case, is it?
I recall sneaking around, getting the 'Defender inbound' message and being really annoyed. I was taking my infiltration slow, progressing to the exit carefully, then having to start from the beginning.
I read it's based on the status the security team is in. If they're in caution, even without you having been spotted, the defender gets alerted and can retaliate.
That's what I read, at least. So even if you don't trigger an alert, the defender
can come to defend depending on the security team's status.
Anyway, I opened up my FOB today and wasn't invaded at all so far and I played the game for several hours (of course at one point I eventually got disconnected from the server but when I reconnected later I didn't get any notice I lost anything).
I also read that you'll almost never get invaded if you don't set your security too high (so having lowish rank soldiers, no security cams, no sensors, drones, etc.) because your base won't be very highly ranked then and the infiltrator won't be able to get much from you. You'll, at best, only get invaded by people who are also low ranked (as they can only find bases of similar ranks, another thing I read). I'm not sure if the above is completely accurate but that's how my base is and I haven't been invaded yet. I created it mainly to expand my MB (which btw just having an FOB alone increases your MB capacity pretty significantly so it's worth doing when you have the option to build one) and get more fuel resources.