You don't have friends/ family/ significant other/ children/ pets/ etc. that will give you shit for strapping a TV on your face to become more immersed in your escapist fantasies?
It's also not solely about looking silly. I already know a PSVR is going to be a source of contention in my family and I'll probably only use it on weekends or after my wife goes to bed. Completely shutting yourself off like that is so isolating. It feels weird enough playing games with headphones on when my wife is in the same room.
People who live alone won't have that problem, but people with families are going to have a harder time using VR. I have friends with really young kids who are interested in it but are resigned to not grabbing one right now because it's just not feasible to cut themselves off like that for extended periods.
I will have the same problem. Not sure I'm willing to invest the energy into convincing my S/O that VR is worth the money and sheer isolation it promotes. I will likely have to sit it out. As you mentioned, it's going to act as a point of contention. An even more pronounced one than video games already are.