Limitations of VR - a virtual reality check

Sep 12, 2013
15,859
0
485
#1
So I came across this post from a dev whose opinion I regard pretty highly and felt like I was hit by a ton of bricks. The post is buried in an E3 Morpheus thread, so I felt it deserved a dedicated thread to gain more visibility. Would like more experts to weigh in on the real world limitations of VR so that all of us can keep our expectations in check.

Feeling pretty disillusioned right now. Brace yourselves :/

Let me give an actual, in the real world example of how limiting VR can be - the VR application we are making right now for gear VR - we are using a note 4 currently. In stress tests, outside of a VR application, we were able to push about 500k polygons in about 400 draw calls a second at an acceptable frame rate. Trying to stress test inside of VR, however? For one, we had to eliminate environmental shadowing and reflections entirely, because those post effects were too latent. All of our texture sizes needed to be extremely reduced - we wound up using 128 x 128 8-bit textures. We had to constantly micromanage unity's garbage collector just to get the thing to run without running out of memory.

In the end, how much did we have to work with? We had about 20k polygons a second and about 40 draw calls to work with.

Extrapolate that performance difference to other hardware, because it's applicable. Virtual reality isn't a simple task to achieve at all, it's not merely "dialing things down," it's not something trivial to pull off. Every demo sony has shown off has been extremely well designed to hide all the very real, very obvious short comings. This isn't simply the PS4, either, it affects all VR devices. A Vive headset on a titan X SLI setup isn't going to look like modern-gen gaming. I see people left and right saying "I'll be fine with PS4 games running at PS3 specs." What does that even mean? There are numerous things the PS3 did which will not be feasible in VR without a massive increase in power behind the hardware that the PS4 has. They point to things like the shark demo:



And you look at it and start asking "what exactly is going on in this scene"? There is nearly no lighting, there is maybe 10k polygons going on screen at once. The entire thing takes place in a blue foggy void. So people push the luge demo - a demo which is built in a world where they can very aggressively cull everything around you to make it run faster because it goes down linear paths. No real lighting, no advanced shader calls. It's all primitive stuff.

What's left, people ask. Well, stuff like Luckey's Tale? The Mario 64-esq platformer for the rift? I don't expect the PS4 to be able to pull it off, for all the reasons I put forth above. I don't doubt there will eventually be a PS4 VR platformer, probably from MM, but it won't be anything like Mario 64. Anything with a true sense of freedom - a complex world to interact with more than a room at a time - these kind of experiences will not be possible. And it's not just sour grapes.

I'll take it back even further - Half Life 2 VR? The game we work on? It stresses my PC like hell. Our lead modeler, Jazz, is constantly redesigning things like the gun models to remove additional polygons to get it running acceptably. This is a game from 11 years ago, and it can barely run in VR with a ton of reworking. VR is so hard to work with that people honestly would be surprised what little power you actually have left over once you begin designing your game.

But let's keep going. So with the limited amount of calls I'm making, just how much script execution time do I have? VR development feels almost like retro console development in that you must carefully manage your remaining execution time down to the milisecond in order to keep things running at an acceptable framerate. With everything I said I did to reduce complexity, I still only had about 1.5 ms of script execution time to work with. Thats 1.5 ms to do everything I could possibly need to do to actually run my game. All my AI pathfinding execution, all my hardware polling. Things like audio mixing, logic updates... everything in 1.5 ms of execution time.

Again, this is extremely limiting.

And before people jump in with "but but but optimization!" This is already AFTER batching had been done, to a ridiculous level. This was AFTER we were already using multithreaded rendering. This was AFTER we were already disabling android performance throttling. In other words, we were already optimizing.
I am discussing the reality of VR development, something many people don't want to hear. I see lots of DBZ power-level like development talk in these threads and it's grating. It's not as simple as just "turning down the graphics." This is going to be a real big change in game design, because as developers many will be going from an era with virtually unlimited resources to do whatever they could dream of, back to an era where your creativity and design is again restrained by the hardware.

And, again, I'm not just talking about morpheus. I'm talking about all VR for the near future. But, specifically with regards to morpheus, there will be experiences a very high end PC can pull off that the PS4 cannot.

I said earlier in this thread, what will re-usher in that era of unbridled design decision back into VR will be foveated rendering and a rolling asynchronous time warp display. We are already starting to move towards the type of rendering pipelines that will enable foveated rendering in the future. For those unfamiliar with foveated rendering - it's a technique to enable us to mimic more accurately how our vision actually works. We don't see with clarity but save for a very tiny area in the center of our vision. This area - about the size of a pin-head - is where our fovea is centered. Extending from that point outword to the extents of our vision, we get progressively blurrier. Most of our vision and what we see is our brains filling in the gaps with the limited amount of extremely blurry visual data we are getting from our eyes.

By contrast, the view ports we use in VR maintain clarity throughout the entire area. We render the extents of our view ports at the same clarity as the center. This is because we cannot tell which area of the view port our eye is actually looking at. Once we get extremely low latency eye tracking down, we can track our eyes in the headset and figure out which area of the view port we need to be clear. We can render that in normal resolution, then render the rest of the scene in multiple passes at, say, reduced resolutions and levels of detail. This would massively speed up our ability to render scenes in VR.

Beyond that, rolling asynchronous displays will turn our displays from entirely progressively updated screens to something more resembling rasterline displays of the past, where entire vertical bands of resolution will be independently and constantly updating. In essence, we would stop updating the display in frames, and start updating in blobs of up-to-date visual data many times a second. Again, this more closely resembles how our eyes actually operate, and, most importantly, it would decrease rendering latency considerably.

Both of those advancements are still many years away, however. For the time being, we just have to live with the hardware limitations.
 
Jun 28, 2014
1,283
0
0
#2
He's right that people shouldn't expect too much but having had time with Oculus at various parties over the years, people don't NEED wowing freedom or intense visual experiences to have their minds absolutely blown. Guided roller coaster type experiences, and pacman with monkeys chasing you - I've never seen reactions like it. You see someone playing these things, and you KNOW its a paradigm shift. Devs will have to work hard to make things look good, and get creative and stylistic, but I don't feel too bummed about the software prospects.

The limiting factor IMO is the fact a lot of these things are launching as add-ons, and they're not going to be cheap.
 
May 20, 2010
1,352
0
555
Paris, France
#3
As I said in the other thread, we already have this shark demo and the Heist to judge morpheus and they look great, so why should we worry ?

You don't need top graphics to make great games anyway. Limitations will boost creativity, which can only lead to better games (see the older games if you have any doubt that).
 

cyberheater

PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 PS4 Xbone PS4 PS4
Mar 10, 2005
19,061
1
0
#4
I get compelling VR experiences using my Google cardboard and my iPhone5. I'm sure the PS4 will be able to pull of something a lot better then that.

And devs will develop techniques and presentation styles that will work well within the performance budgets.
 
Jun 9, 2004
5,126
0
0
Quebec, canada
#5
I already said in the thread that i agree with him on all aspects.

But then when you see Half life 2 in VR, even with DK1, it becomes more impressive than any top tier visuals, because the immersion and scale are just unbelievable (if you have not experienced it already).

So Half life 2 visuals or a little better for low persistence and immersion? Im ok with that. Its just that peoples are expecting way too much i think.
 

RoboPlato

I'd be in the dick
Oct 29, 2006
42,985
0
0
#7
What's this about HL2 running poorly on a 980 in VR? Haven't people been running it on significantly lower powered rigs relatively easily for a while now?
 
Oct 1, 2006
48,836
1
0
peter.metaclassofnil.com
#8
I said it from the start, AMD and NV must be salivating at the prospect of at least another decade of truly meaningful, significant performance upgrades that everyone will notice.

What's this about HL2 running poorly on a 980 in VR? Haven't people been running it on significantly lower powered rigs relatively easily for a while now?
It ran well on my 770 back on DK1.
 
Jun 4, 2011
30,337
2
0
#10
It's funny. I am a gamer and have no interest in using this for the kinds of games I like to play anyway. However, the experiences outside of gaming actually has made me curious. Being on Mars, vacations and that sort of thing. If I tried it and it works as advertised for these types of experiences, then sure, I would buy one.
 
Aug 6, 2011
5,969
0
570
#12
So I came across this post from a dev whose opinion I regard pretty highly and felt like I was hit by a ton of bricks. The post is buried in an E3 Morpheus thread, so I felt it deserved a dedicated thread to gain more visibility. Would like more experts to weigh in on the real world limitations of VR so that all of us can keep our expectations in check.

Feeling pretty disillusioned right now. Brace yourselves :/
What is VR doing other than multiple renders? I've seen people running HL2 on Dev Kits just fine with room to spare.
 

Pie and Beans

Look for me on the local news, I'll be the guy arrested for trying to burn down a Nintendo exec's house.
Apr 23, 2010
13,053
0
730
#16
Seems like a reality check for mobile and console VR, which should always have been obvious though.

Things like Colosse should be what those tier devs look towards rather than trying to half-ass photo-realism:
 
Oct 1, 2006
48,836
1
0
peter.metaclassofnil.com
#17
What is VR doing other than multiple renders? I've seen people running HL2 on Dev Kits just fine with room to spare.
The biggest difference is really 11ms frametimes maximum, always, no exceptions (ever).

That's vastly different from how most games run on consoles, and how most people run their PC games.

Edit: Oh, and all those deep pipelines you use to get better parallel scaling? Forget about those, too much latency.
 

iceatcs

Junior Member
Jul 9, 2007
10,615
0
860
London
#19
I don't expect it will be new way to play any game. I'm still expecting it will be like Wii library where most gameplay are made for limitation of wiimote.

PC, Nvidia, AMD seem want it to play a new way, that's why it require very high specification that can play any game can be VR.
 
Oct 26, 2007
9,042
0
905
PA
#20
I have a feeling that VR will give me motion sickness, I've become more prone to it as I've gotten older

Clearly I'm biased but I really don't think VR will be as big as many think, at least in the short term.
 
Aug 6, 2011
5,969
0
570
#21
The biggest difference is really 11ms frametimes maximum, always, no exceptions (ever).

That's vastly different from how most games run on consoles, and how most people run their PC games.

Edit: Oh, and all those deep pipelines you use to get better parallel scaling? Forget about those, too much latency.
See, THAT makes sense.
 
Jun 9, 2004
5,126
0
0
Quebec, canada
#22
Why? He isn't Sony.

Ark and No Man's Sky are both coming to Morpheus. Both open world games.
Ah yeah, that Sony magic ;)

You can make any games compatible with VR, after downgrades, unless they were really not pushing the hardware in the first place.

The biggest difference is really 11ms frametimes maximum, always, no exceptions (ever).
Thats actually what im fearing with Morpheus's unveilling or the games developed for it, because maybe they just dont care. You can give a VR experience, but it might be a shitty one and hurt the consumer perception of the tech.
 

boutrosinit

Street Fighter IV World Champion
Jun 11, 2004
4,422
12
1,520
39
L.A
#23
Does Morpheus have processor hardware in the visor itself?

Even if not, yes, VR is tough. But VR's special sauce is presence. Not simply the act of being in a 3D environment, but having a psychological buy-in to it as well.

Impossible to write something one can empathise with in text. You need to experience it. The best example of it out there is Valve's HTC VIVE.

The second best example; Crescent Bay.

To be clear, "best" is a shorthand metric for "What was able to make me psychologically tricked to believe I was in a real environment, even when the art was low quality".
 
Nov 23, 2011
1,876
0
490
England, United Kingdom
#24
I already said in the thread that i agree with him on all aspects.

But then when you see Half life 2 in VR, even with DK1, it becomes more impressive than any top tier visuals, because the immersion and scale are just unbelievable (if you have not experienced it already).

So Half life 2 visuals or a little better for low persistence and immersion? Im ok with that. Its just that peoples are expecting way too much i think.
I played Half Life 2 with a DK2, man the opening to that game in VR is incredible. What got me was the woman holding up to the chain-link fence at the beginning of the train station. That was a sense of true immersion for me. Walking around, looking at other people just idling was mesmerising, in a weird way; you'll obviously know exactly what I mean since you've played with it but people who haven't won't fully understand.


As for the VR argument, I don't want to go against what actual developers of VR/Games are commenting they obviously do know more than me, with direct first hand experience in developing, but I feel really conflicted when games like Project Cars will support Project Morpheus and No Man's Sky almost definitely is supporting Morpheus, and both them games whilst achieving different aesthetics, are doing some really intensive stuff; I just don't understand how there's such an drastic technical overhead with a VR application.
Sony from the beginning developed the PS4 with VR in mind, we know that from the camera choice, the controller's lightbar and oft-hand comment by Yoshida I think it was when they revealed the Morpheus as well as PS4 hardware choices with the amount of ROPs and such, Sony planned well in advanced to deliver VR.
 

Pie and Beans

Look for me on the local news, I'll be the guy arrested for trying to burn down a Nintendo exec's house.
Apr 23, 2010
13,053
0
730
#25
I have a feeling that VR will give me motion sickness, I've become more prone to it as I've gotten older

Clearly I'm biased but I really don't think VR will be as big as many think, at least in the short term.
VR is actually completely different to the normal videogame causes of motion sickness because its using your natural vision cues and higher framerate to mitigate such issues.

A lot of motion sickness is caused by a disconnection between how you view the world naturally and strange mechanical camera controls. Not so when you're turning your head and its being tracked flawlessly.
 
Jul 31, 2009
8,943
0
0
#27
This is a first step. These VR devices like phones and tablets will improve each year or every few years as technology becomes cheaper. I'm not expecting Crysis graphics in the Vive in the first revision but I think we'll get there with subsequent revisions. Alternatively, what if it pushes graphics card manufacturers to put out incredibly powerful cards, moving technology forward at a faster pace?
 
Sep 10, 2009
29,419
0
0
#28
Don't forget about this:



Essentially, just because Morpheus has a 1920x1080 display doesn't mean it wont actually need to render at something like 2560x1440 internally.
 
Oct 1, 2006
48,836
1
0
peter.metaclassofnil.com
#29
See, THAT makes sense.
That said, I'm not as pessimistic about high-end graphics in VR as Krej is. Between low-level APIs, engines optimized for it, and good driver-level and hardware-level support I think a lot will be possible.

I am definetly expecting some of these supposed VR versions like ARK to be quietly cancelled eventually.
Me too.
 
Oct 23, 2009
23,428
0
0
France
#30
I don't expect it will be new way to play any game. I'm still expecting it will be like Wii library where most gameplay are made for limitation of wiimote.
I may be the only one but that kind of talk is actually making me more interested in VR than the usual talk of "teh FUTURE" or some crap like that.
Very interesting thread.
 
Nov 1, 2006
11,062
5
910
39
#32
Everything will be OK. Programs will be tailored to the hardware and while not as pretty as non-vr games they will still be mindblowing because of the feeling. VR is so... Different. I've gotten hours of joy out of some of the most simple demos. Racing games work very well also. Try not to focus on what can't be done and instead focus on some of the attainable things.


Killer thread title too :p
 
Nov 13, 2013
8,099
0
0
Germany
#36
Seems like a reality check for mobile and console VR, which should always have been obvious though.

Things like Colosse should be what those tier devs look towards rather than trying to half-ass photo-realism:
This looks amazing! I got to read up on that, I assume it's a VR title since you mentioned it in the context which gets me even more excited.

I am definetly expecting some of these supposed VR versions like ARK to be quietly cancelled eventually.
Yep, the oculus support is just horribly tacked on right now... I think many developers and consumers think they could simply make their game a compelling VR experience by enabling the corresponding plugin in the used game engine
 

Feep

Second-hand Citizen
Sep 14, 2006
14,847
0
0
Los Angeles, CA
www.playiridium.com
#39
Asynchronous time-warping is a very effective technology to assuage the occasional frame drop, IMO. With all respect to Krej and the magic he's pulling on the Gear VR, I've seen and even developed enough content on PS4-like specs to know some amazing things are possible.
 
May 27, 2014
25,437
5
0
#40
Further, reprojection isnt magic nor universal, it has severe limitations. Reprojection really only works for yaw, pitch, and rotation because of where it occurs in the rendering pipeline, which is at at the very end after culling has been done. This means positional reprojection is not very applicable. I have seen demos from oculus themselves trying positional asynchronous timewarp, and, due to the nature of the process, what you end up with is one of two things - either you can see where the geometry ends (I.e. You look at a box that is supposed to be 6 sided, but it has culled the polygons from the back, so when you lean over and look at it from another angle, you only see 3 or so polygons that remain until the scene rerenders), or, using a visual fudging, you can see edges and seems of the geometry warp and skew, which looks a lot like the swimming effect from affine texture mapping, only occurring to the actual geometry.

The process is limited in utility.
 
May 29, 2012
2,003
0
0
#41
I see many people talking about No Man's Sky in Morpheus discussions but I am unable to find any announcement that it will be compatible with Morpheus.

I have to doubt that it will be possible on the PS4 but would love to be proven wrong. Obviously the game is an amazing candidate for VR support but I I don't think you can assume that NMS will be a Morpheus game.
 
Feb 15, 2014
9,027
1
0
Vancouver, BC
#42
You don't need photorealistic games or GTA5 running in VR to understand the potential of it all. A PS4 or a phone are both significantly weaker than a high-end PC, and both can do compelling VR experiences right now. The next few years will see mid-range hardware being able to do crazy things with VR, photorealism comes later.
 
Sep 26, 2008
6,784
0
0
#43
I think it's more a matter of tempering expectations of graphical fidelity more than anything else. There are experiences that will be tailored to their respective VR devices, and they'll use every trick they can to pull off the immersion they want to get. Meanwhile, I was giddy as a kid on Christmas morning playing Skyrim with a DK1 with a shitty driver-wrapper. It's really going to be the epitome of YMMV.
 
Sep 10, 2009
29,419
0
0
#44
I see many people talking about No Man's Sky in Morpheus discussions but I am unable to find any announcement that it will be compatible with Morpheus.

I have to doubt that it will be possible on the PS4 but would love to be proven wrong. Obviously the game is an amazing candidate for VR support but I I don't think you can assume that NMS will be a Morpheus game.
You cant assume it, but I would be very surprised if they didn't find a way to make it work. I think Kreijlooc's post brings up some good points, but he also probably shouldn't assume that what he cant do with an old, existing game is going to apply to everything else, particularly with console API's, future advancements like DX12/Vulkan, GPU manufacturer involvement, etc.
 
Nov 23, 2011
1,876
0
490
England, United Kingdom
#46
I see many people talking about No Man's Sky in Morpheus discussions but I am unable to find any announcement that it will be compatible with Morpheus.

I have to doubt that it will be possible on the PS4 but would love to be proven wrong. Obviously the game is an amazing candidate for VR support but I I don't think you can assume that NMS will be a Morpheus game.
Regarding No Man's Sky, I've self-quoted myself a few times because I posted a bunch of stuff all together and figured it's worth repeating in different threads, it's definitely not official for Morpheus but I don't see why it'd change now.

Anyways, here's the self-quote;

Hello Games was going to do a GDC talk with No Man's Sky in VR but cancelled it, and they have a Morpheus prototype/devkit in house.

Here's an article with some Vine's of them messing around with it: http://vrfocus.com/archives/1906/hello-games-teases-oculus-rift-support-mans-sky/

Image of their desk at Hello Games which includes a Morpheus Devit, source here


Sean Murray tweet about cancelling GDC talk about VR with No Man's Sky.
https://twitter.com/NoMansSky/status/446408671795609600
 
May 27, 2014
25,437
5
0
#48
Asynchronous time-warping is a very effective technology to assuage the occasional frame drop, IMO. With all respect to Krej and the magic he's pulling on the Gear VR, I've seen and even developed enough content on PS4-like specs to know some amazing things are possible.
Of course amazing things are possible, but within context and expectations. Everything gets muddled when talking to laymen because there are constant trade offs between hardware and software, but everyone expects neat linesr comparions. The oft repeated "just dial it back to ps3 levels" as an example.

Take the thread this was taken from, the retort someone gave was "so why do you think this is going to make for bad vr?" I dont. I said I think the smartly designed applications built around these limitations will be fun and fine. That we shouldnt pretend that current hardware - all hardware, not just the ps4 - isnt going to influence vr design. I think people underplay to what extent our current hardware will drive vr application design
 
Feb 24, 2008
9,648
0
0
#50
It's good a reality check but...

Haven't people already played games like HL2, Doom3, Alien Isolation or Elite Dangerous with the DK2? And fine? With their current hardware, without need of TitanX SLI?
edit: though of course the rest and the framerate increases from DK2 to CV1... mmm