-
Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.
SMI shows eye tracking and foveated rendering on GearVR. For high-end/dev market.
- Thread starter ashecitism
- Start date
Danlord
Member
I am still confident that they'll be integrating their eye-tracking technology with PlayStation VR, especially now after they've been pretty vocal this past few months with their solutions for Oculus Rift and now GearVR. They had previously worked with Sony's Magic Labs to make eye-tracking for an Infamous Second Son demo, and it was shown before we knew of PlayStation VR so they couldn't say it would be for VR integration, but the technology naturally fits. I hope we find out soon, I am so hyped for this tech inside VR.
Hah. Not happening.
Unless Sony is interested in inflating the cost of their VR solution.
at least not happening on the 1st gen of PSVR
Danlord
Member
Sony's (Magic Labs) been working with SMI to do their Infamous Second Son demo and that was at GDC in 2014. Sony could have pushed this research further with their own resources especially in their optics division to create their own, low-cost version of this eye-tracking technology and primed it enough ready for Mid-2016 launch.
Only conjecture of course.
Expect this to cost >5000. At least. Normal SMI glasses cost 20,000. Those are the ones that don't have anywhere enough Hz to do foveated rendering, so even at 5000 SMI would be severely undercutting themselves.
There's no way Sony could make this anywhere affordable.
Don't expect this in 2nd Gen hmds. 4th gen maybe.
Danlord
Member
What is in these devices that inflates the cost of the device so high?
I don't know if it's a complete false equivalence but it reminds me about VR and people thinking it'd be similarly priced and we're seeing a £499 priced Oculus Rift, compared to the projected costs of a few thousand £ for workable VR.
Danlord
Member
Apologies for bumping this in-active thread with a douple post, but I thought my question prior was relevant. Anywho, I just wanted to contribute a little information further to my point in regards to Sony using eye-tracking with PlayStation VR. It is more speculation, but it's with some merit.
Sony has been working on VR for a while. They had previously worked with SMI that produced a working demo on PS4 with Infamous Second Son. That company has now unveiled their latest product that integrates their eye-tracking tech into a VR form-factor, working with both Oculus Rift and Samsung GearVR. Just recently, we heard (via VR Focus) that Sony's Magic Labs is working on Eye-tracking (and hand-tracking) with the suggestion it is for VR, which is logical given the implications for Foveated Rendering.
Each time, Sony has been counted out in competing with VR. I remember the 1080p 120hz OLED screen reveal shocked many people, and for good reason. The PS4 announcement itself they one-stepped MS's expectations with 8GB of RAM. Now I'm thinking the same thing with eye-tracking and the price is happening. Sony has been ahead of everyone's expectations here since Day 1, and I think with VR Sony are very serious here and that they could deliver all of this for PSVR.
Sony has been working on VR for a while. They had previously worked with SMI that produced a working demo on PS4 with Infamous Second Son. That company has now unveiled their latest product that integrates their eye-tracking tech into a VR form-factor, working with both Oculus Rift and Samsung GearVR. Just recently, we heard (via VR Focus) that Sony's Magic Labs is working on Eye-tracking (and hand-tracking) with the suggestion it is for VR, which is logical given the implications for Foveated Rendering.
Each time, Sony has been counted out in competing with VR. I remember the 1080p 120hz OLED screen reveal shocked many people, and for good reason. The PS4 announcement itself they one-stepped MS's expectations with 8GB of RAM. Now I'm thinking the same thing with eye-tracking and the price is happening. Sony has been ahead of everyone's expectations here since Day 1, and I think with VR Sony are very serious here and that they could deliver all of this for PSVR.
Good question. I have no doubt their prices are heavily inflated. They're selling to businesses and universities who need it for research and can't make the things themselves, so obviously they'd pay almost whatever. The 25k Euro also gets you processing and analysis software and support, which you don't need as much as for a VR headset. But it is wellknown that SMI & Tobii >>>> everything else in terms of eyetracking on the market, so their R&D / patent portfolio and materials apparently amount to something, because others can't significantly undercut them.
But even if you estimate a very generous 5x markup (so the production cost is 5000) and remove most of the software (say 2000?), you're left with 3k euro costing glassing... that operate at 90-120 Hz, much too low specced for foveated rendering.
There's just no way it's going to be feasible within the coming 5 years.
Unless Sony outright buys them and wants to eat the cost.
serversurfer
Member
SMI also talked to Road to VR about this. I got the impression that they're saying the only reason this is so expensive is because they're building them to order for medical professionals and whatnot, but they'd be pretty affordable if they were simply being built in to lots of headsets.
FWIW, Anton Mikhailov seems to agree with this assessment. (Note his up-thumbing of my paraphrasing of SMI's answer to the question of affordability.) No, we probably shouldn't hold our collective breath, but I'd keep a close eye on GDC, since SMI also seem to indicate their solution is basically ready to ship, in one form or another. If the cameras can be made cheap enough by building them in huge lots, that would make PSVR considerably more functional and future-proof.
FWIW, Anton Mikhailov seems to agree with this assessment. (Note his up-thumbing of my paraphrasing of SMI's answer to the question of affordability.) No, we probably shouldn't hold our collective breath, but I'd keep a close eye on GDC, since SMI also seem to indicate their solution is basically ready to ship, in one form or another. If the cameras can be made cheap enough by building them in huge lots, that would make PSVR considerably more functional and future-proof.
vpance
Member
Apparently the cost is not that high, at scale, if you believe them
http://uploadvr.com/smi-hands-on-250hz-eye-tracking/
So why is no hardware manufacturer biting? Well, maybe Sony but it doesn't sound likely from yosp comments. I think it boils down to a software maturity issue. People are only just toying with foveated rendering, why bother adding even $9 to the cost of the HMD if the initial adopters will be plenty happy with 'regular' VR for now?
The company seems desperate to snag a buyer tho for 2nd gen headsets, maybe before everyone decides to figure out their own solution, which guys like Valve will most likely decide to do.
http://uploadvr.com/smi-hands-on-250hz-eye-tracking/
So why is no hardware manufacturer biting? Well, maybe Sony but it doesn't sound likely from yosp comments. I think it boils down to a software maturity issue. People are only just toying with foveated rendering, why bother adding even $9 to the cost of the HMD if the initial adopters will be plenty happy with 'regular' VR for now?
The company seems desperate to snag a buyer tho for 2nd gen headsets, maybe before everyone decides to figure out their own solution, which guys like Valve will most likely decide to do.
serversurfer
Member
I missed the bit about it only costing a few dollars a unit at scale, but that doesn't really surprise me. That being the case, I hope Sony push for it. You may well be correct about the software being immature, but that's a temporary situation at most, and PSVR is probably going to be Sony's hardware solution for 4+ years.
Most early VR development is likely to be done with UE4 or Unity 5, or possibly PhyreEngine on PS4/cross-platform. (I'm assuming the latter has VR support as well. /shrug) Foveated rendering will take a bit of doing to set up the engines, but once that's done, it shouldn't be too difficult for developers using those engines to leverage in their own games. So while the current batch of games in development may not be able to leverage foveated rendering, perhaps the sophomore batch will. A few extra bucks today ensures that 6-18 months after launch, PSVR games are able to get a pretty significant graphical bump, and make it considerably easier for devs to hit native 120 fps. Hell, they could even bump the resolution a bit, and just upscale games until the engines get foveated.
Eye tracking brings a lot of advantages beyond foveated rendering too. Targeting things with your eyes is far more natural than trying to aim at stuff with your face; we're not owls. It's also very useful for communication, allowing you to both emote and gesture to other users, and AI actors can be taught to recognize your gaze and what it may mean. Then there's less glamorous but equally important stuff, like IPD adjustment. That's an awful lot of bang for less than ten bucks, especially when you consider it'll be a few years minimum before we see PSVRv2. If they get the resolution high enough, v1 may even be worth using on PS5, even if v2 launches alongside PS5 and is crisper still.
Anyway, Shu doesn't seem to be above bluffing to throw us off the track. He seemed to think that 60 Hz was plenty fast enough for now, until they announced 120 Hz.
Sidewinder
Member
Wow single digits cost for eye tracking, hot damn, that'd seriously be worth a delay to fall 2016 for the PSVR, would ensure the headset to be more future proof and maybe even allow compatibilty with PS5...because without it, I can't imagine it being an option, as the headsets by then will be so much better in every regard. It'd make perfect sense to integrate it into a console exclusive headset, as it won't be possible to throw more power at it with a new graphics card.
Hmm really interesting stuff, but for now I'm still dreaming about a 299$ price megaton reveal at GDC, so adding eye-tracking and foveated rendering being a possibility in the near future with the PSVR would be like a dream inside a dream
And yeah I agree, Shu might've been bluffing there, he is a smart dude.
Hmm really interesting stuff, but for now I'm still dreaming about a 299$ price megaton reveal at GDC, so adding eye-tracking and foveated rendering being a possibility in the near future with the PSVR would be like a dream inside a dream
And yeah I agree, Shu might've been bluffing there, he is a smart dude.
serversurfer
Member
I'm not even sure they'd need to delay it. SMI said they'll be ready to deliver in 4-6 weeks, and I can't imagine this comes as a surprise to Sony, since they've been working together closely for years. For all we know, this is what Sony have been waiting on, and that's why we only got a vague "first half" window. Seems like this could easily be included in any headsets launching between GDC and E3, assuming this was the plan all along. SMI have probably known their packaging requirements for a while now. Plus, Magic Lab does their own eye-tracking research in addition to their cooperation with SMI, and Sony certainly know how to build at scale.Wow single digits cost for eye tracking, hot damn, that'd seriously be worth a delay to fall 2016 for the PSVR, would ensure the headset to be more future proof and maybe even allow compatibilty with PS5...because without it, I can't imagine it being an option, as the headsets by then will be so much better in every regard. It'd make perfect sense to integrate it into a console exclusive headset, as it won't be possible to throw more power at it with a new graphics card.
Hmm really interesting stuff, but for now I'm still dreaming about a 299$ price megaton reveal at GDC, so adding eye-tracking and foveated rendering being a possibility in the near future with the PSVR would be like a dream inside a dream
And yeah I agree, Shu might've been bluffing there, he is a smart dude.
So yeah, I'd say this is certainly well within the realm of possibility for Sony. It's simply a matter of what they decide to do. If they can add eye-tracking and bump the resolution to ~1440p and only increase the BOM by like $50, that seems like a pretty good move if the headset is gonna be $200, and kind of a no-brainer if it's gonna be up in the $300-$400 range. That would literally add years to the useful life of the headset. Eye-tracking alone would be a pretty big win at $10, really, even if they don't bump the resolution.
They should totally bump the resolution; a 4x efficiency increase from FR should take your game nicely from 1080p60 to 1440p120.
Neuromancer
Member
There was a Morpheus prototype a year ago or so with eye tracking built in. Chris Kohler from Wired tried it and talked about it on the Game | Life podcast (sadly I don't remember which episode.)
It would be great if that is the plan with PSVR and they're just being coy about it, but I won't get my hopes up too much.
It would be great if that is the plan with PSVR and they're just being coy about it, but I won't get my hopes up too much.
serversurfer
Member
Do you happen to recall specifically what was said WRT eye tracking? I've heard lots of speculation, primarily stemming from the SMI-powered eye-tracking that Kieghley used to play Second Son at the launch event. The only semi-concrete stuff I heard was a couple of smaller sources at the Morpheus reveal reporting there was something inside the face mask that looked like it could be a camera lens, but nobody was willing or able to tell them what it was or what it was for.
At the time, I was thinking it was probably related to their auto-IPD patent, but it could easily be used for both functions, assuming it had a good view of both eyes during use. I dunno if they'd need a second camera for that, but whatever they case, it sounds like their solution is lightweight and affordable.
They say SMI's camera(s) look at a reflection of your eyes being cast on specially treated glass, so I suspect the camera is mounted above the eyebrows, pointed towards the floor, looking at this specially treated glass. The glass is sits between your eyes and the lenses, and is angled such that it gives the camera a good view of your eye. The glass is coated with a material that allows light to pass through from the screen side, but the light being reflected from your eyes can't pass through the other direction, and is reflected up towards the camera. Even if the image picked up by the camera is somewhat ghosted, there's probably enough contrast there to figure out what's happening.
I'm not really sure if that was in the original Morpheus kit though. /shrug
VR is feasible right now, hence why we have GearVR available right now on mobile, 2 devices releasing in a matter of weeks and a console headset later this year. Foveated rendering is the future of VR, but VR is absolutely here.
Neuromancer
Member
I don't remember other than he said it was amazing. I scanned through all the available Game | Life summaries but it isn't mentioned in any of them, unfortunately, so tracking down the exact episode is difficult.
I got the feeling that he may have been breaking some kind of nda since no one else was talking about it.
Neuromancer
Member
Just got a tweet from Chris Kohler saying it was the ISS/TV demo. I could have sworn it was Morpheus but I guess I got confused. =\
I certainly remember the Infamous: Second Son eye-tracking demo. Video.
There is a lot to look forward to in 2-3 years with what the second generation of VR will bring, but that's a long time to wait. And the Vive is almost here!
I'll just have to suffer the wait for Pascal for my new build. :'(
Foveated rendering rally isn't needed until the screens suddenly become too high res for GPUs to keep up. And 14/16nm GPUs are just about to start rolling out sosell see a decent increase in performance over the coming 18 months.
If generation 2 of OR/Vive have 4K per eye, then you'd probably want to also include foveated rendering.
E-Cat
Member
Danlord
Member
To an extent, whilst the benefits of Foveated Rendering won't be as pronounced until the conditions you mention are met, I do think the benefit of even using low* resolution VR rendering with foveated rendering will still herald significant improvements to VR at an early stage. Especially with the closed-hardware of the PS4 and limited GPU budget, getting the most out of the GPU will be helpful for developers to hit their 60/90/120hz target for VR.
Also the research into using foveated rendering tech and developing new rendering techniques that take advantage of this will have matured to a point so when we're >8K screens for VR and the benefits are more pronounced and needed, we'll have a mature toolset and advanced rendering techniques to efficiently scale detail and resolution to improve performance at such high resolutions with more ease and delivering a better VR experience. That's why I'm excited to see Valve's GDC talk coming soon.
*low in the context of first-gen VR headsets that will eventually go up to 4K and 8K and something like 16K to be the best fit for VR, which in this comment is what it's intended.
serversurfer
Member
Ah, right on. I did find an article where they went to look at Morpheus, but it wasn't working, so they played with they SMI/I:SS demo and were super impressed, yeah.
Then maybe Anton's response to you meant, "I sorta doubt he saw that one…" lol
I disagree. Foveated rendering is a pretty significant win today, and as Dan points out, this is especially true for a fixed platform like PS4, where you can't ask users to simply throw more silicon at the problem, or pick up the new and improved headset in 18 months.
You're right about the technique becoming more effective at higher resolutions, but SMI say that "the current HD headsets" get a 2-4x efficiency increase from FR. So the "minimum" boost takes you right from 1080p60 to 1080p120. I don't see how that's anything but a massive win, and if your boost is up in the 3-4x range, after you've doubled your native frame rate, you can start cranking up scene and simulation complexity.
Then look at a game like Driveclub. A 2x boost puts you at 60fps with no graphical compromises, making the game a candidate for VR effectively as-is. If they do get a 4x boost, now you're talking about full-quality Driveclub running in VR at a native 120 fps. A middle-of-the-road 3x boost puts them at 90 fps native. I'd say any of those possibilities are probably worth bumping the price of the headset by $10. Also, Gran Turismo.
And again, that's just foveated rendering, which is a more or less meaningless graphical boost. Personally, I'd gladly pay an extra $15 or even $20 just for a headset that didn't force me to aim with my whole face.