• Hey, guest user. Hope you're enjoying NeoGAF! Have you considered registering for an account? Come join us and add your take to the daily discourse.
  • Hey Guest. Check out the NeoGAF 2.2 Update Thread for details on our new Giphy integration and other new features.

Sony GDC eyetracking tech demo [using eyes to aim inside Infamous Second Son]

FrunkQ

Neo Member
Jul 26, 2013
197
0
0
Scotland
I can see limited use of this within the VR headset being useful, but it's use outside being even more useful than that. i.e. an AR setup. Use the AR to see the real world (perhaps even just your TV screen) and the eye tracking cameras are nice and close to your eyes to do their magic like we saw in the video.

Then bring the AR angle properly into the equation and supplement the view of your TV with AR - giving you screen a 3D overlay and providing illumiroom type stuff to - to fill out your vision and then all the eye tracking gubbins too on top of that... my brain just melted with the possibilities.

On second thoughts ignore everything I said - if they combined all this lot in a horror game with monsters hiding behind your own furniture, hearts would just fail!
 

Dre_G_Writer

Member
Dec 7, 2013
2,154
0
0
GOT DAYUM SONY!!!


give a dude some time to calm down and take a breather. Dropping so many bombs, they gonna give me a heart attack.

Man, they are going all out this gen.
 

Ran rp

Member
Sep 12, 2011
12,201
0
750
Gimmie


 

Drencrom

Member
Aug 17, 2011
13,146
2
660
If could not only change the rendering of the game [which would be invisible to the user], but could change how gameplay elements are made:

- pretty girl smiling when you look her in the eyes
- NPC gets annoyed at you if he notices that you are looking at his face scar too much
- crazy Nazi torturer seeing which of your NPC friends you wish NOT to die
- out of 3 manequines in the ruined asylum only those that you are not looking at are opening their eyes
- using your Superman eye lazers to burn enemies
- marking things with your eyes [HUD]
- solving puzzles with your eye gaze


As DemonNite said, with eyetracking "center of the screen" can finally be decoupled as the main way to select or interact with things.

These are things I want from nextgen
 

gofreak

GAF's Bob Woodward
Jun 8, 2004
43,345
2
1,645
How this going to track our eye inside VR headset, using IR?

Yeah, SCEJ actually did a glasses-mounted prototype for eyetracking a little while back with IR.

http://www.ijsei.com/papers/ijsei-10812-13.pdf

I think in an ideal implementation, you could do fantastic things with eyetracking in a HMD. But I'm not sure the latency performance is there yet. Have no idea, but they'd be extremely tight to do the really cool things anyway. Maybe it's good enough now to help augment aiming, like say, using eye tracking as well as head tracking to aim in EvE.
 

hydruxo

Member
Nov 2, 2013
13,040
1
395
Promising but I can't help but to think it would strain your eyes after a while and you'd just want to go back to using analog sticks. I don't know. It's hard to say without trying it out myself.
 

Rogue_Ledr

Banned
Jan 25, 2010
7,081
0
0
IF they can merge this tech into Morpheus, it would blow Oculus away.

It would be more Oculus than Oculus!
 

Marc

Member
Aug 14, 2013
3,948
0
370
London, England
Wow, looks brilliant, usually it takes some getting used to a new technology but he is using it with ease.

I think I would prefer to control the camera with the right analogue and have the targeting reticule follow my eyes. Would find that less intrusive and more natural so your eyes are limited to the screen.


EDIT: Ooooh, just thought that strategy game would finally be very natural on consoles if this were incorporated,
 

gofreak

GAF's Bob Woodward
Jun 8, 2004
43,345
2
1,645
Caveats aside in my last post, though, that does look amazing. Incredibly intuitive vs dual analog.
 

Wray

Member
Jan 22, 2008
2,148
0
0
Player 1: Why do you keep aiming at her tits???

Player 2: Uh...I'm not. The controllers broke dude.
 

rjinaz

Member
Feb 7, 2012
17,934
1
0
37
Promising but I can't help but to think it would strain your eyes after a while and you'd just want to go back to using analog sticks. I don't know. It's hard to say without trying it out myself.

I don't see how it would strain your eyes. You're not doing anything extra, other than what your eyes would already be doing one way or the other when playing a game. The camera is what is doing the work, all you're doing is looking around.
 

Tabby

Member
Aug 18, 2013
7,597
0
510
Player 1: Why do you keep aiming at her tits???

Player 2: Uh...I'm not. The controllers broke dude.
Pretty much this.

That's impressive though, opens up so many opportunities.
If they include this in Morpheus, I don't think I'll be able to handle it...
 

OryoN

Member
Aug 10, 2004
1,941
0
0
But how? Not sarcasm, I'm really trying to wrap my head around how this could do anything inside a VR headset.

I thought I was the only one that noticed. People want VR because they agree it's a more interactive experience... Yet they think auto-targeting wherever you look is a feature that complements that whole experience?

The video was nice too look at from a spectator's perspective, and I understand it's impossible to convey - via youtube - how this would work in VR without actual hands-on, but, I am certain that I would not enjoy THAT implementation (gameplay wise) in a VR game, which basically auto-aims wherever I look. What a way to kill the whole interactive VR experience!

Now give me VR, very limited eye-tracking(to simulate field of vision), with head-tracking, and Wiimote Plus/Move-style gun-controller, and drop me in the back alleys of a stunningly rendered Racoon City, and we're on to something special! :D

We'll get there, eventually...
 

majik13

Member
Mar 5, 2012
6,242
0
0
so is it really tracking where you are looking or just where your eye balls are in space? At first I thought the former(such tech does exists) but it seems based on how he has to move his head around it looks to just be tracking how the eyes balls are moving with the head. Not necasarilly how the eyes rotate/look in their sockets. Which isnt nearly as impressive or useful.

While it is still useful on its own, it doesnt seem much different than say track IR except with the added benefit on not having to wear any tracking device.
 

majik13

Member
Mar 5, 2012
6,242
0
0
I thought I was the only one that noticed. People want VR because they agree it's a more interactive experience... Yet they think auto-targeting wherever you look is a feature that complements that whole experience?

The video was nice too look at from a spectator's perspective, and I understand it's impossible to convey - via youtube - how this would work in VR without actual hands-on, but, I am certain that I would not enjoy THAT implementation (gameplay wise) in a VR game, which basically auto-aims wherever I look. What a way to kill the whole interactive VR experience!

Now give me VR, very limited eye-tracking(to simulate field of vision), with head-tracking, and Wiimote Plus/Move-style gun-controller, and drop me in the back alleys of a stunningly rendered Racoon City, and we're on to something special! :D

We'll get there, eventually...

I think there is a couple things here. Based on the video it is not necessarily tracking where you are looking exactly, just how your eyes in relation to your head are moving. Basically if you kept you head perfectly still, and looked around by rotating just your eyes, it wouldnt do anything or work.

It seems this same idea/tech could be replicated in VR by tieing your head rotations to control the camera. And to me that seems to be what will make VR useful for third person games like this, basically as a tool for perfect, or at least much more intuitive camera control. And could free up the second analog stick for some other function.

How I see it, its basically tracking your eye location in relation to each other(2 point track) to determine gaze, the same way the PS camera determines orientation of the Morphoues based on the blue LEDs on its outside . It doesn't seem to know where your eyes are actually looking independently though.
 
Aug 19, 2006
25,598
1
0
so is it really tracking where you are looking or just where your eye balls are in space? At first I thought the former(such tech does exists) but it seems based on how he has to move his head around it looks to just be tracking how the eyes balls are moving with the head. Not necasarilly how the eyes rotate/look in their sockets. Which isnt nearly as impressive or useful.

While it is still useful on its own, it doesnt seem much different than say track IR except with the added benefit on not having to wear any tracking device.

Good question. Both techs exist but the way it's presented here looks like where your eyes are rather than your iris is. I'd have to try it for myself but what they show here doesn't look amazingly useful, as you've got to be looking to the side of the screen to rotate the camera, which isn't what you'd want to be doing in most situations. There are some great uses for eye tracking, like he says the attacks go where you're looking, that part is good, but I'm not sure that camera control should be put on top of that.
 

sono

Member
Jan 20, 2013
3,695
78
675
If they can get the precision of this technology down that is a big deal.

You could put more processing power into the area of the screen the player is looking at, our eye doesnt take in all 1920*1080 pixels in the same level of detail at any point in time.
 
Aug 19, 2006
25,598
1
0
You find looking at things tiring?

What they're showing here is something that could cause your character to fall off a ledge simply because your eyes were pointing in slightly the wrong direction, it's possible that the amount of concentration you need to place on your eye movements isn't the most relaxing thing.
 

majik13

Member
Mar 5, 2012
6,242
0
0
Eye tracking is the future of VR - it just makes too much sense.
* Foveated rendering, saving order(s) of magnitude in rendering power
* Eye-relative motion blur, removing stroboscopic artifacts ("1000Hz-feel")
* "real" depth of field
etc.

agreed, though I dont think this tech is necessarily tracking or detecting where you are actually looking.

I hadnt necessarily considered how eye tracking could benefit VR like you have, except for your last point. Ive always thought that stereo 3d wont really take off or be universally comfortable for people until they could really simulate true depth of field or eye focus. So that depending on where you are actually looking then you would get accurate focus change. Look at something close in the FG and the BG goes out of focus and vice versa, like with how are eyes work with real world depth perception.

Many people get queasy and headaches from stereo 3D because of things like this missing. Focus doesn't change when your brain thinks it should.
 
Aug 19, 2006
25,598
1
0
I hadnt necessarily considered how eye tracking could benefit VR like you have, except for your last point. Ive always thought that stereo 3d wont really take off or be universally comfortable for people until they could really simulate true depth of field or eye focus. So that depending on where you are actually looking then you would get accurate focus change. Look at something close in the FG and the BG goes out of focus and vice versa, like with how are eyes work with real world depth perception.

Many people get queasy and headaches from stereo 3D because of things like this missing. Focus doesn't change when your brain thinks it should.

It's a long way off but should happen eventually.
http://www.youtube.com/watch?v=deI1IzbveEQ
 

majik13

Member
Mar 5, 2012
6,242
0
0
It's a long way off but should happen eventually.
http://www.youtube.com/watch?v=deI1IzbveEQ

nice, hadnt even considered using light feild to solve this problem, so does the implementation of how the light feild display is physically set up(prism like) just automatically control focus/defocus accuratly or does it simulate it by tracking your eyes? i would guess the former, but I am not famliar with light feild displays, I was only aware of the cameras and the semi interactive pictures.
 

SolidusDave

Member
Apr 14, 2009
10,169
0
0
Germany
The potential is immense (better immersion+ new gameplay options + saving rendering power), but if it's not miniaturized and/or built into the Morpheus prototype yet, I don't have hopes that we will see it in the actual release, unless that's even later than 2015 which would also suck :p
 
Aug 19, 2006
25,598
1
0
nice, hadnt even considered using light feild to solve this problem, so does the implementation of how the light feild display is physically set up(prism like) just automatically control focus/defocus accuratly or does it simulate it by tracking your eyes? i would guess the former, but I am not famliar with light feild displays, I was only aware of the cameras and the semi interactive pictures.

It's physically set up to look right, so flexing your lens muscles will make it look different without the display changing, like a hologram. It probably needs many more images than they're using there to look perfect, I've not tried it myself.
 

Cuburt

Member
Nov 14, 2012
6,298
0
0
That's actually pretty cool. Still the applications of the tech can't be used for every game but it already would be a step towards making a VR helmet a viable input device instead of mostly just a display device for traditional style games.

Too bad for Occulus Rift that Sony is throwing so much support behind this already, Sony will probably beat them to the punch with more infrastructure and support to get it on the ground running before the Occulus Rift even comes out the gate.

I still have my skepticism that VR will be as huge as people think but at this rate, we'll at least be able to see the tech capitalize on some of that potential even if it's just a modest success.
 
Aug 27, 2007
32,911
0
0
Too bad for Occulus Rift that Sony is throwing so much support behind this already, Sony will probably beat them to the punch with more infrastructure and support to get it on the ground running before the Occulus Rift even comes out the gate.

I really don't think Sony sees it this way. OR and Morpheus are mostly targeting different demographics (platforms aside) but they can both benefit from increased exposure on both ends. There are already plenty of developers aware of and interested in Oculus Rift development and just like Sony's stance with bringing indies on board PS4, I figure they see OR's success as an opportunity for more devs/games on PS4 VR. They're not really in direct competition and I don't think Sony's in any hurry to add PC support and eat their piece of the pie.
 

majik13

Member
Mar 5, 2012
6,242
0
0
It's physically set up to look right, so flexing your lens muscles will make it look different without the display changing, like a hologram. It probably needs many more images than they're using there to look perfect, I've not tried it myself.

nice, defintly sounds like a good direction for the the tech to go, wouldnt it also better mimic accurate light exposure(HDR) based on where you are looking?
 
Aug 19, 2006
25,598
1
0
nice, defintly sounds like a good direction for the the tech to go, wouldnt it also better mimic accurate light exposure(HDR) based on where you are looking?

I'm not sure about that, it would be interesting to see, could be applied to this Sony demo too. Ideally the display would get as bright as the sun and you could just let your eye handle it naturally again ;)
 

raebodep

Member
Jan 27, 2011
529
0
0
If could not only change the rendering of the game [which would be invisible to the user], but could change how gameplay elements are made:

- pretty girl smiling when you look her in the eyes
- NPC gets annoyed at you if he notices that you are looking at his face scar too much
- crazy Nazi torturer seeing which of your NPC friends you wish NOT to die
- out of 3 manequines in the ruined asylum only those that you are not looking at are opening their eyes
- using your Superman eye lazers to burn enemies
- marking things with your eyes [HUD]
- solving puzzles with your eye gaze


As DemonNite said, with eyetracking "center of the screen" can finally be decoupled as the main way to select or interact with things.

Perfect for a trailer park boys game
http://m.youtube.com/watch?v=r7FcHGsDJVE
 

dose

Member
Jul 15, 2004
3,065
0
0
The eye tracking is just the reticule, you can look at whatever you want. When in combat, your reticule would ideally be on enemies, it's not the time to be sightseeing.

.....They just demoed him looking at the Space Needle?
Yep, when he wasn't actually firing. I'm talking about when you're in combat. Say some object catches your eye or you want to glance at your energy or something else. You don't want your fire to always track where you are looking.
 

quetz67

Banned
Jun 24, 2004
5,795
0
0


On rails Robocop VR shooter with eye targeting. I´ll buy that for a Dollar!

I think that would be a nice gimmick, but to me that is the only use for eyetracking in gaming. Technology is old anyway, but surprisingly nobody who is not handicapped would control their mouse pointer in Windows with their eyes.

To move the view direction on screen to where the eyes look is totally counterintuitive as your eyes are on that spot anyway. So you move your eyes to your targeted area, then back to center then same again and so on.

For VR - OK, there might be some more gimmicky uses with the game knowing where you looking. But again that is counter-reality, as in real life you neither can move your gun with your eyes nor would the world around you usually react (or even know) on where you are looking.
 

cakefoo

Member
Jun 24, 2006
5,264
0
1,130
usa
This was awesome. I can't believe how natural it looks. It fits perfectly with the superhero genre.

One issue occurred to me though. I'm shooting a guy on the ground and an enemy pops out of a window, and I want to keep shooting at the guy on the ground while I identify what type of enemy is up top. What then?
 

vpance

Member
Jun 20, 2005
8,655
2,876
1,650
I think the express purpose of this tech is solely for Morpheus, which is exciting.

Sony's already said they're looking into eye tracking for VR, but the fact that they've gone out of their way to look at 3rd party solutions (SMI) and now a Sony engineer saying it could be miniaturized and used inside Morpheus tells me there's a good chance it'll make it into the retail product. They wouldn't bother to research and demo it to this extent otherwise. It's not like they'd release another camera with this tech to sit alongside the existing one. And every accessory so far has been created with the idea of eventually incorporating it into the entire VR experience.
 

hesido

Member
May 23, 2013
4,625
0
0
Turkey
www.hesido.com
The VR application would be nice:

Focus / Defocus according to where you are looking at, simulating the natural DOF of the eye.. AAAND,
The VR setup could be further calibrated to detect convergence / divergence of the eyes, which would solve one of the most important problems of 3D and VR, This way you wouldn't have the convergence / divergence race of the eye when looking at things close by, which causes a lot of strain on the eyes and causes headaches.

I'm prone to the convergence / divergence problem a lot, this would be awesome.
 

ZehDon

Member
Jun 13, 2013
3,294
4,345
700
Australia
Fuck me, Sony just keep the big shots coming. If they could put this inside of Morpheus, that's a game changer inside of a game changer. A "change-ception", if you will.