Support NeoGAF

[Mabase]

Very nice writeup, thanks Krej!

Some questions for the intitated:

1. So if I understand correctly, the lighthouse base stations projects a spatial pattern of Infrared light/lasers across the room. Two of these are used to balance out occlusion that happens through objects, people etc. in the projection's way.

2. The sensors on the tracked headset, controllers etc. are what exactly? Infrared filtered CCDs, or complete cameras incl. lenses? Or something else? I suppose it's something like a CCD if it's supposed to be lightweight, cheap and numerous.

3. So what are the drawbacks of Lighthouse? Do I understand correctly that

- only works inside, to a maximum size of room
- reflective surfaces, like mirrors, or other IR sources could throw off tracking (include a calibration process to amend for this?)
- occlusion of the base stations, like a person standing directly in front of it, could throw off tracking

This is all very exciting. Forget about VR for a second: just having a reliable, acurate and cheap motion and position tracking device that's plug and play is a great benefit of all of that.

 

[Liabe Brave]

The idea is so simple and elegant it makes you scratch your head wondering why it wasn't done before. Between the Wiimote, PSMove & Kinect you would think someone would stumble over this combo earlier. Or maybe it was just tech holding us back from realizing the concept earlier?

The Wii mote is the same idea. This just paints the IR all over the place, instead of using one bar of LEDs.

Move is outside-in probably because of legacy tech--it was based on the Eyetoy camera. Kinect is outside-in because it has different goals. For example, it can see hand gestures and body movements that are invisible to this Valve tech (without strapping even more cameras to your body).

The Lighthouse tech is a good approach, but it isn't the absolute end state. Despite what the OP says, it's still subject to occlusion. That's the reason the demo units are mounted to the ceiling. If you put it on, say, the shelf of your TV stand, then the screen, your speakers, and your living room furniture will all occlude the laser spray. Positioning it right won't be hard, but will be a factor. A sunny room might be too; it depends on the laser used, but sunlight has the potential to wash out IR markers.

 

[Mabase]

The Wii mote is the same idea. This just paints the IR all over the place, instead of using one bar of LEDs.

not quite. The Wii Bar is "just" two IR points being tracked by the built-in Wiimote camera. The Wii Bar does not project/paint onto walls etc. Notice how it only tracks the Wii cursor when you point the Wiimote at the bar.

 

[Seanspeed]

That is where the redirected movement comes into play. Just because you turned 45 degrees or move 2 ft in the game doesn't mean that's how far you really moved in real life. The game can stretch or shorten those movements a little bit to change your position and orientation. You'll never be going in a straight line. You'll be circling around the center even though you won't realize it.

Nah, redirected movement is for like continuous moving, namely the idea of going straight, which obviously presents problems in limited spaces. Its not going to apply to side stepping, or single steps whichever direction. These steps will add up and put you up against the walls soon enough. There's no 'correcting' for that unless there's like ultra smart AI that knows to throw obstacles at you that make you dodge the other way(which actually sounds kinda brilliant, but runs under the assumption you are successful!).

And regardless of how far you move in the game, you've still moved 'x' distance in real life, probably at least a foot or two. There's only so much distance it can help you out with before it becomes a bit ridiculous and unbelievable(or nauseating).

Hey I'd love to try it too. I've never been in VR so all of this is theory to me. I'm just going off of what others have said is possible and putting it all together. For example, one test I'd love to try is to see how well such a system would work fighting a huge ogre swinging a club at you. You'd have to sidestep his vertical strikes and duck his horizontal ones while trying to strafe around him to strike with your sword. I wonder just how much space that would take.

You would need to figure out how much space that would take over many, repeated attempts. And is that it? Is that the whole game? What about the next area or whatever?

Give me some padded walls or some sort of safe room to try it and it would probably be a blast. I just really cant imagine this is something people will be doing in their living rooms apart from shorter, small scale demos.

 

[onQ123]

not quite. The Wii Bar is "just" two IR points being tracked by the built-in Wiimote camera. The Wii Bar does not project/paint onto walls etc. Notice how it only tracks the Wii cursor when you point the Wiimote at the bar.

I'm guessing that he/she was just saying that the lights being projected everywhere are being used just like the lights in the sensor bar while the IR sensors of the VR headset & controllers is using the lights as the sensor in the Wiimote would.

 

[AllIsOneIsNone]

Don't have time to read through this whole thread at the moment, there's a ton of info, but I did skim through and see the redirected walking thing which is crazy. This relies on a wireless system though, correct? Is that doable in the near future?

 

[show me your skeleton]

this is a great thread, thank you! this, of all things, has me more interested in VR than any of the announcements haha. i guess i like the tech. more than the usage.
apologies if you answered this already but do you work in a particular field of study that relates to this?

 

[Zaptruder]

Hey Krej.

Apparently Valve's solution doesn't use cameras.

They're IR light sensitive diodes. Not cameras.

The tech doesn't work like you've described in your write up.

I thought it did too, but I was proven wrong with a tweet from Valve D:

https://twitter.com/vk2zay/status/572676775915872256


They haven't released details of how it works yet...

But from what people can figure; the base station actually sweeps the room with the lasers. When it hits a diode, the diode reports the time that it was activated. As it hits the other diodes, the pattern of activation reports the orientation of the unit.

The light towers and the various units are synchronized so that the angle of the laser and the time when the diode was activated can be figured out.

What's worth noting is, what's bounced off the walls doesn't matter. Only what directly hits the diode.


Which makes a lot more sense. Diodes are cheap as shit. Cameras are cheap, but not quite 16 cameras per trackable unit cheap.

I think ultimately inside-out tracking solutions will be more robust and... less fugly. The mushroom controllers that they showed necessitates that shape and size - the diodes need to be a certain distance apart to get sufficient gaps between detection.

Lighthouse is better characterized as... angular timing detection method of tracking.

 

[Crispy75]

That's very clever.

So not inside-out tracking at all: Just a very very accurate outside-in, but therefore just as susceptible to occlusion.

 

[Dash Kappei]

Great op cooljerk

 

[ASIS]

Is this the right place to say <3 motion controls? Cause I really really do.

Thanks OP for the write up, very cool stuff.

 

[ashecitism]

Might as well post it here too

Was posted in the other thread, but Tested put up a vid and in the beginning they talk with Ken Birdwell about the tracking.

https://www.youtube.com/watch?v=leg2gS6ShZw

 

[Mabase]

Might as well post it here too

Was posted in the other thread, but Tested put up a vid and in the beginning they talk with Ken Birdwell about the tracking.

https://www.youtube.com/watch?v=leg2gS6ShZw

This is amazing and even better than I originally thought.

 

[SpaceDrake]

The only ways to do that is to either rely on a person actually turning, or relying on galvanic vestibular stimulation - attaching small electrodes to the back of your ear and shocking your brain into making you think you are spinning. I am a huge proponent of VR, and even I won't try galvanic vestibular stimulation.

Aww, c'mon, Krej, don't be a wuss! Electrocute yourself! For science.

But seriously, this is a fantastic thread and I linked to it over my Twitter. Thanks for explaining this so virtually everyone can understand!

And just to spur on discussion - there's tons of "middle-ground" applications for this technology as well, right? Like there's no reason you couldn't be sitting and just control it like an FPS, with your head controlling camera tilt, but it still recognizes your body in a general sense? Or it can scale up to full walking around.

I need to schedule a demo for myself (yay, living a few hours from Valve HQ :v: ). This is cool stuff.

 

[Zaptruder]

That's very clever.

So not inside-out tracking at all: Just a very very accurate outside-in, but therefore just as susceptible to occlusion.

There are 2 light houses. So... still 'subject to occlusion', but not under normal circumstances. (i.e. you have to have 2 jerk friends covering the base stations for it to stop working).

 

[JaggedSac]

Hey Krej.

Apparently Valve's solution doesn't use cameras.

They're IR light sensitive diodes. Not cameras.

The tech doesn't work like you've described in your write up.

I thought it did too, but I was proven wrong with a tweet from Valve D:

https://twitter.com/vk2zay/status/572676775915872256


They haven't released details of how it works yet...

But from what people can figure; the base station actually sweeps the room with the lasers. When it hits a diode, the diode reports the time that it was activated. As it hits the other diodes, the pattern of activation reports the orientation of the unit.

The light towers and the various units are synchronized so that the angle of the laser and the time when the diode was activated can be figured out.

What's worth noting is, what's bounced off the walls doesn't matter. Only what directly hits the diode.


Which makes a lot more sense. Diodes are cheap as shit. Cameras are cheap, but not quite 16 cameras per trackable unit cheap.

I think ultimately inside-out tracking solutions will be more robust and... less fugly. The mushroom controllers that they showed necessitates that shape and size - the diodes need to be a certain distance apart to get sufficient gaps between detection.

Lighthouse is better characterized as... angular timing detection method of tracking.

Thanks for the info.

 

[dumbo]

But from what people can figure; the base station actually sweeps the room with the lasers. When it hits a diode, the diode reports the time that it was activated. As it hits the other diodes, the pattern of activation reports the orientation of the unit.

The light towers and the various units are synchronized so that the angle of the laser and the time when the diode was activated can be figured out.

Yep - that certainly seems to be what they describe.

If they knew the distance of the headset to the lighthouse, it would be relatively easy to deduce the position and orientation of the headset... (and doesn't require 'speed of light' calculations).

But I'm not sure how they know the distance of the headset to the lighthouse without "speed of light calculations".

 

[CTLance]

Ooh, what a clever idea.

Not quite sure how they avoid interference from reflections from walls, shiny surfaces and nearby items with a "stupid" diode as a sensor, but if that's how hey do it it's definitely cheap as shit. Awesome.

Also, thanks for the thread. A delightful read, no matter what.

 

[Waveset]

Fucking great OP and subsequent responses, thank you.

Getting excited about VR as an end user is one thing but then to read devs responses and passion for (what I see as) small components of the whole tech is really inspiring.

VR's been a great ride in terms of reading about the problems that Oculus have been discovering and then the solutions that different companies are coming up with - not only considering physical hardware but also psychological factors and the tricks the brain plays with visual and auditory information.

 

[Zaptruder]

But I'm not sure how they know the distance of the headset to the lighthouse without "speed of light calculations".

They absolutely do speed of light calculations. And apparently it's computationally trivial.

So the pattern of activations and the difference in time it takes to activate them gives the unit all the information it needs to figure out its absolute position relative to the base stations.

 

[Crispy75]

There are 2 light houses. So... still 'subject to occlusion', but not under normal circumstances. (i.e. you have to have 2 jerk friends covering the base stations for it to stop working).

Or just turn 180 and hold the controllers against your chest. You'd need at least 3, maybe 4 base stations to get good coverage.

Ooh, what a clever idea.

Not quite sure how they avoid interference from reflections from walls, shiny surfaces and nearby items with a "stupid" diode as a sensor, but if that's how hey do it it's definitely cheap as shit. Awesome.

Also, thanks for the thread. A delightful read, no matter what.

If the laser is reflecting off a shiny surface and hitting a sensor, then the sensor's location will appear to be far away from others on the same device and can be automatically ignored.

 

[call_kotaku]

Wait, so it doesn't work as described in the OP?

https://www.youtube.com/watch?v=leg2gS6ShZw&feature=player_detailpage#t=48@50s the Valve engineers describe it more as a LIDAR system with lasers, and it seems like lasers just scan the room and the receiver pattern on the headset calculates the position based on when they receive the signal from the base station.

If so, how does this solve for Occlusion? By having 2 laser sources?

 

[bj00rn_]

[snip]

Damn, good stuff..

And thanks for an interesting thread krejlooc (and anyone else who contributed). Bookmarked..

 

[Crispy75]

Wait, so it doesn't work as described in the OP?

Correct. It's still much better than traditional outside-in setups, because of the absolute knowledge of which tracking point is which (compared to having to work out which blob is which when viewing an image of IR dots)

https://www.youtube.com/watch?v=leg2gS6ShZw&feature=player_detailpage#t=48@50s the Valve engineers describe it more as a LIDAR system with lasers, and it seems like lasers just scan the room and the receiver pattern on the headset calculates the position based on when they receive the signal from the base station.

If so, how does this solve for Occlusion? By having 2 laser sources?

Correct. It's not solved as such, just improved. The absolute knowledge of tracking points is helpful here too, because even just one visible IR sensor is enough to help keep track of the location during occlusions, if there's inertial sensors as well.

 

[mrklaw]

Or just turn 180 and hold the controllers against your chest. You'd need at least 3, maybe 4 base stations to get good coverage.



If the laser is reflecting off a shiny surface and hitting a sensor, then the sensor's location will appear to be far away from others on the same device and can be automatically ignored.

Aren't the base stations at opposite corners of the bounding box? So turning 180 you'd be facing the other base station.

Occlusion still sounds like a potential issue as the scanning laser would need to hit x number of diodes to get a decent signal. but it can probably manage a temporary loss of signal using the other sensors to mitigate that.

Sounds like a mini GPS setup

 

[call_kotaku]

Correct. It's still much better than traditional outside-in setups, because of the absolute knowledge of which tracking point is which (compared to having to work out which blob is which when viewing an image of IR dots)


Correct. It's not solved as such, just improved. The absolute knowledge of tracking points is helpful here too, because even just one visible IR sensor is enough to help keep track of the location, if there's inertial sensors as well.

I really wish there was a picture of the setup they actually used for the demos. I assume there's a laser station at opposite corners of the ceiling and the headset diodes can see atleast partially behind your head.

But this still means that wireless devices can use the system flawlessly, correct?

 

[Blanquito]

Enter valve - they had a really good idea. Rather than painting your walls blue, why not shine a blue light in the room? The blue light in the room would turn all your walls blue for you, without needing for you to manually paint your walls every time you wanted to play your game. Valve's Lighthouse positional tracking base does just this. Essentially, it works like this:

Except invisibly. The fundamental problem of logistics for inside-out positional tracking is now solved. This has been the holy grail of tracking for many years now, and valve has deftly solved it.

In a real-world situation, isn't this still subject to occlusion? Let's say I have a couple of couches on two sides of that light-emitting thing in the middle of the room. The couches would block the light from hitting large sections of the walls behind them (depending on how close the couches are, for example).

So what happens if I'm looking at those walls that don't have light on them? Even worse (just for an example), what happens if I'm behind the couch and looking at the wall that doesn't have any light on it? Would I be completely lost to tracking then?

In the end, I think it still comes down to a problem of being impractical in certain common real-life situations, just like outside-in tracking doesn't work in certain common real-life situations.

 

[mrklaw]

nice tested video, watching now.

The fake wall that comes up sounds fun, but I hope there aren't bugs in the tracking that report the boundaries incorrectly. Made me think of Die Hard 2

https://www.youtube.com/watch?v=QkCQ_-Id8zI

 

[Graphics Horse]

Correct. It's still much better than traditional outside-in setups, because of the absolute knowledge of which tracking point is which (compared to having to work out which blob is which when viewing an image of IR dots)


Correct. It's not solved as such, just improved. The absolute knowledge of tracking points is helpful here too, because even just one visible IR sensor is enough to help keep track of the location during occlusions, if there's inertial sensors as well.

I wonder if mirrors are a problem, maybe you can configure it to not aim at or ignore certain directions. It looks like they must have a lot of redundancy built in to the design.

 

[Crispy75]

I wonder if mirrors are a problem, maybe you can configure it to not aim at or ignore certain directions. It looks like they must have a lot of redundancy built in to the design.

mirrors will create reports of sensor locations very far away from the core group of sensors. It will be very easy to ignore.

See how the reflection of the obscured sensor appears to be greatly separated from its neighbours? That's a physically impossible situation, so the software will ignore it. Only if a majority of sensors are visible in the same mirror would a false location be reported.

mirrors *are* a problem when you're detecting a dumb projected pattern with a camera. This system avoids that problem because the markers are smart.

 

[Mr.Green]

Wait, so it doesn't work as described in the OP?

No. Not at all.

 

[Foggy]

My word, if redirected walking becomes predictable/replicable/practical enough to become consumer friendly(or if it already is) I'm gonna shit.

 

[Graphics Horse]

mirrors will create reports of sensor locations very far away from the core group of sensors. It will be very easy to ignore.

See how the reflection of the obscured sensor appears to be greatly separated from its neighbours? That's a physically impossible situation, so the software will ignore it. Only if a majority of sensors are visible in the same mirror would a false location be reported.

mirrors *are* a problem when you're detecting a dumb projected pattern with a camera. This system avoids that problem because the markers are smart.

Sorry I forgot to reply earlier, thanks for the answer! So as long as there's enough sensors getting hit by the unit it should work fine.
I wonder how the base stations work out their own position, do they have to be able to see each other too?

 

[Raticus79]

Oh wow, so it's kind of outside-in after all (although quite a different approach to it). Well, that makes the details here make way more sense.

The 15x15 restriction seemed small for the wall-painting approach, but there's a good reason for it with scanning lasers. The further away you are, the more space there is between the arcs the laser traces. So, as you get further away, the beams would need to be making narrower and narrower passes to avoid missing an increasingly-distant sensor, which means it would take longer to do one full scan of the room. If it takes too long then the system gets too slow about updating your position, so there's a max distance you can be at to get a reasonable update frequency. They can't make the lasers too bright or they'd no longer be class 1, and they can't scan the laser over the sensor too quickly or it won't register enough of a hit. So, there's where the max distance comes from.

Good thread regardless. It'll be good to reboot this one once the details are nailed down.

Edit: Here's the new thread:
http://www.neogaf.com/forum/showthread.php?p=154826197

 

[Nafai1123]

In a real-world situation, isn't this still subject to occlusion? Let's say I have a couple of couches on two sides of that light-emitting thing in the middle of the room. The couches would block the light from hitting large sections of the walls behind them (depending on how close the couches are, for example).

So what happens if I'm looking at those walls that don't have light on them? Even worse (just for an example), what happens if I'm behind the couch and looking at the wall that doesn't have any light on it? Would I be completely lost to tracking then?

In the end, I think it still comes down to a problem of being impractical in certain common real-life situations, just like outside-in tracking doesn't work in certain common real-life situations.

The lighthouses are placed on the wall, so you should never have to worry about it being in the middle of the room and some object occluding it. In addition, they are using at least 2 lighthouses (at 90 degrees to each other on different walls) so you should have good coverage of the entire space. It's not like you will want a couch or something in the middle of this setup anyways. You need a clear space to move around in.

 

[hesido]

Very exciting. We'll have Star Trek VR levels of awesome in 15 years (full body tracking inside an empty room with ultra realistic graphics.) The redirected walking fixes the finite room problem very cleverly! You can have an infinite virtual world in a sufficiently big and empty room.

 

[-Ryn]

Very cool read. Thanks for posting dude.

 

[Blanquito]

Is the only difference between inside-out and outside-in tracking the chance that one offers better protection against occlusion? Or are there other advantages/disadvantages to having inside out vs outside in, such as computational effort or things like that?

 

[Crispy75]

Is the only difference between inside-out and outside-in tracking the chance that one offers better protection against occlusion? Or are there other advantages/disadvantages to having inside out vs outside in, such as computational effort or things like that?

Outside-in means you can have one camera tracking multiple objects. Inside-out requires each object has a camera, so it makes the equipment more expensive.

Valve's tracking is a hybrid. The sensors are on the objects, but they're very dumb sensors.

 

[Blanquito]

Outside-in means you can have one camera tracking multiple objects. Inside-out requires each object has a camera, so it makes the equipment more expensive.

Valve's tracking is a hybrid. The sensors are on the objects, but they're very dumb sensors.

I hadn't thought of that, thanks.

Does that mean that it would take more processing power to have 3 sensors tracking locations (e.g. HMD, two wands) vs one sensor tracking multiple objects? Or would they be about equivalent?