-
Hey Guest. Check out your NeoGAF Wrapped 2025 results here!
MS HoloLens
- Thread starter cyberheater
- Start date
The Cowboy
Member
Hmm, hopefully this is OK to discuss here (gaming side).
But the military applications for this thing is just insane when you think about it, imagine being on a real battlefield with these headsets built in a helmet and having various tracking things all reading into the device.
You could have solders on the ground with on screen way pointing using GPS (location of targets, squad members etc), an actual full on HUD telling you ammo remaining (software built in to the gun to read ammo count), direct video feeds from other squad members or the higher ups issuing orders, real time target info being sent to you in 3d etc - or heck bomb disposal people having a real time on screen display to help diffuse bombs (much like they showed with that fitting a light switch demo).
You could basically have a full on gaming (battlefield etc) HUD system for solders.
But the military applications for this thing is just insane when you think about it, imagine being on a real battlefield with these headsets built in a helmet and having various tracking things all reading into the device.
You could have solders on the ground with on screen way pointing using GPS (location of targets, squad members etc), an actual full on HUD telling you ammo remaining (software built in to the gun to read ammo count), direct video feeds from other squad members or the higher ups issuing orders, real time target info being sent to you in 3d etc - or heck bomb disposal people having a real time on screen display to help diffuse bombs (much like they showed with that fitting a light switch demo).
You could basically have a full on gaming (battlefield etc) HUD system for solders.
No it doesn't - just the current solutions being put together require a camera and IR tracking. There is no reason you couldn't put an AR style markerless tracking system on a VR system for tracking external objects. Might even be useful to prevent you hitting objects if you're walking around in VR.
Theonik
Member
A static tracking camera like a Kinect. Still not trivial.
Alx
Member
I'm not quite sure I understand. If it's in a real-time interaction between several users, the location of the object will be handled by the game/software engine, which knows the positions of everything it has to render.
If it's about keeping track of an object between different sessions, it's basically doing a "game save" of all current object positions, and the only difficulty is when starting a new session, making sure you're using the same reference. That's only if you want the virtual objects to appear at the exact same place though, you could very well decide to continue your game on a different table at a different place.
Imagine playing Halo on you're TV/Monitor and having the Spartan HUD displayed on your hologlass. It would remove the HUD from the TV space so less clutter and more immersion. Also, imagine hit markers, ADS, waypoints, weather affects like rain and frost, shield regen, blood spatter, sneezes...
The wait is killing me.
First thing in my mind is playing Metro. Finally I can wipe out the mud on my mask with my own hand.
efyu_lemonardo
May I have a cookie?
Theonik
Member
In reference to their respective sensors. Specifically, what the system can 'see' is a series of points, which correspond to distances from its emitter.
Not really the same thing though. AR would have to know about the table the object is on, it's doesn't have to know about smaller items littering my environment that I may want to avoid ("who dropped that fork there!!!").
Catch up MS I had one of these as a kid
I'm glad I failed to convince my parents to buy this for me. They new better than I did back then.
efyu_lemonardo
May I have a cookie?
And there's no attempt to convert that data to a more absolute frame of reference?
Like something most star chart or navigation apps try to do?
edit: for example, if I'm in a house, I would maybe try to identify and locate doorways, window frames, stairs, etc. Things that are unlikely to be moved between successive sessions or changed often. Something like this could be part of every "boot up" sequence. The more information I can gather and analyze about any given environment the better I can get at storing and retrieving positional data that is consistent between users and between sessions. A lot of the heavier analysis could be done offline by a powerful home computer. This is probably closer to how our own brains do it.
It was actually ok for it's time.
It's crazy how we was able to enjoy them single game tiger handhelds with just black dots & lines as kids now babies are walking around with HD tablets with endless games at their fingertip.
Theonik
Member
Oh yeah, since we are talking about that. This is something that has been done for a fair amount of time. Oculus bought one of the providers of such a system too. (Nimble VR)
Leapmotion also has introduced a head-mounted version of their SDK for months and a mount for the Oculus Rift DK2.
https://www.leapmotion.com/product/vr
On the demos the press has shown I don't think. It will be interesting to see how MS would solve that problem or if they even bother to address this at all in the first iteration of the technology.
The military is already working on/has prototypes for tech like this. It's not news to them.
Here's one example:
http://www.thedailybeast.com/articl...lity-system-goes-from-military-to-market.html
But DOD has several projects like this.
CyberPunked
Member
Couldn't you just play the game on the HoloLens it's self? Why buy a TV/Monitor, if you could have a 90inch hologram monitor? You could still have everything you mentioned above, but less limitedImagine playing Halo on you're TV/Monitor and having the Spartan HUD displayed on your hologlass. It would remove the HUD from the TV space so less clutter and more immersion. Also, imagine hit markers, ADS, waypoints, weather affects like rain and frost, shield regen, blood spatter, sneezes...
The wait is killing me.
Anton Sugar
Member
The FoV of the hololens is probably not great for playing games fully like that. And the real world would be partially visible through the lens which would decrease image quality. This is not a VR headset.
CyberPunked
Member
True, but wouldn't further development allow for completely opaque holograms in the enviroment? If not, the oculus or other VR devices would fare far better as gaming devices as you could play games on your person VR IMax, and still have all the added HUD effexts mentioned earlier.
The Cowboy
Member
Interesting, thanks for the linky.
Well resolution is the main issue here. In this case the added HUD elements would actually be eating in more of your screen real estate than it does normally. Now a 1080p screen is drawing the standard TV view, the surrounding environment AND the HUD.
AR would just be drawing the HUD elements, whilst you'd still have a full 1080p screen for all the other game content. you'd be freeing up screen space in this case.
Anton Sugar
Member
I generally think FPS games will be better suited for VR, but I would love to play a shooter with an augmented HUD. It's a nice go-between.
killer rin
Member
Thats only if you wanted to render the entire game world to its respective point in the real world. What you do is you take the same game-render you get now and display it to FOV=0
Well, its either that or they sell a plastic shield to make the front of the glasses opaque
SystemUser
Member
I think replacing your TV is a hard AR application. MS Hololens is a first generation soft AR device. There is a link to an article explaining the difficulties with doing hard AR and the differences between hard and soft.
Found it: http://blogs.valvesoftware.com/abrash/why-you-wont-see-hard-ar-anytime-soon/
Alx
Member
It only needs to use an arbitrary reference, like any game does.
Let's consider a regular chess videogame. The game engine will define its own geometry, for example with the origin at the center of the board. When you move around the ingame camera, the engine renders the view of that camera from its position and orientation, and displays it on your TV. What appears there is what somebody would see if he was in the game world with his head at the camera's position.
Now let's imagine common AR using QR codes. If you put a card with a QR code on a table and watch it through a camera, a standard software can estimate its relative position and orientation to the center of the card.
If you decide to consider that the center of the card is the origin of your game, and use the estimated position and orientation of your real camera for the game to render what the virtual ingame camera would see, you'll get a view of the chessboard that when overlayed with the live image will line up with the card. As you move around, the virtual camera position will be updated accordingly, and you'll keep the illusion that there is a 3D chessboard on the table.
A second player would do exactly the same thing, and doesn't even need to know the position of the first player. He'll just detect the card too, ask for the game to render the chessboard from his point of view, and have it displayed accordingly.
The game itself will keep considering the virtual object positions in its own coordinates centered on the chessboard, which for the cameras are the same as "card coordinates".
Hololens does the same thing, except it's not using cards, and will track real 3D objects in the environment instead (or so I suppose). The added difficulty is first to find such objects, and second to make sure that all players are using the same objects as reference (it was easier with the card since it can be identified by its very specific pattern... on the other hand with Hololens you won't need to stick QR codes everywhere in your appartment
).
Anton Sugar
Member
I haven't heard any technical discussion, but one of the impression (RPS?) mentioned how the lack of or minimal lag is impressive.
Theonik
Member
Doing hands in Oculus is doable but you'd need some kind of tracking system for it. Current versions of the oculus use sensors for head orientation. Markers for head position in 3D space. (relative to the camera)
Currently, Oculus simply lacks hand tracking but it's not a matter of lag. It's just lacking something that can track your hands. RIGHT NOW. Which is also not true.
There are 2 major devices available today that let you do just that in Oculus.
Nible VR which is a time of flight camera system (like Kinect) *
Leapmotion (normally for table hand tracking but got Oculus support several months ago)
You can buy the later RIGHT NOW with a mount for the rift.
*Oculus bought them and will probably be on the Consumer Rift if not the second version.
With regards to lag, I haven't seen any response with regards to that for the MS Holo thing.
Pretty much every impression has mentioned that the lack of lag is very impressive.
Iced_Eagle
Member
I believe The Verge and Gizmodo mentioned how impressively little lag there was.
I think one of the benefits is that the entire system is one integrated piece (and their HPU is likely a dedicated chip to do all of the sensor processing as fast as possible), and it can be highly optimized in terms of latency since it's a single known device with known hardware. Plus, MS writes the OS, and created all of the hardware, and they have a lot of experience and learnings to take from Kinect in terms of latency of gestures and depth tracking (Kinect is impressive if you consider that it can track 6 entire skeletons, including joint rotations at ~60ms -- trim that down to one person's hands with a limited set of gestures to recognize and they can probably get it going really fast).
Holo at me Bro!
Everything that MS learned with Kinect, which has to be a hell of a lot in terms of sensor technology and the software needed to accurately process that data to determine what people are doing, is in HoloLens.
Kinect 3.0 is pretty much a critical component of Hololens as far as I can see. A heavily evolved version of the chip in Kinect 2.0 has to be the 'HPU' in Hololens.
Kinect 3.0 is pretty much a critical component of Hololens as far as I can see. A heavily evolved version of the chip in Kinect 2.0 has to be the 'HPU' in Hololens.
You could also have enemies spawn in training.
I await the day we can larp ragnaros.
I hope they have the option to buy a dev kit like OR and someone pushes Hololens to see how much it can do. I don't see any point in projecting things miles away anyway.
Alx
Member
Yes, and as the 2010 roadmap suggested, a device designed for outside will probably require data connexions like 4G, GPS etc. Besides they'll probably want to have a cleaner design for people to wear it in the street.
We'll need to wait for the second or third generation...
Michanical
Member
Current AR technology has limitations. There will be opacity issues in bright lights. Microsoft is doing their test events in controlled environments, so they can control how much light a room has, but projecting images onto glass has a physics problem when light is competing from behind the glass.
When we see an advancement in AR technology that either uses a different types of materials, different projections techniques, or something entirely new, then I can't imagine AR having any significant downside relative to VR, especially if AR sets have a way of "isolating" the player from exterior, immersion breaking distractions.
Mister Zimbu
Member
My first instinct with the HoloLens is to write something that autodetects wires or other objects on a floor and hides them so people will trip over stuff.
efyu_lemonardo
May I have a cookie?
It only needs to use an arbitrary reference, like any game does.
Let's consider a regular chess videogame. The game engine will define its own geometry, for example with the origin at the center of the board. When you move around the ingame camera, the engine renders the view of that camera from its position and orientation, and displays it on your TV. What appears there is what somebody would see if he was in the game world with his head at the camera's position.
Now let's imagine common AR using QR codes. If you put a card with a QR code on a table and watch it through a camera, a standard software can estimate its relative position and orientation to the center of the card.
If you decide to consider that the center of the card is the origin of your game, and use the estimated position and orientation of your real camera for the game to render what the virtual ingame camera would see, you'll get a view of the chessboard that when overlayed with the live image will line up with the card. As you move around, the virtual camera position will be updated accordingly, and you'll keep the illusion that there is a 3D chessboard on the table.
A second player would do exactly the same thing, and doesn't even need to know the position of the first player. He'll just detect the card too, ask for the game to render the chessboard from his point of view, and have it displayed accordingly.
The game itself will keep considering the virtual object positions in its own coordinates centered on the chessboard, which for the cameras are the same as "card coordinates".
Hololens does the same thing, except it's not using cards, and will track real 3D objects in the environment instead (or so I suppose). The added difficulty is first to find such objects, and second to make sure that all players are using the same objects as reference(it was easier with the card since it can be identified by its very specific pattern... on the other hand with Hololens you won't need to stick QR codes everywhere in your appartment
Your last paragraph is exactly what I was referring to. Creating positional data that is independent of the viewer and consistent with its surrounding environment. Exactly as you described, a QR code has the ability to "inform" any observer of its existence and provide additional (absolute) data that allows the virtual object to be consistently reconstructed.
Kinect 1 (and probably 2) used something similar to what many cameras have had since decades, an ordered grid with a very specific and meticulously calibrated pattern which could be projected on the environment (using a frequency slightly outside the human visual spectrum) in a manner that is reproducible with high accuracy between sessions (as long as the camera is stationary).
The challenge Hololens has to solve is much bigger.
Alx
Member
In the case of Hololens, the QR code will be replaced by any recognizable object (or set of objects) in the environment. Just like a regular camera will be able to recognize and track the pattern of a QR code, the depth cameras will be able to recognize and track a shape / depth pattern (for example, all planar objects in the scene when the user initialized the tracking, and especially their 3D edges and salient points).
To have other viewers align to the same coordinates, it will be necessary to match the objects they see to those used as a reference by Player1, which can be done manually or automatically (it's quite similar to the problem of creating a panoramic picture actually, trying to match the overlapping parts of two different points of view). But it only has to be done once per session : once all users have decided of common coordinates ("let's use that corner of the table as the origin !") they just need to do their own tracking.
efyu_lemonardo
May I have a cookie?
I think we're pretty much on the same page with regards to what needs to happen on a technical level for this to work. The thing I find most surprising is that other attempts to do this accurately and quickly seem to have been noticeably less successful than whatever method MS is employing, hence every article mentioning how stable and lag-free the experience is. That along with the final concept version of the product shown being quite small and portable caused me to wonder if maybe they were doing something more than 3D environment feature extraction.
Like I said I have no professional experience in this field, so perhaps my impression that this is a big improvement over what is currently available is incorrect.
Interestingly enough, Microsoft's done some serious work for this type of algorithm with Photosynth, which can automatically take a bunch of images, do feature extraction, match them, generate point clouds and then re-project the original images...
https://photosynth.net/preview/about
If they have a simple black LCD coating on the outer glass, it should be possible to make the visor reasonably opaque for VR-style environments.
Also, I'm seeing very little talk about the actual image projection tech. I remember reading about such technique a while ago: layered RGB sheets where the light bounces inside and exits in controlled positions, but this is the first time I see it in practice.
Also, I'm seeing very little talk about the actual image projection tech. I remember reading about such technique a while ago: layered RGB sheets where the light bounces inside and exits in controlled positions, but this is the first time I see it in practice.
I will call this kinect lens because basically they took a regular pare of glasses and taped it to a kinect. I bet if I did the same thing, I could view some crazy shit in AR mode. Reminds me of how the Rift is just a tablet with ordinary lenses taped to it. Funny how inventions are that simple. Bet I could create an uber Wii remote by taping some camera shits to it.