I've been wanting to make this thread for some time, because of all the discussions about motion controls and several frequent misconceptions about how they work, what they can and cannot do. With the latest discussions, rumors and leaks, I think the time is right.
My original intent was to focus on kinect, but while we're at it let's cover them all.
Before we start, I'll ask you to please refrain from posting your favorite sarcastic gif, or detail how you hate/love motion gaming or ads or whatever. There are many other threads for that, this one is only for discussing how things work.

Disclaimer
- Many may have noticed that I'm a "kinect enthusiast", but I have no professional interest in all of that. I don't work for MS, Sony, Nintendo, or even in the gaming business.
- I'll try to do my best to share what I know, but although I have some experience in computer vision, there are many things I'm less familiar with. Feel free to correct me or share you own knowledge (I wouldn't have the time to do it all by myself anyway ). I'll update the OP with links to the most interesting and complete posts.

If you don't have the patience, you can also follow Lik's link to many interesting videos.

So, let's list them all, and get started :

Kinect
Basics and how it tracks bodies (thx Randdalf). Other details by sub_o.
Basics (voice processing)
Strength and Weakness
Truth and misconceptions

PSEye

Move (PSEye & Wand)
Basics : here (Alx) and here (sub_o).
Strength and Weakness

Eyetoy

Wiimote

Wiimote & Motion+

"Kinect 2" (rumor)

"PSEye 2 / Dual Camera & Move 2 / New DualShock" (rumor)

Kinect - the Basics



As an introduction, I'll need to talk about stereovision. That's how we humans estimate depth, using two eyes. Because there is a space between them, each eye sees a slightly different image. And that difference will depend on the distance to the observed objects : the closer the object, the more difference between both images.



So to measure depth with two cameras, you'll have to find what pixel of the left image goes to what pixel on the right image, and deduce the distance to the associated object from the space between those pixels. That task is not always an easy one, since sometimes there is no way to find that match for sure. Imagine that your pair of cameras is watching a huge blue wall, the left and the right images will be all blue, so there's no way to know if the wall is close or far, or even if it is a plane or a huge sphere or any kind of surface.



Enters the kinect. Its purpose is to do exactly what is described in the previous paragraph, only its inventors found a smart way to make sure that there will be recognizable features to match : they project them in the scene, as an IR laser pattern. Like the one you can see below.

Click to expand...

So even on a flat an uniform surface, we now have things that are easy to detect. A side-effect of projecting that 2D pattern is that you won't need two cameras any more : the projector replaces the right camera, and although it doesn't "see" the dots, it "knows" where it is projecting them.
So the matching between a dot sent by the right projector and the pixel associated to said dot seen by the left camera gives the same disparity measurement as in stereovision.



The result data is an image where (almost) each pixel value is a depth measurement. That depth map is then sent to the console, that will detect different parts of the body in it, and then fit a skeleton model to estimate the user's location and pose.

(more on that later)

Kinect : Strength and Weakness

Strength

• depth measurement is obviously the major innovation of kinect, which allows for a finer and more reliable measurement of the user's position than with standard cameras.
• the included projector makes the sensor rather robust to lighting conditions, which is a common problem for camera-based technology (although other lighting problems can occur, see below).
• the software developed by Microsoft allows for automatic detection and initialization of body tracking. Even the libraries developed by Primesense require the user to calibrate the detection.

Weakness

• because the technology is based on IR, it won't work if powerful sources of infrared light are in the scene. Unfortunately, the sun is a big one. So are halogen lamps.
• some materials and uneven surfaces won't reflect the IR pattern, or distort it too much for depth estimation : leather, hair, fur, ... the Cookie Monster would probably be invisible to the kinect.
• because the depth sensor has a narrow field of view (57°), it may require the user to be quite far away for his entire body to fit in the image. That's why there are big space requirements for kinect games tracking both hands and feet.
• latency : since the data goes through several loops before being usable by the software (depth measurement, transfer through USB, body part identification, skeleton matching, tracking), there is a perceptible latency between an action and the associated motion on screen.

Kinect : truths and misconceptions

It doesn't work in the dark.
FALSE (but also true). As described in the Basics post, kinect uses its own IR projector to measure depth. So that feature doesn't require ambient light, and actually works better in the dark.
But some specific features like face detection and tracking use the regular RGB image, and thus require good lighting conditions.

It only works if you have a big empty space.
FALSE. The sensor is able to measure depth for objects from 0.5m to 4m. But the software can only track a full body if is entirely visible by the camera, and since the field of view is quite narrow, that means that for full body tracking (hands and feet), the player must be 2m away from the camera.

It cannot track black people.
TRUE/FALSE ? Kinect technology is based on infrared light, and dark skins absorb more infrareds than paler ones. I lack in-depth info on that subject, but I noticed that while depth measurement will still work on a big range of skin colors, in a few cases of very dark skin the sensor can have more difficulties. I don't know how frequent it may be, but the unlucky ones may want to wear long-sleeved shirts if
the kinect cannot see them.

It cannot track fingers.
TRUE. Tracking fingers needs two things : a good software for finger tracking, and a good view on the fingers. As far as we know, the first one isn't ready yet. And even if it was, it is very difficult to get a good enough image of the user hands. The main reason isn't resolution actually, although it's limited to 320x240 for bandwidth restrictions, it could be enough if the user's hands were close to the sensor. The main difficulty comes from the self-shadowing of fingers in most hand positions, that creates incomplete depth data.

Kinect is bought technology.
FALSE. One year before the kinect announcement, Microsoft bought 3DV, and Israeli company that was selling low-cost 3D cameras, and later Canesta, an American company with similar technology.
But the sensor inside kinect is provided by Primesense, another Israeli company that is still independent.
The software for body tracking and voice processing has been developed by Microsoft Research., and has proven more advanced than the libraries that Primesense itself has been developing.

It cannot track people sitting down.
TRUE at first, then FALSE. The first versions of the tracking software were only meant to track standing people indeed. The libraries have been updated since, and it is now possible to track the upper body of a player sitting down.

Microsoft removed a processor that did all the processing to reduce costs.
TRUE/FALSE ? Before its official release, rumors mentioned that all the body tracking processing was supposed to be embedded in the peripheral itself, but that Microsoft removed it to reduce its price, and let the console do the hard work. We have no precise reference to that processor, and no hard proof of its existence. Primesense was advertising such a processor on its website, but none of the products they sent to market seem to include said chip. Also considering how their own libraries were not able to initiate body tracking without user calibration, my own opinion is that if such a chip was considered, it wasn't good enough for what MS wanted to do.

LukasTaves said:

AFAIK, Ms wanted a dedicated processor on kinect for doing the skeletal tracking. That processor was never introduced to the device, even during test phase were they used a computer to do the heavy lifting, but it does include two processors.

The one from primesense, is included but was never supposed to handle the tracking, it converts the image captured from the depth camera into a depth image, and IIRC, can do some basic recognition separating the depth data of up to 4 people to serve as a base for the tracking. And one from marvell who handles video conversion, audio processing, and registration (aligning the depth and RGB streams).

The actual tracking, that was supposed to handle by the custom processor ended up running on 360.

Click to expand...

Kinect can detect body heat, like the Predator
FALSE. Kinect works with "near infrareds", that is infrared that are just barely invisible. Those are rather easy to detect, actually most cameras would see them if they weren't equipped with a filter that blocks them for better image quality. What you're thinking about is "far infrared", and much harder and costlier to measure (you can't even use regular lenses since regular glass blocks that wavelength)

... move wands look like dildos and have a glowing ball that allows it to be tracked with pseyetoy.

We got a lecture on how Kinect works using computer vision at my university. Microsoft generated thousands of random images of human bodies at different positions but with special colour coding representing the region of each joint on the body. They then extracted different features from each image using depth information and used them to train a classifier (on a supercomputer) which would label each pixel of a depth image (from the camera) with a region for each joint. The centroid of these regions is then considered the position of the corresponding joint, and then they're all linked up. That's a fairly simplified version of what happens, but I recall the feature extraction being the only complex part of the procedure.

Alx said:

It cannot track people sitting down.
TRUE at first, then FALSE. The first versions of the tracking software were only meant to track standing people indeed. The libraries have been updated since, and it is now possible to track the upper body of a player sitting down.

Click to expand...

I really wish tracking when seated worked better than it does. I loved the head tracking in Forza 4 but the Kinect kept losing me unless I basically sat on the edge of the couch. No lounging back allowed. Hopefully this has been resolved with Kinect 2.0.

Randdalf said:

We got a lecture on how Kinect works using computer vision at my university. Microsoft generated thousands of random images of human bodies at different positions but with special colour coding representing the region of each joint on the body. They then extracted different features from each image using depth information and used them to train a classifier (on a supercomputer) which would label each pixel of a depth image (from the camera) with a region for each joint. The centroid of these regions is then considered the position of the corresponding joint, and then they're all linked up. That's a fairly simplified version of what happens, but I recall the feature extraction being the only complex part of the procedure.

Click to expand...

Indeed ; I intended to go back to the research paper before I describe the process, but your summary sounds quite complete. I'll add a link to your post, thanks.
Do you remember what kind of classifier they used ?

saunderez said:

I really wish tracking when seated worked better than it does. I loved the head tracking in Forza 4 but the Kinect kept losing me unless I basically sat on the edge of the couch. No lounging back allowed. Hopefully this has been resolved with Kinect 2.0.

Click to expand...

Hmm, it's possible but the main issue seems to be more on the software side than hardware side, mainly in the body segmentation. I'm assuming they've done a lot of work improving the algorithm and the hardware is probably somewhat different as well but, I think the sitting problem is less hardware, more software based

Alx said:

Do you remember what kind of classifier they used ?

Click to expand...

According to

http://games.slashdot.org/story/11/03/26/2014234/kinects-ai-breakthrough-explained

They use what they call a "decision forest", a vast collection of decision trees.

Highly recommend CotW's videos. They tell you a lot about each one and it's pretty well explained.

http://clanofthegraywolf.com/category/the-way-games-work/

LiK said:

Highly recommend CotW's videos. They tell you a lot about each one and it's pretty well explained.

http://clanofthegraywolf.com/category/the-way-games-work/

Click to expand...

Thanks... for killing my thread.
Will look at the videos, it will be helpful for many of the controllers I don't know well...

Zoramon089 said:

Hmm, it's possible but the main issue seems to be more on the software side than hardware side, mainly in the body segmentation. I'm assuming they've done a lot of work improving the algorithm and the hardware is probably somewhat different as well but, I think the sitting problem is less hardware, more software based

Click to expand...

I think the main problem for tracking sitting people is that the skeleton model may "stick" to parts of the couch or armchair, which would make it less stable. A standing player would most likely be surrounded by a free area and avoid such problems.

Alx said:

Microsoft removed a processor that did all the processing to reduce costs.
TRUE/FALSE ? Before its official release, rumors mentioned that all the body tracking processing was supposed to be embedded in the peripheral itself, but that Microsoft removed it to reduce its price, and let the console do the hard work. We have no precise reference to that processor, and no hard proof of its existence. Primesense was advertising such a processor on its website, but none of the products they sent to market seem to include said chip. Also considering how their own libraries were not able to initiate body tracking without user calibration, my own opinion is that if such a chip was considered, it wasn't good enough for what MS wanted to do.

Click to expand...

AFAIK, Ms wanted a dedicated processor on kinect for doing the skeletal tracking. That processor was never introduced to the device, even during test phase were they used a computer to do the heavy lifting, but it does include two processors.

The one from primesense, is included but was never supposed to handle the tracking, it converts the image captured from the depth camera into a depth image, and IIRC, can do some basic recognition separating the depth data of up to 4 people to serve as a base for the tracking. And one from marvell who handles video conversion, audio processing, and registration (aligning the depth and RGB streams).

The actual tracking, that was supposed to handle by the custom processor ended up running on 360.

Well, that topic doesn't seem to hot right now... I'll add a few details about Move tonight, hopefully the PS4 reveal could give us new things to discuss. If nobody is intersted, I'll let it sink.

Move : Basics

Most motion controllers try to do measurements on 6 degrees of freedom, defining the position of a rigid object in space (3 translations, 3 rotations). By the way, those degrees of freedom use only 3 axes, unlike the name of the sixaxis would make you believe.

So, everybody knows that the Move is based on standard camera technology, being an evolution of the original Eyetoy. A difficult aspect of that technology is to be able to recognize in a potentially complex image the object you want to track.

That's why the move wand has a big colored sphere at its tip : the software will make sure that the sphere color is different enough from the background (or the players' clothes), then will extract from the image the objects of similar color, and recognize the disk-shaped ones as the projection of a Move wand.



Since the projection of a sphere is always a disk, whatever the orientation of the wand, the center of the disk in the image will define a line of sight between the camera and the wand, and the size of the disk will define its distance.



That image processing task is rather easy, fast and robust. It can be done at high framerate and resolution ; fitting a disk model to a blob can even give sub-pixel accuracy on the estimation of its center.
That method "only" estimates the absolute position of the sphere/wand, though. To measure its orientation, inertial sensors similar to those of the Wiimote are used.
(...)

Move : Strengths and Weakness

Strength

• tracking an easy to recognize blob is a fast and reliable task in image processing. The tracking of the orb position can be precise both in space and time (accurate position at high framerate)
• a common weakness of computer vision is its dependency on the environment ; but the abitility to change the sphere color makes it possible to find one that is different than your gaming environment.

Weakness

• because it's based on integration of inertial data, estimation of orientation may suffer from drifting problems. Those are usually corrected by frequent calibration.
• orientation measurement being less precise over time, the Move is less precise at pointing.

It cannot track fingers.
TRUE. Tracking fingers needs two things : a good software for finger tracking, and a good view on the fingers. As far as we know, the first one isn't ready yet. And even if it was, it is very difficult
to get a good enough image of the user hands. The main reason isn't resolution actually, although it's limited to 320x240 for bandwidth restrictions, it could be enough if the user's hands were close to
the sensor. The main difficulty comes from the self-shadowing of fingers in most hand positions, that creates incomplete depth data.

Click to expand...

So Kinect should add some Minority Report-style gloves to overcome the finger tracking problem of self-shadowing.

Instrumenting the fingers is a good way to make their tracking easier indeed (although there would still be cases of occlusion of the leds by the other fingers). The Minority Report solution would be more of a "Move philosophy" than a kinect one though : the leds would be rather tracked by a stereo pair, since the use of the kinect projector wouldn't help. And the general idea with kinect is to track any user without intrusive equipment for an always available UI (but that's more a use case concern than technological one).

I'm not sure a constructor would dare go that way, but if one does, I would bet on Sony. If they confirm using a stereo camera on the PS4, they could do it on this generation.

For the distant future, the use of gloves would also have the benefit of allowing force feedback ; it already exists in current VR systems, but of course it's very expensive and not too convenient :

Bumping informative thread

I'm a computer vision master student delving in human recognition stuff. (I might make mistakes, so bear with me)

For Kinect, there's a paper by Shotton over here: http://research.microsoft.com/pubs/145347/BodyPartRecognition.pdf

Infrared based depth camera and projector are used in Kinect, so basically it projects an infrared pattern, and uses that information to infer the depth of the captured scene.
It doesn't use two cameras (the RGB is just to capture the 2D normal scene), but it uses stereo camera algorithms / principles to infer the depths.

Here are 'simple' slides about them:
http://courses.engr.illinois.edu/cs498dh/fa2011/lectures/Lecture%2025%20-%20How%20the%20Kinect%20Works%20-%20CP%20Fall%202011.pdf

http://users.dickinson.edu/~jmac/selected-talks/kinect.pdf


It labels body part, mostly to identify joints, and some to fill the gaps (and used in combination to predict joints).

Simple depth features, which is basically measuring difference between detected depths between two coordinates, are used as weak signals.

All these features (values that describe the body / joints) are fed into random decision forest classifier. Basically this classifier is trained to learn to identify these body parts.

I'm not really good in machine learning, but I always think of random decision forest is something like a decision tree, where in each branches / leaves of the tree you have a probability value that says, hey this might be this body part, etc.

Not sure about how PSEye exactly works.

But there are some classic computer vision algorithms that might be used (again, I might be totally wrong, and I hope that I am, because there are many exciting algorithms out there)

Firstly the Move controller has a colored bulb, which helps to reduce down the search space when trying to identify where Move is.

It could use Hough transform, which is a robust algorithm to detect circles / ellipses. And it could infer depth by either known ratio between the size of the detected bulb and the depth.

Or it could stereo disparity to find depths of images.

You could read more about those algorithms here:
http://en.wikipedia.org/wiki/Hough_transform

http://www.ece.ucsb.edu/~manj/ece181bS04/L14(morestereo).pdf

Penguins&PolarBears said:

So Kinect should add some Minority Report-style gloves to overcome the finger tracking problem of self-shadowing.

Click to expand...

Previous Kinect SDK itself doesn't provide methods to detect fingertips, but you could add your own algorithm to do it.

Here's one of the small project that I did to use Kinect to detect my fingertips so that I could do things in minority report style (in this case, simple browsing and zooming thru medical images): http://www.youtube.com/watch?v=MX8QYsl5SM0

Basically I'm using Kinect to find location of my hands, so that I could narrow down my search space, and from there on, I could use convex hull to further determine where my fingertips are.

Interesting thread, thanks for taking the time to make this.