Advertisement

Microsoft's hand-tracking magic may lead to gamepad-free VR

The algorithms also gives users "a virtual sense of touch."

The next Xbox One S console doesn't even have a Kinect port, showing that even Microsoft doesn't care about the depth sensor for gaming. However, it may yet make a comeback in VR. Microsoft researchers have made hand- and finger-tracking much more accurate using the Kinect, which may eventually make it a more attractive option than a controller for VR games. "We're getting to the point that the accuracy is such that the user can start to feel like the avatar hand is their real hand," says Microsoft UK researcher Jamie Shotton.

Finger tracking currently works on the Kinect (using Kinect Studio and other apps), but its accuracy is very poor. The problem, says Microsoft, is that your hand is relatively small and complex, so when you rotate it or make a fist, your fingers disappear from view and fool the software.

To solve the problem, Microsoft creates a smooth surface model of the hand from the data points tracked by the Kinect. It then uses an algorithm that was developed way back in the 1940s to calculate where the tracking points should go next in order to conserve the most energy. The system also uses a "reinitializer" to quickly reset a hand pose in case of tracking errors.

The smoother surface model does require more computations, but the software can resolve hand positions more quickly, within four iterations. In addition, it requires fewer tracking points than other solutions. As a result, it "can track hands smoothly, quickly and accurately –- in real time –- but can run on a regular consumer gadget," says the team.

With the app, users can smoothly interact with objects nearly in real time, something that could be highly beneficial for gaming, design and other VR applications. "The system lets you see what your hands are doing, fixing a common and befuddling disconnect that happens when people are interacting with virtual reality but can't see their own hands," according to the blog.

What's more, with proper interface design, users aren't thrown off by the lack of haptics, or simulated touch. "Thin" virtual controls make you touch your own fingers together for a fake sense of feel, for instance. And with a realistic-looking interface that works in real time and includes sound, users can be convinced they're actually touching something.

The finger-tracking algorithms are purely research for now, and problems like collisions and tracking errors still need to be resolved. However, it holds a lot of promise for gaming, user interface design and other applications. Using an off-the-shelf Kinect, Microsoft showed off virtual gamepads, painting tools that let you "grab" a color with your fingers and a game where the player is a "god" that literally gives a helping hand to his characters.