Leap Motion is perfecting natural hand gestures for VR

As immersive as virtual reality can be, you're still left holding awkwardly shaped controllers in your hands -- a reminder that you're very much in the real world. But what if instead of hitting A to move a rock, you could just use your hands? That's exactly what Leap Motion, known for its hand-gesture control sensors, has been working on for the past few years. Most recently, it's been working on something called the Interaction Engine, which aims to take things a step further: to make picking up objects in the digital world feel as natural as it does in the real one.

I had a chance to try the latest demo for myself last week. First, I wore what was essentially an Oculus Rift DK2 with one of Leap Motion's 3D motion sensors strapped to the front. While wearing the headset, I was able to see my real hands in front of me (albeit in an infrared environment), at which point I could push virtual boxes out of my way just by waving them around. There was hardly any latency -- Leap Motion says the camera streams about 120 frames per second -- which I found impressive. Still, this was a demo that I've seen before, and it wasn't too difficult to figure out.

What came next, however, needed a touch more finesse, and it's one of a few new actions enabled by the Interaction Engine. I was asked to select a single ball from a grid of virtual spheres and then drop it off in a cage. I pretended that I was actually picking up a physical object, bringing it to its destination and then letting it go. The cage lit up and made a sound when I was successful. That sounds like an easy task, but Caleb Kruse, who leads user testing at Leap Motion, tells me that the technology behind it is actually quite complex. For one thing, there's no tactile feedback -- I'm not exactly "feeling" anything -- and yet I'm treating this virtual object as if I'm actually holding it.

"What we have to do is figure out the intent of your actions and translate that to the digital objects," he said. "It has to behave exactly as you would hope." One way to do that is with visual and audio cues. "We want you to grab it like a real thing," said CEO and co-founder Michael Buckwald. But not everyone grabs things in the same way, he said; some people simply close their hand around the whole object, for example. Leap Motion has accounted for that, though. "It should still work, no matter how you grab it," Kruse said.

Kruse then showed me a new demo the team has been working on, called Zero G. He instructed me to put my thumb and forefinger together with both hands. Then he told me to do a "pinch out" gesture while moving my hands apart at the same time. Instantly, I found that I was able to create blocks. As I kept doing it, I stacked one box after another, creating a tower. Then I knocked it down with a wave of my hand. What's more, I was able to turn a block on its corner with one hand and then spin it with the other.

This demo, said Kruse, requires precise hand-tracking in addition to a robust graphic engine. Of course, this is still strictly a demonstration, intended mostly to inspire developers. I also kept wishing the blocks felt "denser" so that it wasn't so easy to push them around. But this was a good start. I was impressed by how easy it was to get my hands in and start creating virtual objects. "Now your hands no longer feel like just an input method," said Kruse. "It just feels like you're playing with blocks."

Of course, Buckwald said that controllers still have a place in virtual reality, especially for driving or combat-style games. But for more exploratory uses, hands are so much better, since they let you gesticulate, point and pick up objects without a second thought. "Plus, there's no learning curve," he joked. "You already know how to use them!"

The next step for Leap Motion is to see if it can get this tech in consumer headsets. Buckwald is optimistic. Specifically, he thinks that future VR headsets will be much more mobile: Instead of being tethered to a PC or requiring a phone like the Gear VR, all the processing would be done in the headset itself. This would also make it easier to simply integrate Leap Motion's sensors into the hardware. Plus it makes so much more sense to use motion controls when you're mobile. "We want it to be a true untethered experience," said Buckwald.

"I think there are many big tech companies that will want to be a part of this," he continued. He indicated that Leap Motion is already in talks with a few of them, but he wouldn't divulge any names. "It's a really exciting time to be involved in this space. Keep your eyes open."