NASA's use of HoloLens puts you on Mars with Buzz Aldrin

I got a ticket to Mars.

At the end of an exhibit hall at San Francisco's Moscone Center, a Microsoft representative punched my orange ticket. I was ready to strap on the company's augmented reality headset for a holographic stroll on the neighboring planet. I was one of eight HoloLens-wearing visitors in a group at NASA's Destination: Mars installation at Build. Although the demo was open to a limited audience last week, it will open its doors to all visitors at NASA's Kennedy Space Center in Florida this summer.

Inside a pitch-black room, stars twinkled in every direction I looked through my headset. Within seconds, I heard a male voice behind me. I turned to look in the direction of a life-size and lifelike hologram of Buzz Aldrin, the legendary Apollo 11 astronaut. Standing on the rusty red surface, he welcomed the group to the planet and proceeded to give us a tour. Many arms in the room stretched out to touch him but only cut through the light projection of his NASA jacket.

Halfway through the experience, Erisa Hines, a Curiosity rover driver at NASA's Jet Propulsion Laboratory, showed up on the planet. Her holographic avatar, dressed in dark denim and high boots, introduced the six-wheeled vehicle that has been gathering data on Mars since 2012. As she talked about discovering specific formations based on the rover's findings, a small white arrow simultaneously popped up on the screen to make me look at a large rock before pointing me to a drilling site.

"We didn't concoct this environment for people to look at," Dr. Jeff Norris, lead mission operations at JPL, told me later. "It's not a theme park ride. This is what Mars really looks like. People are looking at it the same way as many of our scientists are looking at it."

Destination: Mars is an offspring of OnSight, a mission-control software that helps JPL scientists navigate the surface of the planet with HoloLens. Through the collaboration between NASA and Microsoft, which kicked off early last year, the team has been able to project an accurate 3D replica of Mars. "When scientists put the HoloLens on, Mars fills out the room around them," said Lorraine Bardeen, GM Windows and HoloLens experiences. "So they can get up, walk around and say, 'Oh, actually we were going to go that way but there's a dip that would've gotten the rover's wheels caught in. Let's go around that way.' They wouldn't see that from the 2D images."

Prior to OnSight, scientists have worked with two-dimensional information, images stitched together for scientific planning and estimation. "They have spent years being trained for this. They're optimized to understanding those images but it's still challenging," said Bardeen. "With augmented reality, they're able to avoid challenges they wouldn't have realized were coming."

Because AR is a medium that modifies a user's reality, it's easy to compare its applications to VR. While the use cases may sound similar, they're not interchangeable. "We use HoloLens for OnSight because we wanted the scientists, who we're building the tools for, to be able to use other tools in conjunction with it," Norris pointed out. "So when they're on the surface of Mars, they can look up and see where things are but also do things on their computer at the same time."

Having successfully employed the device internally and even sending a couple of headsets to the International Space Station, bringing the holographic experience to tourists and space enthusiasts in Florida seems like a natural step forward. "This is the best way for us as scientists and engineers to look at the planet but this is also the best way to involve the public in what's happening in the journey," said Norris. "Space exploration feels so abstract to people when they're just looking at the picture or a video. But this allows us to speak to the part of their brain that will make them realize that this is a real place. We greatly respect the innate ability and desire to explore. We want to unburden and unlock that."

While both VR and AR have the power to solve specific problems and unlock immersive experiences, in some use cases like a public installation, one medium is more effective than the other. Where VR creates a full, deep immersion, AR can build shared experiences. The ability to have a personal moment as you view the spectacle of space through your headset is only made greater when you hear other people in your group "ooh" and "aah" at the same sights. "It's very important to us that it's not a solo experience," said Norris. "We want the group to be self directive, crisscrossing in the room and pointing together. We needed to have an untethered device like HoloLens to be able to do that."

NASA's use of the device is transforming the way scientists are studying an alien surface. It validates the visual capabilities of Microsoft's technology. Imagine being able to experience the solar system with a legendary astronaut in addition to reading about it, seeing your future holographic home in your architect's office or collaborating on a design project in real time over Skype.

But outside of Microsoft's initial partnerships with NASA and Case Western Reserve University (where HoloLens is being tested in a medical setting), the landscape of applications is wide but mostly barren at the moment. The infancy of the medium is its biggest challenge right now. "For Destination: Mars we were figuring out how to render a full-scale person and the Mars terrain together, within the performance constraints of the device," said Norris. "But also how do we tell a story about Mars where people can look anywhere they want but also attract their attention to things that are exciting. We had to find a balance between leaving them alone and drawing them in."

The inclusion of Aldrin and Hines in this narrative keeps the viewer engaged in the experience. "I'm hoping that the way we connect participants with Mars, we will also connect them with the explorers," said Norris. "Mars is a real place and real people are exploring it. Maybe this will help people think, 'I can be a part of that exploration.' I think if we achieve that then Destination: Mars was successful."

Back in the dark-room installation, Aldrin stepped back into view to conclude the holographic journey on Mars. He talked about the future of space exploration as he drew the group's attention to a human-built camp. All HoloLens-wearing heads in the room followed his lead to find two astronaut figures in the far distance, overlooking the vast expanse of the tawny planet. Within moments, a space shuttle blasted into space with a rumbling sound. I turned my head up, way up, to follow its fire trail cut across the dark sky. As I watched it fade into space, I suppressed the lump in my throat before slowly making my way out.

[Image credit: NASA/JPL-Caltech/Microsoft]