Skywalker Sound and the challenges of making audio for VR films
"You have to take a fresh look at it and put aside all your preconceived notions."
Lauded Mexican filmmaker Alejandro González Iñárritu, best known for his work on movies such as Birdman and The Revenant, last year nabbed a "special" Oscar award for his VR exhibition Carne y Arena. The virtual reality project, dubbed Flesh and Sand in English, takes viewers into a world where they can experience what it's like to be an immigrant trying to cross a border. As much as visuals were important to tell this story, one of Iñárritu's focus was to also to create the most immersive sounds -- which can be complicated when going from traditional film to a completely new medium like VR.
To do that, the director recruited Skywalker Sound, an arm of the Lucasfilm motion picture group, to work on making the best spatial, 360-degree audio possible for Carne y Arena. But that wasn't easy, according to Kevin Bolen and Bill Rudolph, two sound designers from Skywalker Sound who worked on the installation and spoke at GDC 2018 about the challenges of bringing it to life. They said that implementation is half the battle, as there are a number of factors to consider when creating sounds for a VR film, like 3D panning, how the waves are positioned and the artistic spread of the story itself. These are things that video game developers don't typically think about, per Bolen and Rudolph, which presents a major challenge during production.
Another major factor to consider is that, in a VR movie, everyone who watches it can have a different experience. This means it's key for the spatial sound to fit into a story's narrative, regardless of where a viewer's eyes are focused on while wearing a VR headset and watching a scene. Bolen and Rudolph said Iñárritu was obsessed with getting this right in Carne y Arena, since it was important for him that people could hear the dialogue as it was intended -- so even if you're wandering away from a character who has some key lines, you won't miss out on what they have to say.
Bolen and Rudolph added that after working with Iñárritu on Carne y Arena, they realized they needed to be naive in their approach to applying sounds to a VR film: "You have to throw as many people under the experience as you can. You have to know if someone's going to stand in a place that none of the developers expected." This is something that Oculus was doing a great job understanding through VR movies like Lost and Henry, though the company shut down its in-house Story Studio (which was responsible for making those) last year.
At the end of the day, Bolen said it's best to approach VR films with a new perspective, rather than try to make them feel like traditional movies. Not just because that's what is necessary to make them feel more realistic, but because it's also what directors like Iñárritu see a challenge they need to conquer. "You have to take a fresh look at it," he said, "and put aside all your preconceived notions."
Click here to catch up on the latest news from GDC 2018!
