The real-time motion capture behind ‘Hellblade’

Mocap suits are an unflattering but essential part of performance capture.

After the divisive DmC: Devil May Cry reboot and its work on Disney Infinity 3.0, Ninja Theory went fully independent. Before, its games had been funded by Sony and juggernaut publishers Bandai Namco and Capcom. But middling sales and an increasingly competitive market, in which blockbuster games are expected to shift millions, made it tough for the team to pitch a title that wasn't "design by spreadsheet," in Antoniades' words. So with a team of just 13, Antoniades decided to change tack and make the equivalent of an indie film -- a beautiful and artistic game but self-published and with a budget magnitudes smaller than at a normal triple-A studio.

Antoniades says Garland is partly responsible for the shift. "Towards the tail-end of Enslaved, in which he worked with us quite deeply, he said that he just couldn't understand why, in the gaming world, we went from the bedroom to blockbusters, and there wasn't the equivalent of independent movies. Movies that can sit alongside the blockbusters in the cinema, but aren't seen as second tier or cheap in any way. That stuck in my mind."

Melina Juergens plays Senua in Hellblade.

"It was always stealing, borrowing, inventing as we went. We felt like rascals."

Early on, for instance, the team looked at photogrammetry, a way of measuring depth through photos, to create a face scan of Juergens. Later, the team built Hydra, a prototype camera rig with multiple GoPros and detachable lenses to track the actor's face and body movements as well as the position of the filmer in 3D space. Another prototype used a cricket helmet and a webcam to record faces. At one stage the team had a lighting system housed inside a plant pot, powered by a Raspberry Pi and some custom code, to capture skin and other surfaces in minute detail.

Some worked, some didn't. When it couldn't solve a problem on its own, Ninja Theory reached out to specialists who were willing to collaborate. "It was always a fight," Antoniades recalls. "It was always stealing, borrowing, inventing as we went. We felt like rascals." But the team never felt restricted or disheartened. Vicon, an expert in motion capture systems, loaned the studio 12 of its Bonita capture cameras. Ninja Theory then converted its largest meeting room into a mocap studio, mounting the cameras on Ikea poles and lighting its actors with cheap LED panels from Amazon.

But what came first, the concept or the need to keep the game on a smaller scale? "I think they went together," he says. "The idea can only survive if it's achievable. I did want to do a game based on a character's story, and I knew that we could only afford to focus on one character, in terms of technology and resources. So, then the question became, 'How do we make this truly intimate story about one character? Is it even possible to carry a whole game with just one speaking character?"'

Ninja Theory co-founder and Hellblade creative director Tameem Antoniades

"It was just unproven at the time, and we were the guinea pigs. But it worked beautifully. The detail was just incredible."

Next, the company turned to Cubic Motion, a team in Manchester focused on computer vision. Its technology serves as a middleman in performance capture, tracking and analyzing the actor's face while she performs in the studio. The resulting data -- a 3D point cloud, consisting of roughly 200 virtual markers -- is then read and replicated by the digital rig controlling Senua's face in the game. The best part is that Cubic Motion can gather this data with video footage alone, removing the need to plaster the actor's face with Ping-Pong balls or crosses.

"We can track 30 to 40 points just on the inside of the lips, and you could never get any of that from an optical-based system," David Barton, a producer at Cubic Motion, explains. His team has worked with 3Lateral before, combining its computer vision system -- known in the industry as a facial solver -- with the latter's rigs. It's a perfect partnership; after all, granular facial tracking is pointless if the rig powering the digital character isn't capable of replicating the same subtleties.

The only problem? GDC was eight weeks away. Ninja Theory, Epic, 3Lateral, Cubic Motion and Xsens, a company brought in to handle body tracking, needed to move quickly. For Cubic Motion, it was particularly tough. Typically, the team takes hours to crunch, or "solve," facial data gathered during a mocap shoot. "Now we had about sixteen milliseconds to track, solve and output that data to Unreal," Barton explains. Thankfully, Cubic Motion had been working with Ninja Theory for some time and had been training its system to work with Juergens' face. Still, it needed some refinements.

A week before the presentation, the system was barely working.

"When she's driving it live on the big screen, there can be no tracking errors, certainly no catastrophic tracking errors, because that would just make her face explode, for example," Barton says.

A week before the presentation, the system was "barely working," according to Antoniades. All five companies spent the last three days in San Francisco fighting to iron out the kinks. "It was like an operations room," he recalls. "I saw it as compressing two years of R&D effort by lots of different companies into a few weeks." But everything came together. On the day, Juergens was able to drive Senua without any problems. Once the scene had ended, Antoniades explained that it was, in fact, an actor controlling the character live. The crowd went wild as Juergens sang "Do You Want to Build a Snowman," dispelling any fears that the presentation had been faked.

Barton says he felt relief more than amazement or pride. "Because there are a lot of things that can go wrong in a real-time demo," he says, "especially when it's the first time anyone has done it at that level. So it was relief, but also a lot of pride that it came together in such a short amount of time."

Later that year, Ninja Theory demonstrated the technology again at Siggraph, a conference for visual effects and interactivity. It was part of a real-time graphics competition that included Pixar, Industrial Light & Magic, Oculus, Square Enix and Uncharted developer Naughty Dog. For its second outing, Ninja Theory showed how it was possible to shoot, capture and edit a scene using performance capture and Sequencer, a cinematic editing tool that runs inside Unreal. In this version, Juergens performed twice in quick succession -- once as Senua and a second time as a projection of her inner voice.

On the second time through, Juergens was able to act against her previous performance. Both takes were then combined inside Sequencer to create the final scene. It was enough to impress the judges and land Ninja Theory the award for Best Real-Time Graphics and Interactivity showcase. "We had just created a whole scene with two characters talking to each other," Antoniades says. "Camera, framing, environment, everything. I think that really demonstrated how powerful it is."

"You'll be able to have this sort of mass-performed digital theater of the future."

Real-time motion capture could enable new kinds of experiences too. "The fact that you can live-drive a character means that a famous character from a video game can now be interviewed, as if it was you and me talking right now," Libreri says. "The same goes for concerts or live performances for people at home, either watching through a web browser or in VR. You'll be able to have this sort of mass-performed digital theater of the future."

Hellblade is Ninja Theory's attempt to show that independent games can still have jaw-dropping visuals. Regardless of how the game is received, it's hard to argue with the quality of the cinematics. With a team of just 20, Ninja Theory has produced some truly dramatic and emotional scenes that rival the best in the industry. And along the way, it's pioneered a new form of motion capture with a hobbyist attitude that nothing is impossible.

"If you focus on one thing and want to do it really well, anything's possible," Antoniades says. As I perform the macarena in my mocap suit, watching a strange, Viking warrior follow along, I can't help but agree.