Kinect's value to artists overshadowed its gaming roots

Former NIN art director and a game developer discuss the camera's legacy.

The Kinect is officially dead. But the reality is that Microsoft signed the do-all sensor's fate years ago. Faced with slumping hardware sales in 2014, then-new Xbox chief Phil Spencer had a decision to make. Microsoft could either drop the price of the Xbox One, or continue letting Sony and the $400 PlayStation 4 eat its lunch. So it stopped bundling the Kinect with the console and cut $100 off the asking price.

It worked. Microsoft doubled sales the next month, and this move has set the tone for Spencer's tenure: reversing the string of bad decisions Microsoft made leading up to Xbox One's debut. To illustrate the sensor's waning importance to Microsoft, the Xbox One S didn't have a dedicated Kinect port on the back when it was released in 2016. It's the same with the upcoming Xbox One X, except Microsoft isn't offering a free USB adapter anymore. The writing has been on the wall for a while now. If this week's news was surprising, you probably haven't been paying attention.

The truth is that Kinect's greatest successes had nothing to do with gaming. Hackers adopted the sensor with open arms, using it for everything from interactive art installations to motion capture and even trippy stage shows for massive bands like Nine Inch Nails. Why? Because for the tech that's on board, Kinect was relatively inexpensive and easy to use.

You can buy one online for $55. The adapter needed to connect it to a PC, however, is no longer available direct from Amazon, and there was only one new unit available from a third-party seller at press time.

"It's become the standard for interactive art because it's affordable," Rob Sheridan, NIN's former art director, told Engadget. "That's the best gift you can give someone who has a vision of something creative."

He compared the sensor and Microsoft's quasi-open source approach to it to the advent of digital video at the turn of the century, and how that democratized video editing and production. "People can create interactive art in their bedroom now, and that's something that everybody remembers as an important milestone in moving a creative medium forward."

Two years prior to the Kinect's 2010 debut, NIN went out on the road for its Lights in the Sky tour. The stage production featured all manner of high-tech toys, including giant "touch screen" LED curtains and video screens. The problem was, it was all proprietary software and hardware from entertainment company Moment Factory, and as such, the sensors and screens around the stage were a bit kludgy. NIN shows are far from relaxing; sweat and artificial fog are everywhere, and both make a mess.

Sheridan said there were "numerous" times where the sensors placed around the stage would need to be cleaned during a performance because they weren't working right. That'd lead to the notoriously meticulous Reznor getting pissed off because the pantomiming he was doing wasn't translating to the audience.

For 2013's tour, things were different. NIN partnered with Moment Factory again, but instead of homemade gear, the production company had a simpler solution. "All of the clunky hardware they labored over so hard to get to work was now [replaced with] 'Oh, just plug a Kinect in,'" Sheridan said.

He used the sensors extensively as a way to capture the band's shadows, and to digitally manipulate the silhouettes on rollable LED panels. Each panel had a Kinect mounted to it, and as the set list went on, the shadows would get progressively glitchier. With Kinect, the only problems the band encountered were ones that involved a cord getting knocked loose. "The difference was mind-blowing," he recalled.

Independent game developer Mattia Traverso hasn't designed cutting-edge concert productions, but he did make one of Kinect's best games, last year's Fru. It has sold only a paltry 10,000 copies, but getting rich off of Microsoft's sensor was never his goal. "I saw Kinect as a way to experiment and make cool shit," he said. "We never really thought that we were gonna use Kinect to make money."

In 2014, he entered a game jam with a few friends, not knowing one of them had a Kinect in his bag. Fru was the result, a game that doesn't use the sensor's voice recognition or skeletal tracking tech, but instead takes one player's silhouette and projects it in-game as a means to bridge gaps or reveal secrets; the player becomes part of the 2D game environment itself. A second player moves a fox-masked girl along (or within) the in-game silhouette to reach platforms that would otherwise be inaccessible. It's both charming and inventive as hell.

Traverso said that Kinect in general was an interesting premise because of how it brought the human body into digital games. "When you were playing Fru, you were moving and contorting your body. You were balancing on one leg while moving your hands in a way that you would've not done in any situation unless you were a yoga master," he said, laughing.

The problem, Traverso said, was that Microsoft marketed the sensor poorly. He said that when the Xbox One launched, there wasn't a promise of what Kinect could do aside from control a TV with your voice. If you're not a game designer, explaining what the Kinect can offer your living room is pretty tough. "That's why it didn't work. Nobody could imagine what their Kinect could be."

"We needed a bigger gestation time to figure out how we could use it to make new kinds of games, and we didn't necessarily have that. It wasn't meant to be 'Hey, here's this cool fucking thing that has so many games!'" he said. "No, it was meant to be 'You don't have to think about this as a weird, extra thing -- think about this as a thing you use to control your console, like a remote.'"

Essentially, Microsoft positioned the last version of the Kinect as a mouse, and once the company changed the Xbox One's focus from being "the all-in-one games and entertainment system" to just being a video game console, the Kinect had to go.

Traverso wasn't surprised by this, but he did seem frustrated. Fru garnered plenty of praise from the press and the thousands who played it, but gaining any sort of momentum was hard. "Nobody cared about Kinect anymore, because the entity making Kinect was trying to pretend it didn't exist," he said. "This was the best business decision for them, but at the same time, my team and I were making a game in a market that the owner of the market was trying to shut down."

Traverso doesn't think history will be kind to the Kinect, and that it'll sit alongside Nintendo's Virtual Boy as one of gaming's biggest failed experiments. That's in part because the vocal minority of hard-core gamers will control the narrative. Since they were fed a steady stream of gimmicky titles, or games that they weren't used to, they're going to lash out. The problem is, for every Fantasia: Music Evolved from Rock Band developer Harmonix, there were a handful of clunky mini-game collections like Kinect Sports Rivals. So maybe the anger is justified.

Looking to the future, he hopes some of the Kinect tech, like facial recognition, keeps popping up elsewhere -- like it already has in Instagram, Snapchat and the iPhone X. He proposed that since it's happening for face filters and without "a crazy NASA-like" device such as Kinect, maybe we aren't far off from easy silhouette recognition in more places.

"The pessimist in me thinks this technology would just be used for shitty Facebook filters, but I'd love a future where we try to actually make interesting interfaces for using our body," he said.

Sheridan hopes Microsoft recognizes the impact it has had by opening the Kinect up to everyone, and that Redmond keeps embracing the hacker community moving forward.

He isn't sure what to make of Microsoft stopping Kinect production, but he thinks that ending manufacturing isn't going to make the device itself extinct. There are countless sensors out in the wild, and plenty more still on Amazon and in Microsoft's closets. "They might be killing it off as an official product, but I don't think it's ever going to die as something that's beloved in the interactive [art] world," he said.

Microsoft has said that there's plenty of Kinect's technology in HoloLens, too, and that its work on the sensor helped push HoloLens to where it is today. Sheridan has seen Microsoft's helmet, and while he's cautious, he thinks it could have a similarly bright future for interactive art.

"I've seen where they're taking [HoloLens] with AR, and I feel like if they embrace creativity the same way they did with Kinect, it could be the next Kinect."