An AR film explores the worst tech companies could do with your face

I made a beeline for Stealing ur feelings when I entered the Tribeca Film Festival Immersive Arcade last week. Officially, I'll say that my reason for doing so was because it was the most relevant exhibit for Engadget's scope of coverage: This interactive AR short studies what companies like Snap, Facebook and Google are doing (or can do) with the data they have on your facial expressions. But let's be honest, I'm pretty sure I was drawn to it because of the big selfie camera perched over the display.

In the festival's Arcade section, which is full of eye-catching installations and people wearing headsets, Stealing ur feelings is easy to miss. It consists simply of a 46-inch display mounted onto a pillar in the middle of the aisle separating the exhibits in the long rectangular room. There's a camera above the screen and a button below it to start the video, with minimal signage explaining the project.

When you're ready to experience Stealing ur feelings, hit the button and the video begins. The presenter delivers his fast-paced, upbeat spiel while a mix of stock images and clips of people using Snapchat appear on-screen. A few seconds into the film, you see your face along with an explanation of how the computer picks out features like your eyes and mouth to understand if you're smiling, frowning or neutral. Throughout, the narrator verbally pummels you with revelation after revelation about what the film asserts companies are doing. Things like using your smartphone camera to watch you as you're scrolling through your Facebook feed, for instance, and determine how you react to what you see.

It doesn't just suggest possibilities, either. The video highlights patents filed by Snap, Apple and Facebook that show diagrams and registration numbers with detailed descriptions of how these companies can score your mood based on your facial expression.

It's worth noting that big companies like Snap, Facebook and Apple patent all kinds of technologies all the time, and sometimes never use the intellectual property they own. Whether they will actually deploy or have actually started using these features isn't something we know for sure just yet, but Stealing ur feelings imagines the worst that could happen if they did.

To prove its point, the film shows you what it learned about you from your expressions in the past few minutes. According to Stealing ur feelings, I have a:

• 0.890247 positive reaction to Kanye West

• 0.547226 positive reaction to pizza

• 0.012154 bias for black people

Throughout the film, I felt confused about how the system was assessing these supposed biases. It wasn't until after the film ended when I asked creator Noah Levenson how these numbers were calculated that I learned that the system had been assessing my reaction to the footage it was showing me. Some of its findings were fair, like my preference for celebrity gossip over political analysis in my feeds, and that I like looking at my own face. (To be fair, I was just smiling a lot at the funny masks and filters that the film overlaid on my face as I looked at the screen.)

Many of the film's assessments were wildly inaccurate, though, like when it said I was 100 percent Republican and that I like pizza. (To the chagrin of my friends, I am not a big fan of pizza.)

But the inaccuracy is the point.

I asked Levenson whether he's uncovered any evidence to suggest that these companies are using our phone's cameras to spy on us. "Is it definitely happening? No," he said. "But maybe." Levenson pointed to all the patents these companies have filed for different purposes as proof that they're at least looking into it. "Snap is the most obviously nefarious example," he said.

Indeed, Snap has a patent titled "Determining a mood for a group" that shows it using smartphone cameras to study a user's expressions and assign a score to indicate how they feel. These groups are shown in patent drawings as being at concerts or political rallies, with multiple labels showing the rating for various individuals' mood scores. According to Stealing ur feelings, Snap could then sell that information alongside other data like where the person is or what they might be doing at the time.

This information can then be used to cater content to you. It's pretty much up to marketers to do whatever they want with the data once they have it. Levenson offered examples like the profiles you're shown on dating sites or the political messaging you receive. The danger here is, that sometimes -- often -- the information extrapolated about you is wrong. "When they extract data from you, they market that data as truth," Levenson said. "They decide how much of a racist you are or what races you seem to prefer."

Even data that's not about you could affect your life, Levenson said. "What if Kanye's tour bookers bought Snapchat's aggregated responses as a way to decide where to have his concerts," he said. If you liked the artist but people in your neighborhood overwhelmingly disliked him, then you might lose out on an opportunity to catch a concert at a location convenient for you.

The idea isn't new: Marketers already use old-school information like demographics, survey data or purchasing habits to make similar decisions. But it's the idea that something as new and unpredictable as facial expression can be used in such an exploitative manner that Levenson wants us to think about. I have doubts that big tech companies are actively carrying out such invasive data harvesting methods, but Levenson believes that at the very least, something is going on. "What we know is there are a lot of companies using facial emotion recognition for various purposes," he said.

Some use cases are less nefarious. For example, Snap has another patent titled "Emotion recognition in video conferencing" that's described as a means to enable "another videoconference participant, such as a service provider or supervisor, to handle angry, annoyed, or distressed customers."

Regardless of how a company says it'll use emotion recognition tech, Levenson believes we should be vigilant. He started developing the film after winning a $50,000 Mozilla grant for art and advocacy exploring artificial intelligence last year, and had its official debut at Tribeca last week. You can check out an early version of it on Github. The project will launch as a website later this year, so you can see for yourself what your biases for Kanye and pizza are. Or what Stealing ur feelings thinks they are, anyway.