Latest in Culture

Image credit:

Facebook's new mobile AI can process video in real time

It's the first result of a new push for neural networks on phones.
Steve Dent, @stevetdent
November 8, 2016
Share
Tweet
Share

Sponsored Links

Facebook has started rolling out its "Caffe2Go" AI platform that does advanced style transfer video effects in real time using only your iOS or Android smartphone's horsepower. While the painterly effects are cool (see the video, below), the tech behind it is much more interesting. Deep learning normally requires content "be sent off to data centers for processing on big-compute servers," Facebook wrote, but with Caffe2Go, the processing can be done "in the palm of your hand."

The new platform is part of a larger AI effort that includes the machine-vision Lumos app used to suss out images that violate its community standards. It has also open-sourced similar tech on Github to non-Facebook developers. It's not the only company doing AI projects, of course. Google released its Tensorflow framework to the open source community and Microsoft recently made its Cognitive Toolkit available to developers.

Facebook first flaunted Caffe2Go last month, then brought some of the effects to a new camera in a limited European release. Much like the Prisma app, it transfers styles from Van Gogh or Monet onto any still or moving image. Processing live video normally requires at least a well-equipped PC, but Facebook says "we were able to provide AI [processing] on some mobile phones at less than 1/20th of a second," six times faster than an eye blink.

Company engineers had to design software that worked with your smartphone's limitations on memory and computing power. At the same time, they wanted the app to scale up for use on servers or workstation class machines. To that end, the team created a lightweight UNIX-based system 100 times smaller than similar deep-learning programs that works on CPU, GPU, Android and iOS. They then created add-in modules, including a CPU feature called NEON that improves mobile processing speeds.

Facebook is giving developers access to Caffe2Go via its stack, and plans to open-source parts of it "over the coming months." While the video style transfer is a good test of the technology, it's capable of doing other AI processing tasks related to image, speech and more. It won't result in WestWorld any time soon, but it should open up the possibilities of what you can do on your smartphone in the near future.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.
Comment
Comments
Share
Tweet
Share

Popular on Engadget

Engadget's 2020 Back-to-School Guide

Engadget's 2020 Back-to-School Guide

View
Amazon has eliminated single-use plastic at its Indian fulfilment centers

Amazon has eliminated single-use plastic at its Indian fulfilment centers

View
Facebook used 86 percent renewable energy in 2019

Facebook used 86 percent renewable energy in 2019

View
Google makes a $10 billion bet on India

Google makes a $10 billion bet on India

View
Walmart is turning some of its parking lots into drive-in theaters

Walmart is turning some of its parking lots into drive-in theaters

View

From around the web

Page 1Page 1ear iconeye iconFill 23text filevr