Latest in Culture

Image credit:

Facebook's new mobile AI can process video in real time

It's the first result of a new push for neural networks on phones.
310 Shares
Share
Tweet
Share
Save

Sponsored Links

Facebook has started rolling out its "Caffe2Go" AI platform that does advanced style transfer video effects in real time using only your iOS or Android smartphone's horsepower. While the painterly effects are cool (see the video, below), the tech behind it is much more interesting. Deep learning normally requires content "be sent off to data centers for processing on big-compute servers," Facebook wrote, but with Caffe2Go, the processing can be done "in the palm of your hand."

The new platform is part of a larger AI effort that includes the machine-vision Lumos app used to suss out images that violate its community standards. It has also open-sourced similar tech on Github to non-Facebook developers. It's not the only company doing AI projects, of course. Google released its Tensorflow framework to the open source community and Microsoft recently made its Cognitive Toolkit available to developers.

Facebook first flaunted Caffe2Go last month, then brought some of the effects to a new camera in a limited European release. Much like the Prisma app, it transfers styles from Van Gogh or Monet onto any still or moving image. Processing live video normally requires at least a well-equipped PC, but Facebook says "we were able to provide AI [processing] on some mobile phones at less than 1/20th of a second," six times faster than an eye blink.

Company engineers had to design software that worked with your smartphone's limitations on memory and computing power. At the same time, they wanted the app to scale up for use on servers or workstation class machines. To that end, the team created a lightweight UNIX-based system 100 times smaller than similar deep-learning programs that works on CPU, GPU, Android and iOS. They then created add-in modules, including a CPU feature called NEON that improves mobile processing speeds.

Facebook is giving developers access to Caffe2Go via its stack, and plans to open-source parts of it "over the coming months." While the video style transfer is a good test of the technology, it's capable of doing other AI processing tasks related to image, speech and more. It won't result in WestWorld any time soon, but it should open up the possibilities of what you can do on your smartphone in the near future.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.
Comment
Comments
Share
310 Shares
Share
Tweet
Share
Save

Popular on Engadget

Runkeeper drops its Wear OS app due to a 'buggy experience'

Runkeeper drops its Wear OS app due to a 'buggy experience'

View
Drako's GTE electric supercar will be a four-motor, 1,200HP monster

Drako's GTE electric supercar will be a four-motor, 1,200HP monster

View
Nintendo says there is no Switch exchange program

Nintendo says there is no Switch exchange program

View
IKEA creates a business unit devoted to smart home tech

IKEA creates a business unit devoted to smart home tech

View
US will reportedly give Huawei another temporary reprieve

US will reportedly give Huawei another temporary reprieve

View

From around the web

Page 1Page 1ear iconeye iconFill 23text filevr