Facebook can predict a musician’s movement using only an audio clip

The AR avatars could use an upgrade, though.

Sponsored Links


We're used to seeing the likes of Wii avatars crudely banging at pianos or sawing away at violins, but Facebook is now working on an application that uses music alone to govern the movements of an augmented reality avatar. In other words, input some music, and the avatar will move just like a human pianist or violinist would.

Turn on browser notifications to receive breaking news alerts from Engadget
You can disable notifications at any time in your settings menu.
Not now

The research, completed in partnership with the University of Washington and Stanford, has just been presented by Facebook at the CVPR (Computer Vision and Pattern Recognition) conference. The team wanted to see if natural body dynamics could be predicted from music, and while the base animation is pretty basic and shaky, they're certainly on their way to achieving that. Next on the agenda will be detailed arm and finger motions.

This is the first time anyone has tried to predict body movement from music alone. It's an interesting piece of AR research, and it could also have wide-ranging applications for music not only in gaming, but in the real world, too. If you wanted to learn a piece of piano music, for example, you could simply input the audio and then mimic the key movements of the avatar pianist -- much more fun than those seventh grade lessons you were forced into.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission. All prices are correct at the time of publishing.
View All Comments
Facebook can predict a musician’s movement using only an audio clip