Facebook can predict a musician’s movement using only an audio clip
The AR avatars could use an upgrade, though.
We're used to seeing the likes of Wii avatars crudely banging at pianos or sawing away at violins, but Facebook is now working on an application that uses music alone to govern the movements of an augmented reality avatar. In other words, input some music, and the avatar will move just like a human pianist or violinist would.
The research, completed in partnership with the University of Washington and Stanford, has just been presented by Facebook at the CVPR (Computer Vision and Pattern Recognition) conference. The team wanted to see if natural body dynamics could be predicted from music, and while the base animation is pretty basic and shaky, they're certainly on their way to achieving that. Next on the agenda will be detailed arm and finger motions.
This is the first time anyone has tried to predict body movement from music alone. It's an interesting piece of AR research, and it could also have wide-ranging applications for music not only in gaming, but in the real world, too. If you wanted to learn a piece of piano music, for example, you could simply input the audio and then mimic the key movements of the avatar pianist -- much more fun than those seventh grade lessons you were forced into.
