Latest in Gear

Image credit: ACM Transactions on Graphics

Researchers develop method for real-time speech animation

Audio can be incorporated automatically without time-consuming manual animation.
528 Shares
Share
Tweet
Share

Sponsored Links

ACM Transactions on Graphics

Researchers at the University of East Anglia, Caltech, Carnegie Mellon University and Disney have created a way to animate speech in real-time. With their method, rather than having skilled animators manually match an animated character's mouth to recorded speech, new dialogue can be incorporated automatically in much less time with a lot less effort.

To do this, the researchers recorded over eight hours of audio and video of a speaker reciting more than 2500 different sentences. The speaker's face was tracked as they spoke, which was used to create a reference face for an animation model. Off-the-shelf speech recognition software was then used to transcribe the speech sounds. All of this information was subsequently used to train a neural network to animate a reference face, frame-by-frame, based on phonemes -- or individual distinct bits of sound -- pulled from new audio. That reference face was then superimposed on and matched to computer generated characters in real-time.

Training the AI with the reference video and audio takes only a couple of hours and this method lets you use speech from any speaker with any accent and even in different languages. It also accommodates singing. "Realistic speech animation is essential for effective character animation. Done badly, it can be distracting and lead to a box office flop," said lead researcher Sarah Taylor in a statement. "Doing it well however is both time consuming and costly as it has to be manually produced by a skilled animator. Our goal is to automatically generate production-quality animated speech for any style of character, given only audio speech as an input."

This new method was recently published in ACM Transactions on Graphics and presented at SIGGRAPH 2017. You can check out a video of the method at work through the study's supplementary material download here.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.
Comment
Comments
Share
528 Shares
Share
Tweet
Share

Popular on Engadget

16-bit 'Aladdin' and 'Lion King' cartridges are returning in 2020

16-bit 'Aladdin' and 'Lion King' cartridges are returning in 2020

View
Samsung's fix for Galaxy S10 fingerprint scanning will roll out soon

Samsung's fix for Galaxy S10 fingerprint scanning will roll out soon

View
Apple Pay is more popular than Starbucks for US mobile payments

Apple Pay is more popular than Starbucks for US mobile payments

View
Tesla turns a profit as it spins up trial production in Shanghai

Tesla turns a profit as it spins up trial production in Shanghai

View
Four new 'Adventure Time' specials are heading to HBO Max

Four new 'Adventure Time' specials are heading to HBO Max

View

From around the web

Page 1Page 1ear iconeye iconFill 23text filevr