Latest in Gear

Image credit:

Researchers develop method for real-time speech animation

Audio can be incorporated automatically without time-consuming manual animation.
Share
Tweet
Share

Sponsored Links

ACM Transactions on Graphics

Researchers at the University of East Anglia, Caltech, Carnegie Mellon University and Disney have created a way to animate speech in real-time. With their method, rather than having skilled animators manually match an animated character's mouth to recorded speech, new dialogue can be incorporated automatically in much less time with a lot less effort.

To do this, the researchers recorded over eight hours of audio and video of a speaker reciting more than 2500 different sentences. The speaker's face was tracked as they spoke, which was used to create a reference face for an animation model. Off-the-shelf speech recognition software was then used to transcribe the speech sounds. All of this information was subsequently used to train a neural network to animate a reference face, frame-by-frame, based on phonemes -- or individual distinct bits of sound -- pulled from new audio. That reference face was then superimposed on and matched to computer generated characters in real-time.

Training the AI with the reference video and audio takes only a couple of hours and this method lets you use speech from any speaker with any accent and even in different languages. It also accommodates singing. "Realistic speech animation is essential for effective character animation. Done badly, it can be distracting and lead to a box office flop," said lead researcher Sarah Taylor in a statement. "Doing it well however is both time consuming and costly as it has to be manually produced by a skilled animator. Our goal is to automatically generate production-quality animated speech for any style of character, given only audio speech as an input."

This new method was recently published in ACM Transactions on Graphics and presented at SIGGRAPH 2017. You can check out a video of the method at work through the study's supplementary material download here.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.
Comment
Comments
Share
Tweet
Share

Popular on Engadget

AOC's 'Among Us' Twitch stream peaked at over 435,000 viewers

AOC's 'Among Us' Twitch stream peaked at over 435,000 viewers

View
LG's rollable OLED TV goes on sale for $87,000

LG's rollable OLED TV goes on sale for $87,000

View
GMC's 1,000HP Hummer EV is an 'all-electric supertruck'

GMC's 1,000HP Hummer EV is an 'all-electric supertruck'

View
'Pokémon Go' gets AR Mapping tasks to enable more realistic effects

'Pokémon Go' gets AR Mapping tasks to enable more realistic effects

View
iPhone 12 and 12 Pro review: Apple enters the 5G era

iPhone 12 and 12 Pro review: Apple enters the 5G era

View

From around the web

Page 1Page 1ear iconeye iconFill 23text filevr