Can machines come up with plausible sounds effects for video? Recently, MIT's artificial intelligence (CSAIL) lab created a sort of Turing test that fooled folks into thinking that machine-created letters were written by humans. Using the same principal, the researchers created algorithms that act just like Hollywood "Foley artists," adding sound to silent video. In a psychological test, it fooled subjects into believing that the computer-generated banging, scratching and rustling was recorded live.
Researchers used a drumstick (chosen for consistency and because it doesn't obscure the video) to hit various objects, including railings, bushes and metal gratings. The algorithm was fed 978 videos with 46,620 actions, helping it recognize patterns in the audiovisual signal. "Training a model to synthesize plausible impact sounds from silent videos, [is] a task that requires implicit knowledge of material properties and physical interactions," according to the paper.