For starters, the team educated its system with 150 videos depicting numerous objects made from cardboard, foam and other materials. These gave Galileo a small database of doodads and their physical properties, allowing it to make some rudimentary hypotheses. Next, model information was added from Bullet, a physics engine used in video games such as Red Dead Redemption. These simulated each collision, creating velocity profiles and object positioning which acted as "a reality check" for Galileo. The deep-learning algorithms then helped the system to refine its guesses to the point where, looking at the first frame of a video, it could recognise all of the objects and determine how they would behave.
To assess its performance, the team created some visual challenges which were also performed by human test subjects. "The scenario(s) seem simple, but there are many different physical forces that make it difficult for a computer model to predict, from the objects' relative mass and elasticity to gravity and the friction between surface and object," Ilker Yildirim, a lead author on the team's resulting research paper said. "Where humans learn to make such judgments intuitively, we essentially had to teach the system each of these properties and how they impact each other collectively."
Surprisingly, the computer and its human challengers performed similarly, occasionally making optimistic estimates about how far an object would move. That's a success and testament to the team's underlying methodology, which sought to make an "intuitive physics engine" similar to our own -- mistakes and all. Maybe one day, a robot will be able to walk into a room and produce a Rube Goldberg machine like this one.