MIT's AI knows what's in your cookies just by looking at them
It could eventually help you track your daily nutrition.
Imagine an app that can help you figure out how to replicate what you're eating in a restaurant and help track your calorie intake just by taking a picture of your plate. A team of MIT CSAIL researchers have developed an artificial intelligence system that has the potential to evolve into that kind of application. They call the AI Pic2Recipe, because it can predict the ingredients and recipe used to make a dish from a single snapshot. If that sounds familia, it's probably because of "See Food," the fictional app that made an appearance in HBO's Silicon Valley, and a new-ish Pinterest feature that recognizes the most prominent ingredients in a picture of food.
Other teams and institutions are also working on similar projects, and a common issue is the lack of samples that leads to limited accuracy. The team had to scour various social networks and websites, such as All Recipes and Food, to collect 1 million recipes. They then annotated the collection with more information about their ingredients and used the database to train a neural network.
The result? Their system can identify ingredients like like flour, eggs and butter and can spit out the correct recipe for the dish in an image 65 percent of the time. CSAIL graduate student and lead author Nick Hynes explains:
"This could potentially help people figure out what's in their food when they don't have explicit nutritional information. For example, if you know what ingredients went into a dish but not the amount, you can take a photo, enter the ingredients, and run the model to find a similar recipe with known quantities, and then use that information to approximate your own meal."
Before you can use it as part of your fitness regimen, though, the team still has to work on making it more accurate. They're also developing its ability to infer how a dish is prepared and to distinguish one ingredient from another. You can put the AI to the test yourself if you want -- just upload a pic on the team's demo website or click one of the available images.
