Facebook's new shopping AI knows what exactly you're looking for

Finding fashion that suits your style can be a challenge even when done in person at a brick-and-mortar store. Add to that the vagaries of online commerce, where you can only see a stock representation of what you want, can make for a frustrating, and occasionally ill-fitting, shopping experience -- especially when purchasing items second-hand. That’s an issue that Facebook’s looks to address with its new GrokNet computer vision system that can transform virtually any photo into a shopping opportunity.

GrokNet is designed specifically for shopping and is capable of identifying the attributes -- color, texture, pattern or style, for example -- of the items it sees. Typically, machine vision systems are only adept at picking out one type of object (ie, a chihuahua vs. a blueberry muffin) so a system designed to recognize various styles of shoe is going to struggle with identifying specific models of SUVs. But unlike other machine vision systems, GrokNet is robust enough to do so whether you show it a picture of the newest Yeezys or a red leather sectional sofa.

This enables the system to automatically suggest a product’s attributes to Marketplace sellers when they upload a picture of what they’re trying to list, simplifying the process of posting items for sale. What’s more, it will allow retailers to display their wares in a more natural setting. So rather than listing a dozen catalog items individually, REI for example could instead post an image of a family camping using its products and shoppers would be able to click on each item in that picture to get more information or purchase it on the spot.

Conversely, it allows buyers to be far more specific about what they’re shopping for than they would otherwise. This comes in handy when you’re looking specifically for a blue suede chaise lounge and not just any blue sofa. It will also allow you to find an exact model based only on a photo of it. So if you see that pair of Yeezys in your feed but don’t know exactly what colorway they are, GrokNet can autonomously scan through the Yeezy store and return the precise product listing, saving you the effort of endlessly trawling the site in hopes of stumbling up on the pair you saw. According to a Facebook press release, this new system is twice as accurate as its predecessor.

But as any digital shopper can tell you, products often look and fit slightly differently in real life than as they appeared online. The size and scale of an item can be difficult to discern when looking at it on a monitor rather than holding it in your hands. To help alleviate that, Facebook is also rolling out a feature that allows sellers to generate 3D-like walkarounds of their items using only their smartphones. The seller simply takes a standard video of the item and the system will automatically stitch it together into an interactive, 360-degree view of the object.

Some retailers, such as Ray Ban and NYX have gone even further and adopted AR to help sell their products. Leveraging Facebook’s existing AR and facial recognition systems, these brands can apply virtual lipstick shades or hover digital sunglasses over the shopper’s face to give them an indication of what they’d look like wearing the product. This system is primarily used by optics and makeup companies right now but Facebook hopes to eventually expand those offerings to include the likes of clothing and home furnishings as well.