Google's AI gets human help to avoid offensive search results

It aims to stop users from seeing things like neo-Nazi results for Holocaust searches.

Bloomberg via Getty Images

Google is making a new push to eliminate offensive search results such as those that appeared from US neo-Nazi site Stormfront in response to queries about the Holocaust. As Search Engine Land noticed, the site has revised its guide on how to assess search result quality for around 10,000 of its "quality rater" contractors. That includes a new "upsetting-offensive" content flag for the promotion of violence or hate against minorities and other groups, racial content, graphic violence and human trafficking.

On a search for "holocaust history," for instance, Google instructs raters on how to handle two different results (below). The first shows a post from said racist site Stormfront on Holocaust denial, something that's actually a crime in over 20 countries. Google tells raters to flag that with the "Upsetting-Offensive" flag "because of the direct relationship between Holocaust denial and anti-Semitism.

Google says the second example from The History Channel doesn't require the "Upsetting-Offensive" flag, though. Even though it's clearly an upsetting topic, "this result is a factually accurate source of historical information" that, unlike Stormfront, "does not exist to promote hate or violence against a group of people," the document states.

Once the raters flag a result, nothing happens immediately. Rather, they're used by Google's coding team and, in turn, its AI algorithms, to improve the search engine overall. Once all that kicks in, someone searching for history about the Holocaust will be less likely to run into a denial site, if things go as planned. However, determined searchers will still find such results if they specifically seek them out by naming a site, Google points out.

The company has used the new guidelines with select raters and updated its algorithm late last year. Now, searching with a query like "did the holocaust happen" no longer returns Stormfront as the top result and instead surfaces pages from the United States Holocaust Museum. Other queries still turn out questionable results, but Google told Search Engine Land it's "pleased" with the raters' work so far. "We will see how some of this works out," said Google engineer Paul Haahr. "We're learning as we go."