Latest in Science

Image credit: Getty Images/iStockphoto

Typos are kryptonite to Alphabet's anti-trolling API

It's supposed to detect toxic language online, but it has some flaws.
456 Shares
Share
Tweet
Share
Save

Sponsored Links

Getty Images/iStockphoto

"Don't read the comments" is a cardinal rule of the internet. They're often hotbeds of toxicity and abuse, and rarely does a person come away from them feeling enlightened. Jigsaw, a subsidiary of Alphabet, is working to combat this problem through a project called Perspective, an API that uses machine learning to spot harassment online. But, researchers have discovered that it's easy to game the system.

Perspective assigns a "toxicity score" to comments based on the perceived impact they might have on a conversation. Type the sentence, "It's stupid and wrong," for example, and Perspective might rate it 89 percent toxic. Researchers at the University of Washington's Network Security Lab found they could trick the API into consistently lowering the toxicity score, however, by subtly modifying phrases. They added intentional misspellings ("iidiot" instead of idiot) and inserted punctuation into words ("stu.pid" or "s c r e w"). They also discovered that a benign phrase like "It's not stupid and wrong" scored almost as high as the abusive one.

In a statement first reported by Ars Technica and confirmed to Engadget, Perspective's project manager, CJ Adams, praised the study:

It's great to see research like this. Online toxicity is a difficult problem, and Perspective was developed to support exploration of how ML can be used to help discussion. We welcome academic researchers to join our research efforts on Github and explore how we can collaborate together to identify shortcomings of existing models and find ways to improve them.

Perspective is still a very early-stage technology, and as these researchers rightly point out, it will only detect patterns that are similar to examples of toxicity it has seen before. We have more details on this challenge and others on the Conversation AI research page. The API allows users and researchers to submit corrections like these directly, which will then be used to improve the model and ensure it can to understand more forms of toxic language, and evolve as new forms emerge over time.

It looks like websites like Engadget will be waiting a while before unleashing Perspective on our comments sections.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.
Comment
Comments
Share
456 Shares
Share
Tweet
Share
Save

Popular on Engadget

Engadget's Guide to Privacy

Engadget's Guide to Privacy

View
'Dragon Ball Z: Kakarot' arrives on January 17th, 2020

'Dragon Ball Z: Kakarot' arrives on January 17th, 2020

View
Nintendo is holding an online 'Mario Kart 8 Deluxe' tournament this Sunday

Nintendo is holding an online 'Mario Kart 8 Deluxe' tournament this Sunday

View
LinkedIn's new quizzes can prove you're not lying on your resume

LinkedIn's new quizzes can prove you're not lying on your resume

View
Rockstar rolls out its own PC games launcher

Rockstar rolls out its own PC games launcher

View

From around the web

Page 1Page 1ear iconeye iconFill 23text filevr