Latest in Gear

Image credit: Daniel Acker/Bloomberg via Getty Images

Bots on Amazon's task service may be souring psych studies

Mechanical Turk may need more safeguards to be used as a scientific tool.
351 Shares
Share
Tweet
Share
Save

Sponsored Links

Daniel Acker/Bloomberg via Getty Images

It's not uncommon for scientists to use Amazon's Mechanical Turk for research -- they can quickly gather survey data in return for a small outlay to participants. Academics are quickly discovering the limits of the task service with the modern bot-happy internet, however. Numerous psychology researchers have reported a sharp rise in the number of junk responses to their questions, many of which appear to come from bots. The offenders not only spam nonsensical answers to questions (such as "nice" or "good"), but tend to originate from the same location. And while anti-bot measures like captchas are in use, they aren't always reliable.

There's a chance that these could be people copying and pasting text, or using some form of automation to find the surveys (which Amazon allows). However, Amazon wouldn't provide evidence one way or the other. The company told Wired in a statement that it had "automated and manual mechanisms" to fight bots, but it wouldn't comment on bot activity levels or specific incidents.

Whatever the cause, the psychology study problems underscore deeper issues with Mechanical Turk. Bots are a reality on the service, and Amazon doesn't always appear to acknowledge them even after multiple reports. And then there's Mechanical Turk's low per-task pay. While that can help scientists worried that large rewards could skew results, it encourages completing tasks en masse rather than providing high-quality answers. It may be increasingly difficult to conduct meaningful surveys on Amazon's service, at least without tossing out many of the responses.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.
Comment
Comments
Share
351 Shares
Share
Tweet
Share
Save

Popular on Engadget

'Gears 5' will add new modes and maps in its first six months

'Gears 5' will add new modes and maps in its first six months

View
Hulu's 'Castle Rock' season 2 teaser shows the origins of 'Misery'

Hulu's 'Castle Rock' season 2 teaser shows the origins of 'Misery'

View
YouTube is shutting down its TV-friendly web interface

YouTube is shutting down its TV-friendly web interface

View
SIM-based attack has been used to spy on people for two years

SIM-based attack has been used to spy on people for two years

View
Discord is pulling its subscription service's free games library

Discord is pulling its subscription service's free games library

View

From around the web

Page 1Page 1ear iconeye iconFill 23text filevr