Latest in Gear

Image credit:

Bots on Amazon's task service may be souring psych studies

Mechanical Turk may need more safeguards to be used as a scientific tool.
Jon Fingas, @jonfingas
August 18, 2018
Share
Tweet
Share

Sponsored Links

Daniel Acker/Bloomberg via Getty Images

It's not uncommon for scientists to use Amazon's Mechanical Turk for research -- they can quickly gather survey data in return for a small outlay to participants. Academics are quickly discovering the limits of the task service with the modern bot-happy internet, however. Numerous psychology researchers have reported a sharp rise in the number of junk responses to their questions, many of which appear to come from bots. The offenders not only spam nonsensical answers to questions (such as "nice" or "good"), but tend to originate from the same location. And while anti-bot measures like captchas are in use, they aren't always reliable.

There's a chance that these could be people copying and pasting text, or using some form of automation to find the surveys (which Amazon allows). However, Amazon wouldn't provide evidence one way or the other. The company told Wired in a statement that it had "automated and manual mechanisms" to fight bots, but it wouldn't comment on bot activity levels or specific incidents.

Whatever the cause, the psychology study problems underscore deeper issues with Mechanical Turk. Bots are a reality on the service, and Amazon doesn't always appear to acknowledge them even after multiple reports. And then there's Mechanical Turk's low per-task pay. While that can help scientists worried that large rewards could skew results, it encourages completing tasks en masse rather than providing high-quality answers. It may be increasingly difficult to conduct meaningful surveys on Amazon's service, at least without tossing out many of the responses.

All products recommended by Engadget are selected by our editorial team, independent of our parent company. Some of our stories include affiliate links. If you buy something through one of these links, we may earn an affiliate commission.
Comment
Comments
Share
Tweet
Share

Popular on Engadget

Google's COVID-19 reports show where people are obeying stay-at-home orders

Google's COVID-19 reports show where people are obeying stay-at-home orders

View
Skype rolls out 'Meet Now' calls that don't need a a sign-up or installation

Skype rolls out 'Meet Now' calls that don't need a a sign-up or installation

View
Ten years in, a look at the iPad killers that weren't

Ten years in, a look at the iPad killers that weren't

View
Waymo’s fifth-generation Driver can peek around blind spots

Waymo’s fifth-generation Driver can peek around blind spots

View
'Artemis Fowl' will debut on Disney+

'Artemis Fowl' will debut on Disney+

View

From around the web

Page 1Page 1ear iconeye iconFill 23text filevr