by Kevin Dodds

In crowdsourcing, cheating is a common problem, and it’s a two-way street: requesters can manipulate and deceive workers just as workers can con a requester. Everyone involved needs to be creative and cautious to avoid getting taken.

For example, many crowdsourcing managers use looping—asking a question multiple times to generate a consensus answer—to validate answers. Usually, two matching answers out of three creates a consensus. If a group of workers deliberately enters the same false data point for the same question until they generate a consensus, though, the HIT is closed, and all get paid for a wrong answer. Technology can help prevent this type of attack.

• Code the answer field to not accept spaces or any string of text less than a specific number of characters
• Set the field up to only accept a specific data format such as a URL, an email, a number, a date, etc., and automatically reject answers in other formats

Another common crowdsourcing scam is the repeated use of “N/A” or “no answer” as a response. “N/A” can be a legitimate data point. When researching whether or not a company has an official web site, for example, a worker either enters the correct web address or selects “N/A” if none exists. Unfortunately, dishonest workers can just enter “N/A” for hundreds or thousands of records without doing any research. To neutralize this threat, a crowdsourcing manager can:

• Keep campaign run sizes small to minimize the impact of fraudulent N/As
• Monitor the results of each campaign immediately as it ends to check for unusual numbers of N/A responses as compared to an average run

For requesters, vigilance is key. It’s best to gather an honest group of hard-working people by constantly measuring their accuracy against known answers and by granting and removing the qualification to participate in a campaign as needed.

On the other side of the equation, requesters can literally rob workers of their pay. A dishonest crowdsourcing manager can:

• Gather worker responses and then “reject” the responses without cause
• Pay one cent—essentially nothing—for complex tasks*
• Start a campaign at a high pay rate and then drop the rate during the campaign run

Dishonest requesters should note that crowd workers are a tightly knit digital community and communicate about good and bad requesters openly. Scamming is inevitable, but no one— requester or worker—should be able to get away with it for very long before

being found out.

* A low-paying task is not necessarily bad. Some are very easy to do and can increase your HIT count quickly. Workers just have to be cautious and selective about the work they choose to do.

{ 0 comments }

posted by Shyamali Ghosh on March 24, 2014