Jul 5, 2013

CrowdFlower vs. Amazon Mehanical Turk for Crowdsourcing Tasks

I have used both CrowdFlower and Amazon's Mechanical Turk (AMT) for several tasks including judging the sentiment in a sentence, obtaining relevance judgments, writing summaries of texts, and judging quality of machine generated summaries.

The main difference between CrowdFlower and Amazon's Mturk is that CrowdFlower has a lot more channels (venues) where you can publish your tasks compared to AMT which is just a single standalone channel (not sure if things have changed since I last used it). This covers big micro task sites like Amazon Mechanical Turk to smaller channels like Zoombucks, Probux, Getpaid, FusionCash etc. So rest assured your tasks are completed within a few days even if you have thousands of them.

Another  nice thing about CrowdFlower that AMT does not have is a higher degree of quality control on the results. CrowdFlower has hidden questions called Gold Standard (provided by the requester), which has contributors (workers) perform tasks that have already been completed in order to determine their accuracy and trustworthiness. This helps avoid task spamming and eliminates contributors that are not serious on the tasks.

In terms of ease of use, I think CrowdFlower has a nicer user interface but the learning curve in using both  these Crowd Sourcing applications is about the same. It might take you a day or two to get used to it, but once you get then hang of it...you can't have enough of it as your results come back so fast, and hopefully not all are spam! Did I miss any other major differences? Another thing to consider is to have multiple workers on one task, so that you can use majority voting for as the final answer.

So what can you use crowd sourcing for? For a lot of things! Determining the right category of an item, circling out the sentiment words in sentences, determining the correct sense of a word in text, summarizing long documents, finding websites on a specific topic are just a few out of the many tasks you can have people complete for you. 

More Info:

No comments:

Post a Comment