Paper Reading #6: TurKit: Human Computation Algorithms on Mechanical Turk

TurKit: Human Computation Algorithms on Mechanical Turk

By:
Greg Little, Lydia B. Chilton, Max Goldman, Robert C. Miller

 Presented at UIST 2010

• Greg Little has an Undergraduate from Arizona State University and a PhD from MIT. He currently works at MIT.
• Lydia B. Chilton has an Undergrad from MIT in Economics and EECS. She also has a Masters from MIT also in EECS. She is currently a Graduate student at the University of Washington.
• Max Goldman is currently a Graduate student at MIT.
• Robert C. Miller has a Bachelor of Science and a Masters of Engineering from MIT along with a PhD from Carnegie Mellon. He is currently an Associate professor at MIT.

Summary
Hypothesis

The authors of this paper hypothesised that the use of TurKit would allow for better coding of programs requiring Human Computation. The ability to make MTurk calls as just function calls is supposed to allow users to utilise more of their traditional approaches to programming, allow the functions to be used as building blocks for more complex algorithms while allowing the code to be more traditionally readable. The use of the Crash-and-rerun programming model reduces the expense of Human Computation by allowing the ability to run multiple times without repeating computationally expensive work.

Methods
To evaluate the performance of their they used the results of 20 different experiments run over the period of year using TurKit. The experiments included iterative writing, blurry text recognition, website clustering, brainstorming, and photo sorting.

Results
The 20 experiments resulted in an expense of $364.85 involving 29.731 assignments involving HITs. This showed that TurKit resulted in faster computation times. The fact that waiting for human action takes an order of magnitude longer than computation, it's shown that TurKit is suitable for most applications.

Contents
The authors showed TurKit to be greatly useful, however admitted that it wasn't scalable. They tested it thoroughly, implemented it in the best possible format, however it proved itself to be un-scalable which made it less impressive.

Discussion
While I didn't quite understand everything about human computation, I did understand a fair about it. To me anything that helps things run faster and more efficiently is a step in the right direction. Using Crash-and-Rerun seemed ingenious considering it completely reduced a redundancy in the calls. The fact that TurKit has great usability and allows exceptional ease for writing scripts.