By:
Kayur Patel , Naomi Bancroft , Steven M. Drucker , James Fogarty , Andrew J. Ko , James A. Landay
- Kayur Patel has a BS in Computer Science and HCI from Carnegie Mellon, an MS in Computer Science from Stanford, another MS in Computer Science and Engineering from University of Washington and is currently working on his PhD in Computer Science and Engineering from University of Washington under James Fogarty and James Landay.
- Naomi Bancroft has an Undergraduate degree in Computer Science and Linguistics from University of Washington and is currently works for Google.
- Steven M. Drucker has a BSc in Neutral Sciences from Brown University, an MS in Brain and Cognitive Sciences, and a PhD in Arts and Media Technology both from MIT. He is currently a Principal Researcher for Microsoft Research.
- James Fogarty holds a BS in Computer Science from Virginia Tech and a PhD in HCI from Carnegie Mellon. He is currently an Assistant Professor of Computer Science and Engineering at University of Washington.
- Andrew J. Ko has a BS in Computer Science and Psychology from Oregon State and a PhD in HCI from Carnegie Mellon. He is currently an Assistant Professor at University of Washinton Information school and an Adjunct Assistant Professor in Computer Science and Engineering also at the University of Washington.
- James A. Landay has a BS in EECS from UC Berkeley along with an MS and PhD in Computer Science from Carnegie Mellon. He is currently a Professor of Computer Science and Engineering at the University of Washington.
Hypothesis
The Authors of this paper hypothesised that users would be able to do a better job of debugging code for the purposes of machine learning using their Gestalt Development Environment compared to more traditional methods of debugging. They felt the ability to implement a classification pipeline and then to analyze the data as it moves through the pipeline all the while being able to switch easily between the implementation and the actual analysis of the code.
Methods
For testing their hypothesis they selected a group of 8 participants that matched the target audience of Gestalt, and from there proceeded to have a set of baseline results and a set of Gestalt results. The baseline results involved the creation, modification and execution of scripts. Participants created visualizations by calling functions using a provided API which allowed the reproduction of all Gestalt Visualizations. Both baseline and Gestalt used the same data table structure however the baseline data table structure did not keep track of information generated across the pipeline. To further test the setup users were asked to solve two problems, one of sentiment analysis and another of gesture recognition. Both problems had bugs that replicated common programming errors introduced in it by the authors.
Results
The participants unanimously preferred using Gestalt for their troubleshooting purposes over baseline. They were able to find more bugs faster and similarly fix more bugs, quicker.
Contents
The authors of this paper spent considerable amount of time on their testing and had a rather elaborate set up to make the testing as complete and thorough as possible. They placed fair bit of importance on the data table structure and made it perfectly clear that it was most responsible for the effectiveness of Gestalt.
Discussion
Even though I'm not all that much into heavy programming even I can appreciate the usefulness of this. Debugging has historically been a pain for me and most people I know when it comes to programming and frankly any tool that can help make the torturous process easier is a tool to be worshipped. I felt the authors did a spectacular job of make sure their testing was fair and did a great job following a scientific process of testing comparing a set of control results with a set of experimental results. They also did a great job of stating at the very start what they were trying to do, proved that they were right and then stated exactly what their system could not do.
No comments:
Post a Comment