Paper Reading #21: Human Model Evaluation in Interactive Supervised Learning

Human Model Evaluation in Interactive Supervised Learning


Chi '11

By:
Rebecca Fiebrink, Perry R. Cook and Daniel Trueman.

• Rebecca Fiebrink has a PhD in Computer Science from Princeton University and a Post Doctoral from the University of Washington. She is Currently an Assistant Professor in the department of Computer Science at Princeton University.
• Perry R. Cook is Professor Emeritus at Princeton University in the department of Computer Science and Music.
• Daniel Trueman is a musician that teaches composition at Princeton University.

Summary
Hypothesis
The authors of this paper tried to figure out precisely what model for machine learning users prefer most so as to be able to figure out precisely what criteria is most preferred by the users.

Methods
To test their theory the authors of this paper conducted three studies. One study revolved around the improvement of the Wekinator, where participants met frequently to the uses of the wekinator in their areas of study and improvements needed. In a second study users were asked to create a system that used the wekinator to take in input gestures to create a musical performance system. The final study revolved around gathering data from professional musicians so as to build a sensor equipped cello bow so as to get a gesture classifier.

Results
The users in the first study found the sound control algorithm difficult to use and felt that it resulted in musically unacceptable sounds. The second and third study used cross validation which resulted in higher validation accuracy, which in turn was indicative of higher musical performance. Users in third study used cross validation as a quick check.

Contents
The paper revolves around machine learning and model optimisation. The users of the studies used direct validation far more frequently than cross validation (which was later determined to be the more complex but more accurate form of validation). The direct validation broke down into six key areas: Accuracy, Cost, Decision Boundary Shape, Label Confidence and Posterior Shape, Complexity and Unexpectedness.

Discussion
This paper wasn't easy to understand and I most certainly didn't get all of it. While the concept of Machine Learning and Model Optimisation is now a little more clear, I'm still not a completely certain as to what on earth is going on here. That being said I was very impressed by the tests run and the methodology used by the authors of this paper. It was very thorough and covered all questions that needed answering.