Backtracking Events as Indicators of Software Usability Problems

Creation-oriented software applications such as photo editors and word processors are often difficult to test with traditional laboratory usability testing methods. A diversity of creation goals and strategies results in a diversity of usability problems encountered by users. This diversity in problems translates into the need for a large pool of participants in order to identify a high percentage of the problems. However, recruiting a large pool of participants can be prohibitively expensive, due to the high costs of traditional, expert-moderated think-aloud usability testing. To address this problem, this dissertation contributes a new usability evaluation method called backtracking analysis, designed to automate the process of detecting and characterizing usability problems in creation-oriented applications. The key insight is that interaction breakdowns in creation-oriented applications often manifest themselves in simple backtracking operations that can be automatically logged (e.g., undo operations, erase operations, and abort operations). Backtracking analysis synchronizes these events to contextual data such as screen capture video, helping the evaluator to characterize specific usability problems. The thesis of this dissertation is that backtracking events are effective indicators of usability problems in creation-oriented applications, and can yield a scalable alternative to traditional laboratory usability testing. The investigation of this claim consists of five parts. First, a set of experiments demonstrate that it is possible to extract usability problem descriptions from backtracking events without the aid of a human test-moderator, by pairing participants during an automated retrospective interview. Second, a within-subjects experiment with the Google SketchUp 3D modeling application shows that backtracking analysis is comparable in effectiveness (number and severity of usability problems identified, and percentage of false alarms) to the user-reported critical incident technique, a cost-effective usability evaluation method that relies on participants to report their own difficulties. Third, another experiment generalizes this result to the Adobe Photoshop application. Fourth, this dissertation contributes a theory to help explain the influence of task design on the effectiveness of backtracking analysis. Finally, to situate backtracking analysis within usability evaluation practice, a between-subjects experiment explores the strengths and weaknesses of backtracking analysis compared to traditional think-aloud usability testing.