What you get is what you see: revisiting the evaluator effect in usability tests

Morten Hertzum, Rolf Molich and Niels Ebbe Jacobsen

Behaviour and Information Technology, 2014, vol. 33, issue 2, 144-162

Abstract: Usability evaluation is essential to user-centred design; yet, evaluators who analyse the same usability test sessions have been found to identify substantially different sets of usability problems. We revisit this evaluator effect by having 19 experienced usability professionals analyse video-recorded test sessions with five users. Nine participants analysed moderated sessions; 10 participants analysed unmoderated sessions. For the moderated sessions, participants reported an average of 33% of the problems reported by all nine of these participants and 50% of the subset of problems reported as critical or serious by at least one participant. For the unmoderated sessions, the percentages were 32% and 40%. Thus, the evaluator effect was similar for moderated and unmoderated sessions, and it was substantial for the full set of problems and still present for the most severe problems. In addition, participants disagreed in their severity ratings. As much as 24% (moderated) and 30% (unmoderated) of the problems reported by multiple participants were rated as critical by one participant and minor by another. The majority of the participants perceived an evaluator effect when merging their individual findings into group evaluations. We discuss reasons for the evaluator effect and recommend ways of managing it.

Date: 2014
References: Add references at CitEc
Citations:

Downloads: (external link)
http://hdl.handle.net/10.1080/0144929X.2013.783114 (text/html)
Access to full text is restricted to subscribers.

Related works:
This item may be available elsewhere in EconPapers: Search for items with the same title.

Export reference: BibTeX RIS (EndNote, ProCite, RefMan) HTML/Text

Persistent link: https://EconPapers.repec.org/RePEc:taf:tbitxx:v:33:y:2014:i:2:p:144-162

Ordering information: This journal article can be ordered from
http://www.tandfonline.com/pricing/journal/tbit20

DOI: 10.1080/0144929X.2013.783114

Access Statistics for this article

Behaviour and Information Technology is currently edited by Dr Panos P Markopoulos

More articles in Behaviour and Information Technology from Taylor & Francis Journals
Bibliographic data for series maintained by Chris Longhurst ().