CUE-4 – Usability test vs. inspection

The purpose of CUE-4 was to compare the effectiveness and efficiency of usability testing and inspection techniques. The study showed that inspection is as effective and efficient as usability testing.

CUE-4 was a comparative usability evaluation of Hotel Pennsylvania’s website, www.hotelpenn.com, conducted in March 2003.

Seventeen professional teams simultaneously and independently evaluated the website’s usability. Nine teams used usability testing, and eight teams used their favorite inspection technique.

Overview of all CUE-studies

Practitioner’s Take Away

  • Usability testing isn’t the “high-quality gold standard” against which all other methods should be measured. CUE-4 shows that usability testing – just like any other method – overlooks some problems, even critical ones.
  • Inspections carried out by highly experienced practitioners can be quite valuable – and, according to this study, comparable to usability tests in the pattern of problems identified – despite their negative reputation.
  • Focus on productivity instead of quantity. In other words, spend your limited evaluation resources wisely. Many of the teams obtained results that could effectively drive an iterative process in less than 25 person-hours. Teams A and L used 18 and 21 hours, respectively, to find more than half of the key problem issues, but with limited reporting requirements. Teams that used five to ten times as many resources did better, but the additional results in no way justified the considerable extra resources. This, of course, depends on the type of product investigated. For a medical device, for example, the additional resources might be justified.

Read more in our “Comparative Usability Evaluation (CUE-4)” paper.

Inspections carried out by experienced practitioners are valuable and comparable to usability tests in the pattern of problems identified.

Papers about CUE-4

Joseph S. Dumas, Rolf Molich, and Robin Jeffries, “Describing Usability Problems – Are We Sending the Right Message,”
Interactions, July/August 2004, pp. 24-29.

Rolf Molich and Joseph S. Dumas,
“Comparative Usability Evaluation (CUE-4),”
Behaviour & Information Technology, Vol. 27, issue 3, 2008.

Rolf Molich, Robin Jeffries, and Joseph S. Dumas,
“Making Usability Recommendations Useful and Usable,”
Journal of Usability Studies, vol. 2, no. 4, August 2007.

For copies of these papers, please contact Rolf Molich.

Available Downloads

  • The proposal for the CHI2003 workshop, which describes the study’s background and rules (4 pages, PDF, 23 KB).
  • All 17 test reports in one PDF file (342 pages, PDF, 3,823 KB).
  • List of participating teams (1 page, PDF, 6 KB).
  • Client test scenario that each team received, describing the usability test task (12 pages, PDF, 47 KB).