CUE-2 – Confirm the results of CUE-1

CUE-1 caused considerable interest, but the results were also met with some skepticism from UX professionals. The purpose of CUE-2 was to confirm the results of CUE-1 in a larger, better controlled study. CUE-2 clearly confirmed the CUE-1 results.

CUE-2 was a comparative usability test of the popular www.hotmail.com website conducted in November and December 1998. Nine teams simultaneously usability tested the website. 

Overview of all CUE-studies

Practitioner’s Take Away

CUE-2 offers the following recommendations for development teams, usability professionals, and their managers:

  • Realize that there is no foolproof way to identify usability flaws. Usability testing by itself can’t develop a comprehensive list of defects. Use an appropriate mix of methods.
  • Put less focus on finding “all” problems. Realize that the number of usability problems is much larger than you can hope to find in one or even a few usability tests. Choose smaller sets of features to test iteratively and concentrate on the most important ones.
  • Realize that single tests aren’t comprehensive. They’re still useful, however, and any problems detected in a single professionally conducted test should be corrected.
  • Increase focus on quality and quality assurance. Prevent methodological mistakes in usability testing such as skipping high-priority features, giving hidden clues, or writing usability test reports that aren’t fully usable.

Paper about CUE-2

Comparative Usability Evaluation
by Rolf Molich, Meghan R. Ede, Klaus Kaasgaard, and Barbara Karyukin, 
Behaviour & Information Technology, vol. 23, no. 1, January/February 2004, pp. 65-74.

Contact Rolf Molich for a copy of this paper.

Available Downloads

  • The CHI99 extended abstract “Comparative Evaluation of Usability Tests” (2 pages, Word 97 format, 36 KB).
  • All test reports and addendums in one self-unpacking zipped file (Word 97 and text formats, 730 KB).
  • Each team received a client test scenario that defined the usability test task at the start of the test period (2 pages, PDF, 7 KB).
  • A comparison of the strengths and weaknesses of each report (24 pages, PDF, 93 KB).