CUE-1 caused considerable interest, but the results were also met with some skepticism from UX professionals. The purpose of CUE-2 was to confirm the results of CUE-1 in a larger, better controlled study. CUE-2 clearly confirmed the CUE-1 results.
CUE-2 was a comparative usability test of the popular www.hotmail.com website conducted in November and December 1998. Nine teams simultaneously usability tested the website.
Practitioner’s Take Away
CUE-2 offers the following recommendations for development teams, usability professionals, and their managers:
- Realize that there is no foolproof way to identify usability flaws. Usability testing by itself can’t develop a comprehensive list of defects. Use an appropriate mix of methods.
- Put less focus on finding “all” problems. Realize that the number of usability problems is much larger than you can hope to find in one or even a few usability tests. Choose smaller sets of features to test iteratively and concentrate on the most important ones.
- Realize that single tests aren’t comprehensive. They’re still useful, however, and any problems detected in a single professionally conducted test should be corrected.
- Increase focus on quality and quality assurance. Prevent methodological mistakes in usability testing such as skipping high-priority features, giving hidden clues, or writing usability test reports that aren’t fully usable.
Paper about CUE-2
Comparative Usability Evaluation
by Rolf Molich, Meghan R. Ede, Klaus Kaasgaard, and Barbara Karyukin,
Behaviour & Information Technology, vol. 23, no. 1, January/February 2004, pp. 65-74.
Contact Rolf Molich for a copy of this paper.
- The CHI99 extended abstract “Comparative Evaluation of Usability Tests” (2 pages, Word 97 format, 36 KB).
- All test reports and addendums in one self-unpacking zipped file (Word 97 and text formats, 730 KB).
- Each team received a client test scenario that defined the usability test task at the start of the test period (2 pages, PDF, 7 KB).
- A comparison of the strengths and weaknesses of each report (24 pages, PDF, 93 KB).