CUE (Comparative Usability Evaluation), English

In each CUE study, a considerable number of professional usability teams independently and simultaneously evaluate the same website, web application, or Windows program.

The main purpose is to collect data on a series of questions:

· What's common practice?

What usability evaluation methods and techniques do professionals actually use? Do experienced professions ever shun methods or techniques, even those that receive a lot of coverage?

· Are usability evaluation results reproducible?

· What is a "serious" or "critical" usability problem?

· How many usability problems are there?

What’s the order of magnitude of the total number of usability problems that you can expect to find on a typical, nontrivial website?

· How many test participants are needed?

How many test participants are required to find most of the critical problems?

· Quality differences.

Are there important quality differences between the results the teams obtained?

· What’s the return on investment?

If you invest more time in a usability evaluation - for example, 100 hours instead of 25 - will you get substantially better results?

· Usability test versus Expert review.

How do professional usability testing and expert reviews compare?

DialogDesign and other usability experts use the results from these questions to advise the usability community on quality approaches to usability evaluation.

DialogDesign conceived and managed the CUE studies described in more detail below.

The Nine CUE Studies

CUE-1 - Four teams usability tested the same Windows program, Task Timer for Windows

CUE-2 - Nine teams usability tested www.hotmail.com

CUE-3 - Twelve Danish teams evaluated www.avis.com using expert reviews

CUE-4 - Seventeen professional teams evaluated www.hotelpenn.com (nine teams with usability testing and eight teams with expert reviews)

CUE-5 - Thirteen professional teams evaluated the IKEA PAX Wardrobe planning tool on www.ikea-usa.com (six teams with usability testing and seven teams with expert reviews)

CUE-6 - Thirteen professional teams evaluated the Enterprise car rental website, www.Enterprise.com (10 teams with usability testing, six teams with expert reviews, and three teams with both methods)

CUE-7 - Nine professional teams provided recommendations for six nontrivial usability problems from previous CUE-studies

CUE-8 - Seventeen professional teams measured key usability parameters for the Budget car rental website, www.Budget.com

CUE-9 - A number of experienced usability professionals independently observed five usability test videos, reported their observations and then discussed similarities and differences in their observations (the "Evaluator Effect")

The results are discussed in more detail in the refereed CUE papers.

Questions or comments about CUE? Write to dialogdesign@dialogdesign.dk

The Four Most Important CUE Findings

· The number of usability problems in a typical website is often so large that you can’t hope to find more than a fraction of the problems in an ordinary usability test.

· There’s no measurable difference in the quality of the results produced by usability tests and expert reviews.

· Six - or even 15 - test participants are nowhere near enough to find 80% of the usability problems. Six test participants will, however, provide sufficient information to drive a useful iterative development process.

· Even professional usability evaluators make many mistakes in usability test task construction, problem reporting, and recommendations.

These findings and their implications are discussed in the CUE papers and in our courses.

Home Free Resources CUE Products Events Books About DialogDesign