The purpose of CUE-5 was essentially the same as the purpose of CUE-4: To compare the effectiveness and efficiency of usability testing and inspection techniques.
CUE-5 was a comparative usability evaluation of the PAX Wardrobe planning tool on IKEA’s website, www.ikea-usa.com. The evaluation was conducted in October 2005 as part of a Usability Practicum in connection with the UI10 conference in Boston, Massachusetts.
Thirteen professional teams simultaneously and independently evaluated the website’s usability. Five teams used usability testing (two of these teams used remote usability testing), and eight teams used their favorite inspection technique.
Practitioner’s Take Away
- There are considerable differences in the number of key issues that the various teams report.
A “key issue” is an important usability issue on which many teams agree. For a more precise definition and discussion of this quality measure see our CUE-4 paper. - The number of hours used for the evaluations seems to correlate weakly with the number of key issues reported, but there are remarkable exceptions.
- Teams that use inspection use fewer resources on the evaluation and in general report fewer key issues, but in general their results are as good as usability test results.
- The teams reported surprisingly few positive issues, and there was no general agreement on them. Many positive issues were reported by single teams only. You might ask whether the PAX Planner is really that bad, or if usability professionals are reluctant to report positive findings.
Available Downloads
- The CUE-5 scenario (6 pages, PDF, 79 KB).
- The proposal for the CHI2006 conference Special Interest Group (SIG) event “Tips and Tricks for Better Usability Test Recommendations,” which is based on the results of CUE-5 (4 pages, PDF, 79 KB).
The evaluation reports from this study are not publicly available.