Dialog Design DialogDesign ved Rolf Molich English English Dansk Dansk
DialogDesign, Skovkrogen 3, 3660 Stenløse, Denmark, dialogdesign@dialogdesign.dk, +45 4717 1731
Free Resources CUE Products Events Books About DialogDesign
CUE Papers CUE-1 CUE-2 CUE-3 CUE-4 CUE-5 CUE-6 CUE-7

CUE Paper Summary

Rolf Molich, Meghan R. Ede, Klaus Kaasgaard, and Barbara Karyukin, “Comparative Usability Evaluation,” Behaviour & Information Technology, vol. 23, no. 1, January/February 2004, pp. 65-74.



This paper reports on a study assessing the consistency of usability testing across organisations. Nine independent organisations evaluated the usability of the same website, Microsoft Hotmail. The results document a wide difference in selection and application of methodology, resources applied, and problems reported.

The organisations reported 310 different usability problems. Only two problems were reported by six or more organisations, while 232 problems (75%) were uniquely reported, that is, no two teams reported the same problem. Some of the unique findings were classified as serious. Even the tasks used by most or all teams produced very different results - around 70% of the findings for each of these tasks were unique.

Our main conclusion is that our simple assumption that we are all doing the same and getting the same results in a usability test is plain wrong.



Back to the CUE papers page

About this paper

The paper is 10 pages long in PDF format. The file size of the electronic version is 138 KB.

The paper was accepted by Tom Stewart - the editor-in-chief of BIT - on August 12, 2003.

The December 2005 issue of the UI Design Newsletter from Human Factors International nominated this paper as "if you only read one paper about usability this year, read this one."

Questions? Comments? Requests for e-prints?
Write to

Home    Free Resources    CUE    Products    Events    Books    About DialogDesign