CUE Paper Summary
Rolf Molich, Meghan R. Ede,
Klaus Kaasgaard, and Barbara Karyukin, “Comparative Usability Evaluation,”
Behaviour & Information Technology, vol. 23, no. 1, January/February 2004,
This paper reports on a study assessing the consistency of
usability testing across organisations. Nine independent organisations evaluated
the usability of the same website, Microsoft Hotmail. The results document a
wide difference in selection and application of methodology, resources applied,
and problems reported.
The organisations reported 310 different usability problems.
Only two problems were reported by six or more organisations, while 232 problems
(75%) were uniquely reported, that is, no two teams reported the same problem.
Some of the unique findings were classified as serious. Even the tasks used by
most or all teams produced very different results - around 70% of the findings
for each of these tasks were unique.
Our main conclusion is that our simple assumption that we are
all doing the same and getting the same results in a usability test is plain
Back to the CUE papers page
|About this paper|
The paper is 10 pages long in PDF format. The file size of the electronic
version is 138 KB.
The paper was accepted by Tom Stewart - the editor-in-chief of BIT - on August
The December 2005 issue of the UI Design Newsletter from Human Factors
International nominated this paper as "if you only read one paper about
usability this year, read this one."
Questions? Comments? Requests