Each of the 11 evaluators reported an average of 35 (16%) of the 220 problems. The overlap in reported problems between any two evaluators averaged only 9%. As many as 174 (79%) of the problems were reported by just one or two evaluators. Hence, the 11 inspections exhibit a substantial evaluator effect.

The evaluator effect would be less critical if severe problems were reported more consistently than cosmetic problems, which have little impact on a website’s usability. A problem was defined as severe if it appeared in one or more executive summaries. Each evaluator reported an average of 24% of the 33 severe problems. Seventeen (52%) of the severe problems were reported by just one or two evaluators. Hence, the evaluator effect persisted for severe problems.

The substantial differences in the individual reports stand in stark contrast to the perception the evaluators acquired during the group work. They left the group work with a strong, and reassuring, feeling of agreement. This became evident during the plenary session, as exemplified by the following quotes from five of the evaluators:

·      “I was surprised to see how little we disagreed.”

·      “A very high level of agreement.”

·      “It is not that subjective after all. There is consensus about what the problems are.”

·      “General agreement, but a number of concrete details differ.”

·      “We are all in agreement. We haven’t made the same observations, though.”

Nobody countered these statements.

Read more in our CUE-3 paper. You can download the paper using the link in the left column.