The purpose of CUE-10 was to investigate how experienced professionals moderate usability tests.
Sixteen usability professionals each moderated three usability test sessions using the same six test tasks. Selected video recordings from the usability test sessions were analyzed by the participating professionals.
Practitioner’s Take Away
- Manage the limited time you have with the test participant well:
- Focus on the given tasks. Stop test participants if they explore beyond the given tasks.
- Proceed to the next step or the next task when the usability problem is clear.
- Don’t solicit lengthy opinions from test participants, and don’t get them into design discussions.
- Don’t continue the test beyond the time limits that were part of the recruiting and briefing without asking the participant’s permission.
- Be prepared! Conduct one or more pilot sessions to get familiar with the product, tasks and the moderator’s guide. Learn about issues that could come up so you don’t get surprised. Understand what errors and bugs you might encounter.
- Build trust and rapport during recruiting and throughout the entire test session. Get background information about the test participant. Reinforce the value of the test activity when the test participant self-blames. Compliment the test participants in non-trivial ways.
- Say as little as possible while the participant is carrying out tasks.
- Don’t encourage opinions in briefings or during moderation. Say “What are you thinking?” when the test participant is quiet, instead of intruding with a probing and sometimes leading question.
- Speak up if you think there is a problem with the brief or the tasks provided to you by the client.
- Learn from your peers. Several moderators said that they saw the incredible learning value of listening to the other moderators in the workshop.
- Take a critical look at yourself every now and again. Review your own moderated sessions. Show some of your moderation videos to independent peers and ask for honest feedback.
Papers About CUE-10
How Do Other People Do It? A Comparative Review of Usability Study Reports
by Danielle Cooley and Carol Barnum
User Experience Magazine, March 2019
How Professionals Moderate Usability Tests
by Rolf Molich, Chauncey Wilson, Carol Barnum, Danielle Cooley, Steve Krug, Chris La Roche, Beth A. Martin, Jonathan Patrowicz and Brian Traynor.
JUS, the Journal of Usability Studies, August 2020.
- CUE-10, Call for participation (9 pages, PDF, 97 KB)
- CUE-10, Rules for Usability Test Session and Usability Test Tasks
(21 pages, PDF, 584 KB)
- CUE-10, Anonymous Usability Test Reports (378 pages, PDF, 26 MB)
Fifteen of the 16 test reports from CUE-10 (team C asked that their report should not be made publicly available). Most of the reports are anonymous. When you study the reports, please keep in mind that the focus of CUE-10 was on moderation, not on the usability test reports.