Usability testing

The picture shows the core of a usability test: one of several usability test sessions. In a usability test session, the usability test participant solves tasks on the interactive system while she thinks aloud. The moderator (left) observes and listens, takes notes and says as little as possible.



Usability test: A usability evaluation that involves representative users performing specific tasks with the interactive system to enable identification and analysis of usability problems, or the measurement of effectiveness, efficiency, and user satisfaction.

Usability test session: A part of a usability test where one test participant carries out representative usability test tasks using the interactive system or a prototype of the interactive system.

Results from a usability test are unique in one aspect: They show what representative users are able to accomplish with the interactive system when they carry out representative tasks. Eliciting personal opinions from users, or discussing them, does not support this objective and should be left to other methods.

A usability test shows what representative users are able to accomplish with the interactive system. Eliciting personal opinions from users, or discussing them, does not support this objective and should be left to other methods.

Strengths of usability tests

  • Usability testing makes it realistic for teams to test early and often, catching problems while it’s still easy to fix them.
  • Usability testing is great to demonstrate usability problems in your product to skeptical stakeholders, for example management and development teams
  • Usability testing can be simple to set up and conduct. No special equipment is required, just a test participant and a moderator. It is helpful if stakeholders can observe test sessions from a meeting room using a video connection.

Deviations from established practice

Here are ten deviations from established practice in usability testing that I often see:

  1. Test tasks are too simple
  2. Test tasks contain unintended clues
  3. Moderator helps the test participant too early
  4. Moderator explores the product together with the test participant
  5. Moderator manages the available time for the usability test session badly, for example by allowing the test participant to stray from the given task or by exceeding the time limit agreed with the test participant
  6. Moderator pays attention to test participants’ opinions rather than focusing on what they are actually able to accomplish
  7. Usability test reports include findings that are based on inspection rather than what the test participants did
  8. Usability test reports are unusable, because they are too long
  9. Usability test reports are unusable, because the most important findings are hard to find
  10. Usability test reports are unusable, because they are inconsistent, for example two reports written by the same person or by the same company have widely differing formats

My knowledge about these deviations is based on real-world data. The data comes from three sources

  • CUE-studies
  • Reviews of usability tests
  • Evaluating practical usability test certification exams (CPUX-UT)

For more information, read my blog post, Should usability testers be certified? (as you can guess, my answer is not just yes, but yes, Yes, YES!)

Learn more

There are many good textbooks about how to do usability testing. My favorite is

CPUX-UT Curriculum from the UXQB, 2020, about 80 pages
This curriculum describes much of my current knowledge about usability testing and usability inspection. This is the textbook for the usability testing certification from the User Experience Qualification Board, UXQB.
The curriculum is written in a compact style, with many examples and lots of practical advice. It comes with an extensive, free set of public exam questions that lets you test your understanding of the Curriculum.
The curriculum is freely available.
Disclosure: I work for the UXQB, and I am the editor of this curriculum together with Bernard Rummel from SAP. A lot of great and knowledgeable people contributed.

My other recommendations for practitioners are:

Carol Barnum: Usability Testing Essentials, 2011, about 400 pages
Richly illustrated with lots of examples

Steve Krug: Rocket Surgery Made Easy, 2010, about 165 pages
Written in an entertaining style. Some viewpoints are controversial