The following is a list of the most frequently occurring serious problems that we have observed in our evaluations of more than 100 practical CPUX-UT usability tests. The list is discussed in our paper, Improve Your Usability Tests: Learnings from Evaluating 100+ Usability Tests. The list was created by Bernard Rummel, Susanne Waßerroth and me.
Candidates who want to obtain the CPUX-UT certificate must pass both a theoretical and a practical exam. In the practical exam, candidates must demonstrate their ability to conduct a simple usability test of a website specified by the examiner, for example a weather website.
- Bad audio quality
It is difficult to hear what the test participant or the moderator are saying.
Use a separate microphone for the recording. Do not use the built-in microphone on a laptop. Check the quality of your audio by making a test recording. Also, check the audio quality after the first usability test session.
- Bad video quality
It is difficult to read the contents of the screen on the video.
Check the quality of your videos by making a test recording. Also, check the video quality after the first usability test session.
- Video too long – Test instructions not followed
One or more videos are longer than the prescribed 30 minutes, or a test session ends abruptly to keep it within 30 minutes. Good moderators carefully manage the precious time they have with the test participants.
- Incorrect test tasks – Test instructions not followed
One or more usability test tasks do not match the requirements in the task description for the practical examination, in particular the prescribed general tasks. The examiners consider it essential that a usability professional is able to follow precise instructions, such as those provided in the task description for the practical examination, exactly.
- Too many test tasks – Test instructions not followed
The task set contains more than four tasks. Exactly 4 tasks are required for the practical examination.
- One or more usability test tasks are trivial or have hazy goals.
Symptoms: All four tasks are completed within 10 minutes. It’s hard to identify more than 10 findings. Most of the findings are trivial.
- The moderator is not sufficiently curious
- The moderator talks too much
The moderator asks questions about issues that can be clearly observed. The moderator turns the usability test session into an interview, for example by repeatedly asking “What would you do?”
- The usability test report describes findings that are not evident from the videos
Such findings could be results from inspections or test participants’ opinions
- The usability test report does not describe important findings that are evident from the videos
The list is adapted from the list in UXQB’s CPUX-UT Checklist for the evaluation of the practical test