Questions and answers about usability testing

This page contains questions and answers from my talks about usability testing. For more questions and answers, see Questions and answers about the CUE studies.


Overview of this page


Getting into UX research

QUESTION: For someone who is trying to break into the UX research field, how would you suggest we practice and develop research skills?

ANSWER: The answer depends on your skills and enthusiasm. Ideally, study some of the great resources in our field or take an introductory course. Then practice, practice, practice, for example by conducting interviews and usability tests, which you record on video. Get a mentor—and check that the mentor is respected by independent UX professionals. Show the videos to the mentor, get feedback and learn from the feedback. Get feedback on your reports to insure that they are usable for your difference audiences.

Fundamental skills are asking good questions, probing to get at core issues, and listening carefully to your research participants. There are excellent books that deal with these skills, for example, Steve Portigal’s book Interviewing Users: How to Uncover Compelling Insights.


Briefing

QUESTION: What are the best ways to gain rapport when you first meet the test participant and brief them on the study?  What kinds of conversation might you want to avoid during the introductory briefing?

ANSWER: See the discussion in the CUE-10 article, section “Building Trust and Rapport.” The general rule is to keep the briefing and the interview short, unless the test participant is nervous. If the test participant is nervous, offer a cup of coffee and some small talk, and start with a very simple test task so the test participant will experience a quick success.


Think aloud

QUESTION: Should I ask a test participant to think aloud a little less when I observe that the think aloud is interfering with other cognitive tasks? It’s a rather complicated task at that too.

ANSWER: I did not observe think aloud that interfered with other cognitive tasks in CUE-10. No moderator asked test participants to think aloud a little less. We did see a few examples of the opposite: A few moderators asked test participants to think aloud even though they were thinking aloud all the time.

For situations where think aloud might interfere with cognitive work (say, the work of a flight controller), a retrospective usability testing approach might be appropriate. In this method, you ask the test participants to watch their test session and think aloud about what they did. The moderator can stop the video and probe for what the test participant was doing and thinking.


Encouragement

QUESTION: I find that providing encouragement to test participants (“great job”, “you did it”, etc.) leads them to think there are right & wrong answers, and to seek the moderator’s approval on their answers. How do you provide encouragement without influencing their subsequent answers?

ANSWER: “Great job” and “you did it” is trivial encouragement. In the CUE-10 article, the section “Complimenting Test Participants” contains examples of elegant and non-trivial encouragement.

Too much encouragement during a task could backfire on the moderator from an ethical perspective. If you say “great job” early in a test session and then the participant begins to struggle in a more difficult task, they might start to feel anxiety because things aren’t “great” anymore. Too much encouragement could lead to an emotional letdown in the participant.

I am not worried about encouragement influencing test participants’ subsequent answers. Usability test participants’ answers and opinions are of less importance. See the CUE-10 article, section “Asking Test Participants for Their Opinions,” which says: “A usability test shows what representative users are able to accomplish with the interactive system when they carry out representative tasks. Eliciting personal opinions from users, or discussing them, is not part of a usability test (UXQB, 2020).” Participants’ opinions are important in a user survey.


QUESTION: I try to say “Great”, “Thank you” after each task, but not say too much. What percent of the time should Moderator be talking, besides reading tasks?

ANSWER: Ideally, the Moderator should keep quiet during task solution.

“Great” and “Thank you” are trivial compliments. See the CUE-10 article, section “Complimenting Test Participants”, which contains examples of elegant and non-trivial compliments.

According to the CUE-10 article, Table 4, six moderators were “talkative” while seven were “non-talkative”. Talkative style resembles a conversation between the moderator and the test participant; non-talkative style is when moderators limit themselves to confirmations, occasional assists, limited prompts, and clarifying questions.

Personally, I strongly favor the non-talkative style because the talkative style increases the chance that the moderator provides inadvertent clues.


Opinions

QUESTION: Why do you speak out against opinions? I’ve often found them a good entry point into the user’s needs and emotional reactions?

ANSWER: Usability tests are not for studying users’ opinions or needs or their emotional reactions. If you want to study users’ needs, do an interview or a focus group. If you want to study users’ opinions or emotional reactions, do a user survey, an interview or a focus group.

The CUE-10 article addresses this issue in the section “Asking Test Participants for Their Opinions,” which says: “A usability test shows what representative users are able to accomplish with the interactive system when they carry out representative tasks. Eliciting personal opinions from users, or discussing them during task solution, is not part of a usability test (UXQB, 2020).” You may ask for opinions during debriefing.


Leading the user

QUESTION: What are some examples of “leading the user” during a usability test?

ANSWER: Examples of leading questions from moderators during task solution in the CUE-10 study are

  • Are you confused by that label?
  • Did you find the European time format confusing?
  • What were your thoughts on when you went back to change that, would you expect the details you entered to stay there or be cleared to start again?
  • What do you think of the bar on the top that scrolls?
  • So it does sound like from what I hear you say, it is possible to make the change, at least through customer service?