Craft good questions for usability tests

A short guide to gather more insightful feedback

Adding additional questions and surveys to your test task can enrich the feedback you gather through a usability test. They can give more context to your findings and help to understand the study team why some tasks might have been tricky to solve or have not been completed at all.

Furthermore, they help to ease the usability test situation while conduction a test, but can also be useful when a participant can collect and verbalize their thoughts, opinion and possibly further ideas after the test.

Some general rules for asking questions:

• Listen more than talk
• Let participants speak out about their pain points, their needs and what makes their life easier
• Don’t question the opinions of your participants. They are the master, you are the apprentice
• Avoid questions that can be answered by a “yes” or “no”. Ask open questions
• Don’t ask leading questions, because they tend to be force a user to give you a biased feedback
• Give your participants a moment to think about their answers and stand the silence

Screening Questions

For selecting the participants you need to define a set of criteria about your target users. This can be for example their job role, habits or other criteria you may define based on your persona.

It is helpful to ask potential subjects screening questions to determine if they actually fit into your target audience for the usability test. It prevents you to include people that may not be your target audience and will help to gather more valuable feedback from the users that actually may use your product, service or software in the future.

Pre-Test Questions

Before a usability test starts, you want to learn a bit more about the user’s experience, attitude towards the product and their background (education, job experience, etc.). This can be done through a survey or right before the test (which I personally prefer), because they are also a good ice-breaker for starting with the usability test and ease a bit tension from participants.

Following categories are good for pre-test questions:

• Demographics
  For example age, occupation, education, etc.
• Understand their current approach to solving a problem
  For example: “How do you currently solve the problem the [product, service or software] seeks to solve?”
• Prior knowledge (either in their job, of a service/product/software, etc.)
  For Example: “What do you use the [product, service or software] for?”; “Which [feature, workflow, etc.] do you use the most?”
• Attitude towards the product, service or software
  For Example: “What do you like most about the [product, service or software]?”; “What do you like least about the [product, service or software]?”

In-Test Questions

A Usability Test can be designed in different ways. Depending on your study design also In-test questions can be a good tool to gather a bit more feedback about a specific topic. For example can be a set of (sub-)task cover a certain part or topic about a product or service.

There are several options how to integrate In-test questions:

• You might add the SEQ (single ease question) after a single task
  Example: “Overall, how difficult or easy was the task to complete?”
• Additional questions around the specific task(s) and it’s functionality
  For example: “How would you imagine [something] could be improved in the future?”; “What prevents you from completing a task?”, etc.
• Other related questions to a specific task
  For example: Can you give me a few examples of real situations when [something] would be helpful?

Post-Test Questions

Questions (through an interview setting or a survey) after the usability test focus now on the product, service or software itself.

• System Usability Scale
  Asking the 10 question survey through a form can give you insights about the usability.
• Questions around the product, service or software:
  For Example:
  - “If you could change one thing about [product, service or software], what would it be and why?”
  - “What do you expect to see in our product in the future?”
  - “What was the best/worst thing about the [product, service or software]?”
  - “How would you compare [product, service or software] to [competitor]?”