Cheatsheet: Usability Test Metrics

Overview of most common metrics in Usability Tests

This article covers the most common usability metrics and should give readers a short and crisp overview of the metrics which are commonly used in usability test.

If you are planning to conduct a usability test you can use this as a frame to design your study and plan along the main components of usability, which are effectiveness, efficiency and satisfaction. In the following content we’ll take a brief look at the different metics.

Effectiveness

Effectiveness focusses on the accuracy & correctness: Are users able to achieve their goals?

Task Completion/Success Rate (Binary success)

Binary Success means, that either a user passes or fails a certain task. So when a user receives a hint or needs help, the task will be marked as failed, although the user might be able to complete the task after a moderator gives him the hint.

• Success: User is able to complete the task with all subtasks successfully performed.
• Failure: The user is not able to complete the task

Example: A usability test participant has the task to order flowers on a website; he selects during the process the wrong flowers and the wrong date, but proceeds to the checkout. Within this method, the task would be unsuccessful, as he ordered the wrong flowers and put in a wrong date.

Task Completion/Success Rate (Levels of Success)

Levels of success describes a method where we grant a user partial success if he completes a task.

• complete success: the user completes the task as expected
• success with one minor issue: The user places the order but orders the wrong item or forgot a minor step
• success with a major issue: The user places the order but enters the wrong date or delivery address
• failure: The user is not able to complete the task

Example: The user needs to order flowers on a website; he selects during the process the wrong flowers, but proceeds to the checkout and successfully orders. With “Levels of success” we give the user the credit of completing the order, although the user ordered the wrong flowers, but the process of ordering was correct.

Efficiency

Effectiveness focusses on the ease of use & speed: How much effort which is required from a user in order to achieve their goals?

Time on Task

Time on task is the most common efficiency measurement and relatively easy to measure. It counts the time that a user is attempting a tasks. This can be either done by one of the notetakers or

Be careful: While using a “concurrent thinking aloud protocol” (CTA) technique during the test can lead to false results. If you want to measure time use a “retrospective think aloud protocol” (RTA) technique.

Action Based Efficiency

With “Action Based Efficiency” is meant that you count certain actions a user does to complete a task. This can be literally anything, but the most common ones are for example: Mouseclicks & Keystrokes, Tabs (on mobile devices), Movements of the body (e.g. hands, etc.).

Lostness Factor

The Lostness Factor is a metric that compares the user’s path relative to the most efficient path. For this metric, you need to know what the minimum number of screens/steps are required to complete the task.

L = Lostness, N = Number of unique pages visited during the task, S = Total number of pages visited during the task (including re-visits to the same page), R = The optimal path (minimum number of pages that need to be visited to complete the task)

Validating a Lostness Measure

Satisfaction

Satisfaction focusses on the user’s perception: What do users think about their experience?

Satisfaction: Task Level

Single Ease Question (SEQ)

The single ease questions is a 7-scale question which can be used after each tasks of a usability test. It collects the perception of the difficulty of the task.

Overall, how difficult or easy was the task to complete?
1 = Very difficult
7 = Very easy

Single Ease Question (SEQ)

NASA TLX

The Nasa TLX is designed to measure the perceived workload of a physical or digital product. By workload is meant the effort a user needs to undertake (mentally & physically) to use a certain product.

TLX @ NASA Ames - Home

Satisfaction: Product Level

System Usability Scale (SUS)

The System Usability Scale is a fast approach to measure subjectively the overall usability of system. The SUS comes in handy when a system needs to be assessed and can be used as a “quick and dirty” indicator. The SUS allows to assess a wide variety of products, such as hardware, software, mobile applications, websites, etc.

System Usability Scale (SUS)

User Experience Questionnaire (UEQ):

The UEQ aims to cover different aspects regarding user experience. Those metrics are:

• Attractiveness
• Perspicuity
• Efficiency
• Dependability
• Stimulation
• Novelty

User Experience Questionnaire