A better usability severity scale
Friction, Frustration, Blocker, Disaster
In an ideal world, usability and accessibility testing would be baked into every project plan from the start, and we’d all do multiple rounds of test-iterate-test before releasing anything.
In reality, this is beyond the capacity of many teams due to things like limited resources or low UX maturity. Instead they might do one or two big studies then try to fix as much as they can.
Over the last 18 months I’ve worked with a lot of teams in this position. Most of our studies uncovered a few big problems (higher severity) and a lot of smaller ones (lower severity). In these cases there were recurring pain points for teams around prioritisation:
- Some teams didn’t recognise the need to address lower severity problems. They would dismiss them, or defer even thinking about them until waaay in the future.
- Those who recognised the need found it hard to prioritise among their less severe problems.
One solution to these pain points lies in word choice.
Existing severity scales use unhelpful words
After a round of usability testing, we normally rate the problems we find on some combination of Frequency of occurrence, Percentage of users impacted and Severity. These ratings then feed into how problems get prioritised.
Frequencies and percentages are fine: we can measure or estimate those and the numbers are straightforward to interpret. But severity is subjective, and the word-based scales many of us use are hiding the human impact.
The most commonly used words across 10 popular severity scales are Minor and Major. These are comparative terms, but that doesn’t help when you’re considering individual problems. A problem rated Minor is easy to dismiss — ‘it’s only minor’, ‘just a minor problem’, ‘insignificant’—but for a user facing dozens of them it can be like death-by-a-thousand-paper-cuts.
Most of the words in these scales are abstract, distancing us from the human impact of the problems they describe. And some words are org-centric, focussing us on business needs over user needs. Only two words — Irritant and Unusable — remind us that the problem affects an actual person.
Another issue is lack of differentiation at the lower end of the scales. In one study last year we identified 90 usability problems, 85 of which would have been rated Minor on existing scales: how is a team relying on one big study supposed to prioritise based on that?!
Overall, the words used for existing scales make it easier for teams to ignore their less severe usability problems, and harder for them to prioritise.
We can do better…
…by choosing user-centred words to rate usability problems.
Over the last year I’ve been testing iterations of a user-centred severity scale in various different teams and contexts, and I’ve found that it helps to:
- communicate that every problem is worth addressing
- differentiate between lower severity problems that users experience consciously vs. subconsciously
- expand the empathy beyond the playback, helping the wider team internalise research findings, not just those members involved in design.
All of which increases the number of problems getting fixed!
My user-centred severity scale
Here are the words and descriptors I’ve settled on for now.
Friction
- User is slowed down or distracted, though possibly not aware of it.
- They can still complete their task, but [customise to team priorities e.g. ‘they may lose trust in our information’ or ‘they are less likely to enter the sales funnel’].
Frustration
- User is confused, annoyed or frustrated.
- They can still complete their task, but [customise to team priorities e.g. ‘they are more likely to abandon their cart’ or ‘less likely to recommended us’].
Blocker
- User is unable to complete their task.
Disaster
- User thinks they have completed their task but they haven’t.
AND/OR
There’s the possibility of harm to the user and/or organisation.
Bonus ratings
Like some of the existing scales I also include Positives and Observations when reporting on usability, because highlighting the best bits reinforces good practice.
Positive
- User appreciates this, may even be delighted.
Observation
- Not necessarily good or bad, just worth noting.
Why these words
Friction, Frustration, Blocker, Disaster
These words are user centred
At every level, these words bring to mind what it’s like for a person to experience the problem. Encouraging teams to use these words when talking about problems is like sneaking veggies into pasta sauce: everyone gets a healthy dose of empathy! Everyone and is reminded that they’re building things for people to use, not just some abstract business requirement.
These words are clear
No-one has asked me (yet) what any of them mean, unlike other words I tried and rejected including Impediment and Irritant.
These words stand alone
I think most of us know the ‘Ugh’ of frustration, the ‘Grrr’ of being blocked and the ‘Aaargh’ of disaster, and can recognise that unintended friction is bad, even if it’s not experienced consciously. Each rating makes sense without needing to be compared to the others around it, unlike Minor < Moderate < Major.
These words explain without blame
I rejected a whole group of words including Speedbump, Roadblock and Bomb because these things are made deliberately to slow people down or hurt them. I found that using these words implied intent and would put teams on the defensive: they’d spend more time laying blame or making excuses than trying to understand and fix things. Friction, Frustration, Blocker and Disaster describe the user’s experience in a blameless way.
These words help teams prioritise lower severity problems
For the team with 85 Minor usability problems, my scale rated them more helpfully as 55 unconscious points of Friction and 30 conscious Frustrations, making it easier to decide where to focus next after fixing their 5 Blockers. (There were no Disasters. There rarely are, but I keep it in the scale so I can start playbacks by pointing out how much worse it could’ve been!)
Visual styling
In contexts where colour is a helpful addition to the words, like cards and tags on a board or cell backgrounds in a spreadsheet, I use the following:
- Friction = yellow
- Frustration = orange
- Blocker = red
- Disaster = black
- Positive = green
- Observation = grey
- Recommendation = blue
Templates
You’re welcome to use my templates if you’d like to give this user-centred severity scale a try:
Conclusion
As any content designer will tell you, word choice matters. How we label things affects whether and how we think about them, which then affects our behaviour.
Choosing user-centred words to rate usability problems focuses teams on the people they’re building for, and helps them understand the impact of their decisions and actions.
If you think of anywhere else we could use more carefully chosen words to boost the user experience of user research and improve its impact, I’d love to hear about it!
~~~~~~~~~~~
Big thanks to Tove Markkula and Stephanie Krus for their help polishing up this blog post, and to Andrew Millar for the timely reminder that ‘perfect is the enemy of good’!
