What do you mean we only need to user test with five users? Are you mad? We’re used to doing user research with hundreds and thousands of users, how can you possibly suggest that five users are enough?
OK, so we may have dramatised this a little, but we’ve had many similar conversations over the years we’ve been doing usability. It crops up on a regular basis, so we thought it was about time we anticipated the question and dealt with it here on our blog.
Usability research is not like other types of market research
It’s easy to see how clients can feel that five users are only a drop in the ocean when they are used to dealing with higher numbers of respondents. But this is the point. Usability research is very different to other forms of market research. Most forms of market research deal with opinion gathering. What do customers think and feel about this brand? How do they feel about the service we are offering? And What do they perceive our brand to offer over competitors… When dealing with opinions, clearly five people isn’t going to cut it, you need hundreds to get a meaningful measure.
Most of the usability research we conduct is designed to identify problems with a website. We’re asked ‘Why is my service not converting as well as I was expecting?’, ‘Why do users drop out at this point?’, and ‘What can we do to encourage more users to register?’… We’re asked to find problems our clients know exist somewhere in the site, and to help them fix them. It’s not about what people think or feel, it’s about observing user behaviour and isolating problems with the service.
How many times do you need to see a problem occur?
The great thing about searching for problems is that when you spot them you usually know it. If the zip breaks on your trousers it’s obvious there’s a problem. If you spill your coffee, spot a dent in your car… The problems are obvious. Finding problems with a website or service isn’t so clear cut, but when you watch someone struggle to complete a task because of the way it is designed, irrespective of the user’s level of skill or experience, you usually know you’ve uncovered an issue.
Let’s say we’ve just observed a user get very confused with your online checkout process. You’re pretty sure you have an issue which may be preventing people from buying. To be sure you test another person and they struggle in the same place. Then another person hits the same barrier. How many do you need to see before you’re convinced? Another 5 people? 10 more? 100? Do you need it to be statistically significant before you’re convinced?
What about discovering problems elsewhere? Let’s say you’ve noticed a couple of people trip on a kink in the carpet outside your office, or you’ve spotted a couple of people slipping up on some wet marble flooring. How many more people do you need to see hurt themselves before you’re willing to fix it?
Testing with more than 5 users results in diminishing returns
Sure you can test with more people, and there is a chance that you won’t uncover any issues until your 6th or 7th user. But it is very unlikely if you’ve designed your tests to cover the core areas of the service. What is more likely is that by your 6th user, you’ll start to see the same patterns of behaviour repeated over and over, and your value from the tests start to diminish.
You may have seen the great diagram put forward by Jakob Nielsen to show the diminishing returns of testing with more than five users.
We agree with this entirely, and what many people forget about the accompanying article from NNG is that he advocates testing with five people, fixing the issues, and then testing again. This is very much the way we like to work. We encourage many clients, who come to us with plans to recruit 15/20 people, to reduce their scope and break the project into chunks that allow for design changes to be made before new users are brought in.
You don’t need high numbers of users to identify usability problems. Don’t confuse user research or usability testing with other types of market research. If you want to gauge opinions about competing brands, then five people isn’t anywhere near enough. But if you want to identify barriers to purchasing on your website five users is usually enough.
So, are you convinced? Are five users enough?
