The System Usability Scale (SUS) is a tool used to quantify the satisfaction of a digital user experience. Often used alongside results for effectiveness and efficiency, the scoring provides your project with a valuable quantitative measure of experience.
Often described as ‘quick and dirty’ or ‘at a glance’, it delivers live and insightful data to complement observational findings. In a previous article, we discussed why we had begun to incorporate SUS into our research more frequently. In this article, we will unpack how best to utilise the SUS.
There are many advantages to using the SUS in research. The reliability and validity of the scale has been proven over time through large-scale use and documentation, indeed, it has a 35-year history. This means we can confidently and accurately report on the findings that have either come from a large cohort of users or a smaller sample size.
We must, however state one important caveat: At Experience UX we believe that SUS is best applied when it is used alongside observational research. A stand-alone SUS score will not identify the all-important ‘why’ of user experience challenges.
When should you use SUS?
The key to utilising SUS effectively is to establish which projects would benefit from including this measure. It is easy enough to include SUS in every research and test project you do, but there is no point in collating data for data’s sake.
SUS is a good way to benchmark and improve, so iterative projects benefit greatly by assessing how well a digital product is evolving through an Agile process, for example.
As well as benchmarking progress, comparison testing also benefits from scoring methods such as SUS. For example, we are often asked to test a new prototype against a currently live version, and SUS is proven to work at a qualitative level whilst providing a reliable usability score for all versions.
For many of our clients, reporting to stakeholders is vastly improved with data. We have found that senior stakeholders sit up and take notice when the numbers are presented. Equally, by providing a snapshot view of how a digital experience is perceived to be performed can encourage fast decision making.
Why should you not use the SUS?
It’s important to note that SUS is not appropriate for every research project. The SUS measures users’ perceptions of usability which is subjective. The results must be correlated with objective findings from usability testing or another form of observational research. As we often find in research, what users say and what they do are two different things. The SUS will not diagnose issues and flaws within your product, only provide metrics around how users feel their experience was. SUS is a small yet reliable and useful component in a much larger process and should be considered as such when weighing up its value in assessing digital products and services.
How to introduce SUS during research?
We have always been careful to ensure quantitative scales do not influence or bias a user’s natural reaction or behaviour in a research session. We therefore administer this measure towards the end of the session, prior to our wrap-up questions. By introducing the SUS questionnaire soon after the participant has explored the digital experience, we produce an accurate set of usability data, which can we can explore further in our post-interview questions.
In our experience, it is helpful to record responses via an electronic SUS questionnaire form. We created our own that you can try. This is easier for your research participants to complete in a usability test environment, and for you to capture responses efficiently.
Be aware that some words used in the SUS can cause uncertainty, ‘cumbersome‘ for example. It can be helpful to introduce the questionnaire with the guidance of, ‘pause and take a moment to consider each question, and select an answer based on your gut feel’.
How to analyse SUS?
Whilst the ten-question SUS is straightforward for participants to complete, scoring the data requires some patience. Off-the-shelf Excel Spreadsheets are available to provide the scores, depending on how detailed you need to get (and how many iterations you are comparing).
Fundamentally, SUS will produce a score out of 100, which is simple enough. However, it is not a percentage. Instead, it provides an indication of usability against the SUS average score of 68. A score less than 50 is not acceptable. A score between 50 and 70 is marginal. Scores above 70 are acceptable.
Can I compare SUS results?
Yes (see above). SUS assesses an experience at a given point in time. It can be used to compare variants of a product or service at the same point in time. This could involve a 50:50 comparison, with half your participants using one of two prototypes.
Sit back and observe
Administering this usability assessment alone would not shed much light as to why people perceive the offering the way they do which is why it is important to combine it with observational or interview style methodology. This will continue to be the cornerstone and essential element of our user experience research. Combining quantitative and qualitative insight enables our clients to be that bit more confident in their user-centred way forwards.
