Uncover the “why” with usability testing

A 7 minute read written by Scott June 16, 2015

An illustration of a clipboard and pencil.

Usability testing is an important method to use when building a successful website, application, or mobile app. As Avinash Kaushik, the Digital Marketing Evangelist at Google, likes to say about usability testing: “You watch the video and read the comments, you cry, you fix things, you become rich.”

We’ve been doing a lot more of these tests with our clients and I love watching their surprise when they see something not working as expected. While usability testing is an incredibly powerful method, many still fail to make it part of a project’s scope.

Hopefully I can help you better understand the value of usability testing, give some insight into what a usability test is, why you should test, how to get started, and some of the considerations involved in planning your testing.

Say yes to the test

The great thing about usability testing is that you can do it anytime, quickly, on any budget – and get real-world feedback and data on how you can improve your product or site. What’s not to love?

Here are some reasons you to consider usability testing:

  • To learn how actual users are interacting with your website. What buttons or links are they clicking on? How are they getting through a process or workflow?
  • To uncover problems and issues (sometimes even ones you didn’t know about or that were being tested).
  • To validate an approach for solving a problem. Maybe you’re not sure whether an idea you have will work. It could be a new kind of interaction you’re exploring, a unique way of navigating, a problem with an existing feature, or even the effectiveness of certain content.
  • To settle debates. Everyone has an opinion. Like metrics and analytics, usability testing can help you avoid using personal opinion or instinct to make decisions. It can also be an effective tool in getting team alignment on how to move forward when there is disagreement.

No matter what you are testing, consider the following:

  • Test early and often. Testing early means you can make changes – often at a lower cost – and adapt. Start off by testing with paper prototypes or sketches and early-stage deliverables such as wireframes or HTML. Later, move to higher-fidelity artifacts like your near-to-final product.
  • Collaborate externally and internally. When we do tests, we like to collaborate with our clients on the planning, get them involved in running and viewing tests, and even in reviewing and discussing recommendations and findings. Look for opportunities to also collaborate within your larger project team – maybe your front-end developer can be involved in creating a prototype, or your QA lead in identifying places to test.

Start with a research plan

There are a few things to consider when running a usability test. I like to start by putting together a research plan – a one- or two-pager that summarizes what will happen, why, when, and with whom. The research plan is easily shared and a great tool for summarizing the scope of the testing work. Our research plans contain the following information:

  • Title
  • List of the plan authors and the project stakeholders
  • Date and version (which gets updated as we update the plan)
  • Brief background on the project
  • Summary of the goals – why are we conducting the test?
  • List of questions we plan to answer
  • Overview of the methodology (a summary of our approach) we plan to use
  • The participants – who, how many, and their characteristics
  • The schedule
  • The script (which we will fill as we go)

11 things to consider when putting together your research plan

1. What kind of test do you want to run?

There are two primary kinds of usability tests: moderated and unmoderated.

In a moderated test a facilitator supports the test participants, guiding them through the tasks, answering their questions, and replying to their feedback in real time. What I like about moderated testing is that it is incredibly adaptable. As the facilitator, you can adapt or change the test plan as needed and you can dig into the “how” and the “why” that’s sometimes buried in participants’ answers.

In an unmoderated test, there is no facilitator. Testers walk through a set of pre-determined activities and complete them, recording their thoughts and input as they go. What’s great about an unmoderated test is that you can collect a lot of data from a lot of people in a short time.

2. Who will you test?

Figure out who you will be testing. A great place to start is with users who match your personas (you have personas right?). If you’re running a moderated test, you’ll need to find people who match your testing profile and get them booked for their testing session. This can be one of the most time-consuming tasks, so make sure you leave enough time for it. Most unmoderated tests don’t require recruitment of test participants.

3. How many people?

For a moderated test you can get decent test results with somewhere between five to seven users. For unmoderated tests you often purchase a batch of sessions at a fixed cost, but these can be limitless. There’s been lots written recently on this topic recently about the validity and statistical significance you can get with a small group size. But either way, some testing is better than no testing at all.

4. When do you want to test and how long will the tests take?

Assemble a schedule. Consider who you have participating in the tests and whether you need to do sessions during business hours or off-hours. When you’re running moderated tests you’ll only be able to fit in so many in a day before the testing team gets worn out. And you also want to be cognizant of your participants – it can be mentally exhausting doing tests. I usually find tests of 45 minutes to an hour, and with a 15-minute break between tests, to be a good duration. We can usually squeeze in seven 45-minute tests in a day, with short breaks between tests to get set up again and debrief quickly.

5. What will you test?

Is this a website? An application? A mobile app? Or do you have to test a combination (or all) of these? Make sure you’re able to support the test, have the right tools in place, and that you are testing the right format for the problems you want to solve. Create a script with all the questions and do a dry run to make sure the questions work and you have enough time.

6. Where will you test?

Will the test be in person or remote? In-person testing can be easier technologically since you control the tools being used for the testing but might mean more logistical considerations like booking a room. Remote testing might pose challenges for capturing tests in video – especially when testing mobile – so make sure you consider that when planning.

7. How will you facilitate the test?

We typically use a pretty simple setup when running a moderated test. All we need is some software to capture the screen of the user and, ideally, her voice or face. When doing in-person testing our favourite software is Silverback, but even basic web conferencing tools like GoToMeeting or Webex can be great substitutes for in-person or remote testing sessions.

For unmoderated tests, there are lots of different providers such as UserTesting, Loop11, and others. Do some research based on your needs and find the right fit.

8. Do you need to capture video?

Sometimes people want to watch a test later or review something. In moderated sessions you can capture video using basic tools like GoToMeeting, QuickTime (Mac), or tools dedicated to usability testing such as Silverback or Morae. Video capture can be very complicated when doing remote testing on mobile (most mobile devices aren’t set up to broadcast their screens) so take some time to explore options beforehand. Unmoderated sessions vary on video capture support. With either method, you’ll want to get the consent of participants before the test begins.

An image showing a sample consent form.

9. Will others be watching?

How will you support them during viewing sessions? They could join a GoToMeeting remotely or view from another room. I’d highly suggest having others attend and watch – it’s very powerful.

10. What are the reporting needs?

Some people are good with a final report, others a presentation. Consider how you’ll need to assemble your recommendations for the client or your team.

11. How much will you pay?

Participants in moderated tests are typically paid for their time (about $100/hour, but it can be less). Usually we give gift cards and maybe some company-branded stuff – T-shirts, pens, coffee mugs – as a thank-you. Unmoderated test participants are covered in the fee for the service.

What are you waiting for? Get testing!

Jump in and start planning a test. Either approach will give you good results. Just choose one and go for it. Here’s a table summarizing some of the differences of moderated and unmoderated usability tests:

Moderated Unmoderated
Facilitator needed Yes No
Note taking Yes No
Adaptable during test Yes No
Run testing sessions concurrently No Yes
Scheduled Yes No

If you need help, feel free to contact me. Our team would be happy to help you shape a plan, run a test, or teach your team how to do this well. You can also take a look at our approach to user research and strategic planning.