My main goal of user testing is to guide the design process with a focus on the end user. User-centered design is focused on designing for real users, and user testing tells us who that person is, in what context they’ll use a product, and what goal they are looking to achieve.
UX researchers have developed many techniques over the years for testing and validating their ideas, ranging from well-known lab-based usability studies to those that have been more recently developed, such as unmoderated online UX assessments and guerilla testing.
Some of the most popular forms of testing are usability testing, focus groups, beta testing, A/B testing, and surveys:
Usability testing is the best way to understand how real users experience your website or app. It’s also flexible for collecting a range of information about users, and it’s easy to combine with other techniques. Usability testing is a cornerstone of UX practice.
When it comes to usability testing, one of the most important decisions you’ll make is whether someone should moderate the session.
Moderated Usability Testing
Moderated usability testing is practiced by professionals looking to obtain feedback from live users. During a moderated test, moderators are ‘live’ with test participants (either in person or remotely), facilitating them through tasks, answering their questions, and replying to their feedback in real time. Live communication with test participants is a strength of this type of testing, because nothing beats watching participants in real time, and being able to ask probing questions about what they are doing.
When To Use
I recommended moderate testing during the design phase – when a team has a design that hasn’t yet been fully developed. I can run a moderated test to find the potential issues of a working prototype. By watching participants reactions on my prototype, I can gather baseline data that can reduce a lot of design and development time on a product that is difficult to use. As a moderator I can help probe the participant to delve deeper, keep the participant on track, and help clarify any confusion. However, a very common mistake moderators make is to tell a participant what to do. There is a fine line between guiding and helping a user. I find a balance to keep the participant on task, while not messing with their natural experience.
Unmoderated Remote Usability Testing (URUT)
Unmoderated remote usability testing occurs remotely. It offers inexpensive user testing results. This method is usually based on the use of usability testing tools that automatically gather participant feedback and record their behavior.
Here some usability testing tools
Crazyegg is a click-based user experience tool with four main features:
- ‘Heatmap’ logs where each visitor clicks on your webpage.
- ‘Scrollmap’ shows how far down the page each visitor typically scrolls.
- ‘Overlay’ breaks down the number of clicks on each page element.
- ‘Confetti’ provides detailed insights about visitor sources, search terms, and other components.
Optimizely is an extremely user-friendly A/B testing platform, which allows users to track visits and conversions. The tool offers an impressive range of features, including:
- Mobile website testing
- Cross-browser testing
- Visitor segmentation
- Multivariate testing
URUT has following benefits:
- Participants complete tasks in their own environment without a moderator present, which leads to the product being used naturally.
- URUT is conducted online much like a survey with pre-determined tasks, so it can be completed in the participant’s own time without requiring the hassle of coordinating schedules.
- Unmoderated tests can also be run concurrently, allowing for a much greater volume.
- The turn-around time for unmoderated tests is often significantly faster than that of moderated tests. Data can be collected in as a little as a few hours depending on sample size and testing criteria.
- The cost is usually quite low since you don’t need to pay for moderators or an equipment setup. You can get maximum value for minimum cost when tasks are written as clearly as possible.
I use URUT When I need a large sample to prove key findings from your initial moderated research. And, when you have very specific questions about how people use a user interface for relatively simple and straightforward tasks.
URUT should not be used as a replacement for moderated usability testing. Instead, it’s best when I use it in conjunction with moderated testing. The lack of a moderator means less control, less personal observation, and a higher risk of confusion. Thus, to run a test successfully I need to set clear expectations for participants—it’s crucial to ensure that tasks are clear and user-friendly.
Kyle Soucy suggests an unmoderated test should be 15–30 minutes in duration, and comprised of approximately 3–5 tasks, because the dropout rate tends to increase if a test takes longer.
Founder, UX Research Consultant at Usable Interface, LLC
Greater Boston Area
Focus groups are a tried and true method of communication between a researcher and users. In a focus group, I bring together 6 to 12 users to discuss issues and concerns about the features of a UI. The group typically lasts about 2 hours. I maintain the group’s focus.
This technique helps me assess user needs and feelings both before a product’s design. The proper role of focus groups isn’t to assess design usability, but to discover what users are seeking and expect from a product. Focus groups are not used as my only source of user testing data. They’re a poor method for evaluating interface usability as individuals rarely get the chance to explore the product on their own. I recommend running more than one focus group, because the outcome of any single session may not be representative.
Beta testing allows me to roll out a near-complete product to individuals who are happy to try it and provide critical feedback. This testing method allows me to ask users questions, track their usage and have them file bug reports. I use this testing when the product is near completion and I want to put it in the hands of the end users to gather feedback. Sufficient testing should be carried out before releasing a product to the customers. Naturally, I don’t want my users to find and report bugs, I simply want their feedback on product features and usability.
An A/B test is ideal as the appropriate testing method when designers are struggling to choose between two competing elements. A/B testing is good when trying to detect small differences in designs. This testing is particularly valuable when comparing a revised screen to an older version. With A/B testing I only find the best option from among the available variations. These variations should be selected very carefully. If the variations are only based on internal experience and opinion, the testing won’t find the optimal design.
Questionnaires and surveys are an easy way to gather a large amount of information about users, with minimal time invested. The right questions can uncover customer needs, desires, and pains. Surveys can help me accumulate quantitative data about overall user satisfaction or collect quantitative data to support qualitative research findings. Surveys are also good when I need to gather feedback about a new feature.
Important Limitations of Surveys
I can’t study user behavior with surveys. If I want to study how visitors behave or what usability problems they face during an interaction with a product, I prefer other research methods. Creating a survey looks like a quick and easy task, but in reality it is the opposite. A significant amount of time should be dedicated to preparing surveys. It’s important to get the questions right and direct them at the right audience.