Mistakes to Avoid When Conducting Usability Testing

If the project is in its early stages or involves strategic planning:

Example: “We want to increase conversions by 20% through the new checkout flow.”

Reason: Business goals set the direction and priorities for what should be tested. It ensures usability testing focuses on features or areas that impact the business bottom line.

Practical Example:

Business Goal: Increase mobile app sign-ups.

Usability Testing Focus: Test whether the onboarding process is simple and intuitive for first-time users.

If the project is in the design, or iteration phase, or you are addressing a specific decline in acquisition, retention or known/unknown pain point:

Example: “Users are finding it difficult to find and apply discount codes at checkout.”

Practical Example:

User Goal: Simplify the search function.

Usability Testing Focus: Ensure users can quickly find what they need with minimal effort.

Example Goal:

“Understand how first-time users navigate the sign-up process and identify barriers to account creation.”

Explanation:

By focusing on a specific user scenario, like onboarding, stakeholders can see how the testing will deliver actionable insights.

KPIs for Measuring Success

Once testing has been completed and potential issues identified, you have pinpointed the area for improvement and made the necessary changes post-testing. Below are metrics you can now re-evaluate to determine if your changes have made improvements.

Engagement Metrics

• KPI: Percentage of users who access the budgeting tool weekly.
  • Why It Matters: Tracks how frequently users engage, indicating value and usability.
• KPI: Average time spent using the budgeting tool per session.
  • Why It Matters: Shows whether users find the tool useful and are exploring its features.

Adoption Metrics

• KPI: Number of new users activating the budgeting tool within the first month of its launch.
  • Why It Matters: Measures initial uptake and effectiveness of onboarding.

Retention Metrics

• KPI: Percentage of users who return to the budgeting tool at least three times within the first 90 days.
  • Why It Matters: Indicates whether the tool provides lasting value.

Task Success Metrics

• KPI: Percentage of users who successfully set up a budget within their first session.
  • Why It Matters: Measures how intuitive and user-friendly the setup process is.
• KPI: Error rate during key interactions (e.g., miscategorized transactions or failed budget setups).
  • Why It Matters: Identifies usability issues impacting task completion.

Satisfaction Metrics

• KPI: Net Promoter Score (NPS) for the budgeting tool.
  • Why It Matters: Gauges user satisfaction and likelihood to recommend the tool.
• KPI: Post-session survey rating (e.g., “How easy was it to track your expenses today?”).
  • Why It Matters: Captures immediate feedback on the user experience.

Conversion Metrics

• KPI: Percentage of users who sign up for premium features (if applicable) after using the budgeting tool.
  • Why It Matters: Tracks how effectively the tool drives monetization goals.

Example in Action

After usability testing:

80% of users could set up a budget within their first session (success threshold: 70%).

Average session time increased by 25%, indicating users are exploring more features.

User satisfaction score improved from 7.5 to 8.9 on a 10-point scale.

Use demographic, psychographic, and behavioural data to create user personas.

For example, if designing a budgeting app for millennials, your participants should reflect this audience.

Develop a screener survey to filter participants. Example questions include:

• “Have you used a budgeting app in the past six months?”
• “How frequently do you use mobile apps for financial tasks?”

Ensure representation across different user segments. For example:

Accessibility: Include participants with disabilities to test inclusivity.

Geography: Include users from different regions if your product has a global audience.

The Mistake:

Facilitators often feel compelled to intervene when participants hesitate or struggle. However, jumping in prematurely can disrupt the flow and prevent you from observing natural user behavior.

Solution:

• Practice Patience: Wait a few seconds before intervening. Often, participants are processing information or formulating their thoughts.
• Prompt Thoughtfully: If participants seem stuck, use open-ended prompts like:
  • “What are you thinking?”
  • “What would you do next?”
• Stay Observational: Let participants attempt tasks independently before offering guidance.

The Mistake:

On the flip side, some facilitators avoid stepping in even when participants are visibly distressed or stuck, leading to frustration and disengagement.

Solution:

• Recognize Distress Signals: If participants appear increasingly confused or frustrated, it’s okay to step in.
• Wrap Up Positively: If the task becomes too challenging, conclude it with encouragement:
  • “Thanks for trying that. Let’s move on to the next activity.”
• Ask Reflective Questions: If needed, inquire:
  • “What would you do at this point if you were on your own?”

The Mistake:

Leading participants with suggestive follow-up questions (e.g., “Were you unsure about the register button because it wasn’t clear?”) can bias their responses and skew results.

Solution:

• Stick to Open-Ended Questions: Instead of assuming, ask:
  • “Can you tell me what you were thinking when you hesitated?”
• Avoid Assumptions: Let participants explain their reasoning without introducing your hypotheses into the conversation.

The Mistake:

Facilitators may inadvertently influence participants by reacting verbally or non-verbally (e.g., nodding, smiling, or displaying surprise).

Solution:

• Maintain Neutrality: Stay composed and minimize non-verbal cues. Respond with neutral prompts like:
  • “What’s your thought process here?”
• Practice Consistency: Record yourself during sessions to identify and reduce unintentional reactions.

The Mistake:

While it’s important to make participants feel comfortable, excessive friendliness or joking can distract from the session and affect the quality of data.

Solution:

• Find the Right Balance: Be warm and professional, but avoid excessive humor or casual conversations.
• Embrace Silence: Accept moments of silence during tasks—it’s part of the process and allows participants to focus.

The Mistake:

Facilitators may answer participants’ questions during the session (e.g., “Is this the right link?”), which provides help users wouldn’t have in real-world scenarios.

Solution:

• Redirect Questions: Instead of answering, guide participants back to the interface:
  • “What do you think would happen if you clicked it?”
• Encourage Exploration: Allow participants to make decisions based on their interpretation of the design.
• Avoid Setting Expectations: Once you start answering questions, participants may rely on you for help throughout the session.

The Mistake:

Facilitators may unintentionally over-guide participants, preventing them from exploring the product as they would in real life.

Solution:

• Fade into the Background: Observe rather than guide, intervening only when necessary.
• Encourage Independence: Use prompts like:
  • “What would you do if I weren’t here to help?”
• Review Session Recordings: Identify moments where you might have influenced participants and adjust your approach in future sessions.

The Mistake:

Facilitators may become so focused on completing tasks that they overlook participant feedback or body language.

Solution:

• Prioritize User Experience: Pay attention to verbal and non-verbal cues from participants.
• Ask Reflective Questions: At the end of each task, inquire:
  • “What did you think about this activity?”
  • “Was there anything that stood out to you?”

Identify Critical User Journeys:
Focus on the most critical tasks aligned with your objectives, such as onboarding, completing a purchase, or using a new feature.

Use Task Ranking:
Rank tasks by priority and test the top three to five tasks first, saving secondary tasks for additional sessions.

Limit Session Duration:
Keep each session under 45–60 minutes to maintain participants’ attention and energy levels.

Divide Tasks Across Sessions:
If your test involves multiple tasks, spread them across multiple sessions with different participant groups.

• Example: One session for navigation and another for content comprehension.

Focus on a Few Tasks Per Iteration:
Conduct initial tests on the most critical tasks, gather feedback, make changes, and retest.

Use Feedback Loops:
After refining the design, retest previously untested tasks in subsequent iterations.

Consider Skill Levels:
Screen participants based on their familiarity with your product. Less experienced users may need fewer, simpler tasks.

Tailor Task Length:
Adjust the complexity and number of tasks based on their required cognitive load. For instance, avoid combining tasks that demand heavy problem-solving in a single session.

Plan Scheduled Breaks:
If a session exceeds 60 minutes, include 5–10-minute breaks to let participants recharge.

Allow Flexible Timing:
Let participants pause between tasks if they feel overwhelmed, maintaining the flow of authentic feedback.

Estimate Task Durations:
Test your tasks in advance with a pilot group to gauge how long each takes and adjust accordingly.

Limit Total Time Per Session:
Aim for a total task duration of 10–15 minutes, leaving extra time for follow-up questions or clarifications.