Multivariate Testing In UX

⏱️ How To Measure UX (https://lnkd.in/e5ueDtZY), a practical guide on how to use UX benchmarking, SUS, SUPR-Q, UMUX-LITE, CES, UEQ to eliminate bias and gather statistically reliable results — with useful templates and resources. By Roman Videnov. Measuring UX is mostly about showing cause and effect. Of course, management wants to do more of what has already worked — and it typically wants to see ROI > 5%. But the return is more than just increased revenue. It’s also reduced costs, expenses and mitigated risk. And UX is an incredibly affordable yet impactful way to achieve it. Good design decisions are intentional. They aren’t guesses or personal preferences. They are deliberate and measurable. Over the last years, I’ve been setting ups design KPIs in teams to inform and guide design decisions. Here are some examples: 1. Top tasks success > 80% (for critical tasks) 2. Time to complete top tasks < 60s (for critical tasks) 3. Time to first success < 90s (for onboarding) 4. Time to candidates < 120s (nav + filtering in eCommerce) 5. Time to top candidate < 120s (for feature comparison) 6. Time to hit the limit of free tier < 7d (for upgrades) 7. Presets/templates usage > 80% per user (to boost efficiency) 8. Filters used per session > 5 per user (quality of filtering) 9. Feature adoption rate > 80% (usage of a new feature per user) 10. Time to pricing quote < 2 weeks (for B2B systems) 11. Application processing time < 2 weeks (online banking) 12. Default settings correction < 10% (quality of defaults) 13. Search results quality > 80% (for top 100 most popular queries) 14. Service desk inquiries < 35/week (poor design → more inquiries) 15. Form input accuracy ≈ 100% (user input in forms) 16. Time to final price < 45s (for eCommerce) 17. Password recovery frequency < 5% per user (for auth) 18. Fake email frequency < 2% (for email newsletters) 19. First contact resolution < 85% (quality of service desk replies) 20. “Turn-around” score < 1 week (frustrated users → happy users) 21. Environmental impact < 0.3g/page request (sustainability) 22. Frustration score < 5% (AUS + SUS/SUPR-Q + Lighthouse) 23. System Usability Scale > 75 (overall usability) 24. Accessible Usability Scale (AUS) > 75 (accessibility) 25. Core Web Vitals ≈ 100% (performance) Each team works with 3–4 local design KPIs that reflects the impact of their work, and 3–4 global design KPIs mapped against touchpoints in a customer journey. Search team works with search quality score, onboarding team works with time to success, authentication team works with password recovery rate. What gets measured, gets better. And it gives you the data you need to monitor and visualize the impact of your design work. Once it becomes a second nature of your process, not only will you have an easier time for getting buy-in, but also build enough trust to boost UX in a company with low UX maturity. [more in the comments ↓] #ux #metrics

Getting the right feedback will transform your job as a PM. More scalability, better user engagement, and growth. But most PMs don’t know how to do it right. Here’s the Feedback Engine I’ve used to ship highly engaging products at unicorns & large organizations: — Right feedback can literally transform your product and company. At Apollo, we launched a contact enrichment feature. Feedback showed users loved its accuracy, but... They needed bulk processing. We shipped it and had a 40% increase in user engagement. Here’s how to get it right: — 𝗦𝘁𝗮𝗴𝗲 𝟭: 𝗖𝗼𝗹𝗹𝗲𝗰𝘁 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸 Most PMs get this wrong. They collect feedback randomly with no system or strategy. But remember: your output is only as good as your input. And if your input is messy, it will only lead you astray. Here’s how to collect feedback strategically: → Diversify your sources: customer interviews, support tickets, sales calls, social media & community forums, etc. → Be systematic: track feedback across channels consistently. → Close the loop: confirm your understanding with users to avoid misinterpretation. — 𝗦𝘁𝗮𝗴𝗲 𝟮: 𝗔𝗻𝗮𝗹𝘆𝘇𝗲 𝗜𝗻𝘀𝗶𝗴𝗵𝘁𝘀 Analyzing feedback is like building the foundation of a skyscraper. If it’s shaky, your decisions will crumble. So don’t rush through it. Dive deep to identify patterns that will guide your actions in the right direction. Here’s how: Aggregate feedback → pull data from all sources into one place. Spot themes → look for recurring pain points, feature requests, or frustrations. Quantify impact → how often does an issue occur? Map risks → classify issues by severity and potential business impact. — 𝗦𝘁𝗮𝗴𝗲 𝟯: 𝗔𝗰𝘁 𝗼𝗻 𝗖𝗵𝗮𝗻𝗴𝗲𝘀 Now comes the exciting part: turning insights into action. Execution here can make or break everything. Do it right, and you’ll ship features users love. Mess it up, and you’ll waste time, effort, and resources. Here’s how to execute effectively: Prioritize ruthlessly → focus on high-impact, low-effort changes first. Assign ownership → make sure every action has a responsible owner. Set validation loops → build mechanisms to test and validate changes. Stay agile → be ready to pivot if feedback reveals new priorities. — 𝗦𝘁𝗮𝗴𝗲 𝟰: 𝗠𝗲𝗮𝘀𝘂𝗿𝗲 𝗜𝗺𝗽𝗮𝗰𝘁 What can’t be measured, can’t be improved. If your metrics don’t move, something went wrong. Either the feedback was flawed, or your solution didn’t land. Here’s how to measure: → Set KPIs for success, like user engagement, adoption rates, or risk reduction. → Track metrics post-launch to catch issues early. → Iterate quickly and keep on improving on feedback. — In a nutshell... It creates a cycle that drives growth and reduces risk: → Collect feedback strategically. → Analyze it deeply for actionable insights. → Act on it with precision. → Measure its impact and iterate. — P.S. How do you collect and implement feedback?

I opened Canva the other day and something caught my eye - 👀 A vibrant banner right on the home screen announcing "Droptober is coming." With a countdown, hyping up new features that are set to launch in a few days. It's a simple yet effective reminder for users that new and exciting tools are just around the corner that not only sparks curiosity but also creates anticipation. 👉🏻 𝗜𝘁'𝘀 𝗮 𝗯𝗿𝗶𝗹𝗹𝗶𝗮𝗻𝘁 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝘆 𝘁𝗼 𝗯𝗼𝗼𝘀𝘁 𝗻𝗲𝘄 𝗳𝗲𝗮𝘁𝘂𝗿𝗲 𝗮𝗱𝗼𝗽𝘁𝗶𝗼𝗻 𝗿𝗮𝘁𝗲𝘀 𝗯𝗲𝗰𝗮𝘂𝘀𝗲: ✅ Instead of relying on emails or external announcements that might get lost, a banner on the app's home screen ensures the message reaches active users. It’s an in-app reminder that stays top of mind. ✅ Adding a countdown creates a sense of urgency. It makes users feel like they’re part of something special, something they don’t want to miss out on. ✅ Visual elements like banners can capture attention faster than text-heavy announcements. 🔵 𝗪𝗵𝗮𝘁 𝗼𝘁𝗵𝗲𝗿 𝘀𝘁𝗿𝗮𝘁𝗲𝗴𝗶𝗲𝘀 𝗰𝗮𝗻 𝘄𝗲 𝘂𝘁𝗶𝗹𝗶𝘇𝗲 𝗳𝗼𝗿 𝗱𝗿𝗶𝘃𝗶𝗻𝗴 𝗮𝗱𝗼𝗽𝘁𝗶𝗼𝗻: 💡 𝗜𝗻-𝗮𝗽𝗽 𝗮𝗻𝗻𝗼𝘂𝗻𝗰𝗲𝗺𝗲𝗻𝘁𝘀: Like Canva, using banners or pop-ups within the product helps to keep users informed. It’s a great way to announce a new feature, offer tutorials, or even give a sneak peek. 💡 𝗚𝗮𝗺𝗶𝗳𝘆 𝘁𝗵𝗲 𝗹𝗮𝘂𝗻𝗰𝗵: Products like Duolingo have mastered gamification. What if you could create a mini-challenge for users to try out the new feature? Reward them with badges or exclusive access. 💡 𝗣𝗿𝗼𝗴𝗿𝗲𝘀𝘀𝗶𝘃𝗲 𝗿𝗼𝗹𝗹𝗼𝘂𝘁𝘀 𝘄𝗶𝘁𝗵 𝘂𝘀𝗲𝗿 𝘀𝗲𝗴𝗺𝗲𝗻𝘁𝘀: Netflix often tests new features with a small percentage of users before a full rollout. This helps gather feedback, refine the experience, and build buzz through word-of-mouth. 💡 𝗢𝗻𝗯𝗼𝗮𝗿𝗱𝗶𝗻𝗴 𝘄𝗮𝗹𝗸𝘁𝗵𝗿𝗼𝘂𝗴𝗵𝘀: When Slack releases a new feature, they often integrate it directly into the product’s onboarding flow, guiding users step-by-step. It’s not just about telling users what's new, but how to use it. 👉🏻 𝗧𝗮𝗸𝗲𝗮𝘄𝗮𝘆 𝗳𝗼𝗿 𝗣𝗠𝘀 - Invest in strategies that bring the message to where your users are most engaged i.e. within your product itself. Keep it simple, visually appealing, and engaging. And remember, the more excitement you build around a new feature, the higher the chances of driving adoption. So, next time you’re planning a launch, think about how you can create that “I can’t wait to try this!” moment. PS. What other strategies do you use as a PM for new feature launches? Do share in the comments!

𝑵𝒆𝒘 𝑫𝒂𝒕𝒂 𝑺𝒉𝒐𝒘𝒔 𝑳𝒐𝒔𝒊𝒏𝒈 80% 𝑶𝒇 𝑴𝒐𝒃𝒊𝒍𝒆 𝑼𝒔𝒆𝒓𝒔 𝑰𝒔 𝑵𝒐𝒓𝒎𝒂𝒍, 𝒂𝒏𝒅 𝑾𝒉𝒚 𝒕𝒉𝒆 𝑩𝒆𝒔𝒕 𝑨𝒑𝒑𝒔 𝑫𝒐 𝑩𝒆𝒕𝒕𝒆𝒓 How do you measure app success? Clicks? Downloads? Rating? How about retention? New data from Andrew Chen, a partner at Andreessen Horowitz, and mobile intelligence start-up, Quettra shows: → An average app loses 77% of its Daily Active Users (DAU) within 3 days of installation; AND → The decline slows considerably after 3 days. Interestingly Andrew’s analysis goes on to show that for the top-performing apps, the initial decline is less marked. For example, the top 10 apps only lose around 25% DAU in the first 3 days. However, after around 3 days, the decline rate bottoms out in a similar fashion to the overall average. In other words, the initial few days after installation are critical. Recently, I highlighted the two main ways to raise app awareness and acquire users: searching and paid ads. But what can you do to retain your user after they’ve installed your app? The key is user engagement. 🔹 Build trust with a compelling, but authentic story The first touchpoint is often the app store. App Store Optimisation (ASO) is critical to allow users to discover you in the first place. However, your description must be more than a few lines plastered with keywords. Long-term engagement is established through connection with your user. Convey your app’s value honestly and without exaggeration. Users are quickly turned off when developers over-promise and apps fall short. 🔹Design a frictionless onboarding process Your user’s first real experience of your app is at onboarding, so make it user-friendly with simple step-by-step instructions, and easy-to-understand graphics or videos. There’s nothing worse than a cumbersome onboarding process, and if the app is also difficult to navigate or counterintuitive to use, users will soon be reaching for the delete button. 🔹Make your app sticky Your ultimate goal is to make your app part of your user’s routine. Adding gamified elements that require users to complete daily challenges, tailoring the experience to the user, or incorporating social features are just some of the options available. Kurve’s client, BackThen, had a goal of increasing its user base. We analysed user behaviours to identify the triggers that led to subscription. Based on the outcomes, we crafted an engagement strategy and complementary marketing approach around 4:2:1 – 4 photos uploaded + 2 invites + 1 comment. The result? BackThen’s user base grew 5x in 12 months. With so many apps vying for users’ attention, stemming user decline is a conundrum. There’s no quick fix. It takes a combination of discoverability, seamless onboarding, and a first-class user experience to drive loyalty. Anything you’d add? 👇 #mobileapps #appmarketing #appstore Source: andrewchen.com

Stakeholders often focus on “how many” when presented qualitative research. Which is the wrong question to ask. Qualitative is about understanding the H (human) in HCI. The goal is to understand why they behave like that. When presenting research results: focus on showing clear patterns, supporting findings with evidence like quotes or observations, and connecting everything back to user behaviors and business goals, not sample sizes. Also, combine qualitative with quantitative to explain the what and the why. For example: - Quantitative shows what's happening: 72% abandon the goal-setting flow at account connection. - Qualitative reveals why: Users worry about security, are confused about account selection, and fear they can't reverse connections. - The powerful combination: "Our drop-off problem stems from specific trust concerns and mental model mismatches. By redesigning to address these specific issues, we can reduce the 72% abandonment rate." Beyond Numbers: How to Properly Evaluate Qualitative UX Research (9min) By Dr Maria Panagiotidi https://lnkd.in/gbqRneY4

Publisher experiments fail when they start with tactics, not hypotheses. A/B testing has become a staple in digital publishing, but for many publishers, it’s little more than tinkering with headlines, button colours, or send times. The problem is that these tests often start with what to change rather than why to change it. Without a clear, measurable hypothesis, most experiments end up producing inconclusive results or chasing vanity wins that don’t move the business forward. Top-performing publishers approach testing like scientists: They identify a friction point, build a hypothesis around audience behaviour, and run the experiment long enough to gather statistically valid results. They don’t test for the sake of testing; they test to solve specific problems that impact retention, conversions, or revenue. 3 experiments that worked, and why 1. Content depth vs. breadth: Instead of spreading efforts across many topics, one publisher focused on fewer topics in greater depth. This depth-driven strategy boosted engagement and conversions because it directly supported the business goal of increasing loyal readership, and the test ran long enough to remove seasonal or one-off anomalies. 2. Paywall trigger psychology: Rather than limiting readers to a fixed number of free articles, an engagement-triggered paywall is activated after 45 seconds of reading. This targeted high-intent users, converting 38% compared to just 8% for a monthly article meter, resulting in 3x subscription revenue. 3. Newsletter timing by content type: A straight “send time” test (9 AM vs. 5 PM) produced negligible differences. The breakthrough came from matching content type to reader routines: morning briefings for early risers, deep-dive reads for the afternoon. Open rates increased by 22%, resulting in downstream gains in on-site engagement. Why most tests fail • No behavioural hypothesis, e.g., “testing headlines” without asking why a reader would care • No segmentation - treating all users as if they behave the same • Vanity metrics over meaningful metrics - clicks instead of conversions or LTV • Short timelines - stopping before 95% statistical confidence or a full behaviour cycle What top performers do differently ✅ Start with a measurable hypothesis tied to business outcomes ✅ Isolate one behavioural variable at a time ✅ Segment audiences by actions (new vs. returning, skimmers vs. engaged) ✅ Measure real results - retention, conversions, revenue ✅ Run tests for at least 14 days or until reaching statistical significance ✅ Document learnings to inform the next test When experiments are designed with intention, they stop being random guesswork and start becoming a repeatable growth engine. What’s the most valuable experimental hypothesis you’re testing this quarter? Share with me in the comment section. #Digitalpublishing #Abtesting #Audienceengagement #Contentstrategy #Publishergrowth

A design can pass a usability test and still feel exhausting. That is why 𝗡𝗔𝗦𝗔-𝗧𝗟𝗫 is useful in UX research. NASA-TLX, the Task Load Index, is a post-task rating method for measuring perceived workload. Instead of asking only whether users succeeded, it asks what success cost them. The scale looks at six dimensions: 🧠 𝗠𝗲𝗻𝘁𝗮𝗹 demand: How much thinking, remembering, deciding, or searching was required? 💪 𝗣𝗵𝘆𝘀𝗶𝗰𝗮𝗹 demand: How much physical action was required? ⏱️ 𝗧𝗲𝗺𝗽𝗼𝗿𝗮𝗹 demand: How rushed or time-pressured did the task feel? 🎯 𝗣𝗲𝗿𝗳𝗼𝗿𝗺𝗮𝗻𝗰𝗲: How successful did the user feel? 🔥 𝗘𝗳𝗳𝗼𝗿𝘁: How hard did the user have to work to reach that result? 😤 𝗙𝗿𝘂𝘀𝘁𝗿𝗮𝘁𝗶𝗼𝗻: How insecure, annoyed, discouraged, or stressed did the user feel? In a UX study, the usual pattern is simple. Users complete a task, then rate these dimensions. Researchers compare ratings across tasks, prototypes, user groups, or design alternatives. The result is not just a score. The real value is often in the workload profile: where the burden appears and what kind of burden it is. This matters because ease of use is not the same as low workload. A product can be learnable but draining. A checkout can be fast but stressful. A dashboard can be powerful but mentally expensive. A workflow can produce few errors while forcing users to hold too much in memory. NASA-TLX helps teams see these hidden costs. It turns subjective strain into a structured design signal. For UX designers, that signal is practical: 🔍 Reveal cognitive friction that observation alone may miss. 📊 Compare competing designs beyond task time and completion rate. ⚠️ Explain why users make errors, hesitate, or abandon a flow. ✅ Encourage interfaces that are not only efficient, but sustainable to use. Good UX reduces the work the interface adds to the work users already came to do.

💡Qualitative and Quantitative UX Research Methods When choosing between qualitative and quantitative UX research methods, it’s important to understand what you want to learn and how the data you collect can inform your decisions. 🍎 Qualitative methods Help to understand why users behave the way they do, their motivations, and emotions. When to use ✔ Exploratory research: Research you conduct to gather insights about the target audience and problem space (uncover needs, behaviors, and pain points of potential users)  ✔ Early in design pipeline: When you are in the early stages of product design (i.e., during ideation) and need deep insights into a particular area to guide design. ✔ Small samples: When you don’t need statistical significance but rather deeper insights from a smaller group of users. For example, when you’re trying to understand nuances of experiences of a particular category of users. Popular qualitative methods ✔ User interviews: Gather deep insights into user thoughts and feelings. ✔ Usability testing: Observe how users interact with your product and identify usability issues. ✔ Field studies: Observe users in their natural environment to understand real-world use. Tips for qualitative research ✔ Use a small, diverse sample: Gather insights from a small but diverse group of users to understand different perspectives. ✔ Ask open-ended questions and avoid leading questions during interviews: Encourage users to speak freely to gain deeper insights but keep your questions neutral to avoid influencing responses. ✔ Observe non-verbal cues: Body language & facial expressions can reveal a lot about user frustration or satisfaction. 🍏 Quantitative methods Help to gather data on what users do. These methods typically focus on measurable behaviors and trends. When to use ✔ Validation/Benchmarking: When you need to validate design decisions or test a hypothesis. To do so, you collect the data that is aligned with your goal. For example, when you want to measure the impact of design changes on conversion flow, you will track conversion rate and user satisfaction. ✔ Large scale: You need to have a statistically significant sample to generalize results (i.e., 30% of 1000 users want feature A) Popular quantitative methods ✔ Surveys: Great for collecting large amounts of data quickly. ✔ A/B testing: Compare two versions of a design to find which one performs better.  ✔ Analytics: Track user behavior and engagement. Tips for quantitative research ✔ Ensure sufficient sample size: Aim for a sample size large enough to draw meaningful conclusions. Sample size for surveys https://lnkd.in/dwa-M-82  ✔ Write clear, unbiased questions in surveys: Ensure your survey questions are easy to understand. ✔ Combine with qualitative insights: Quantitative data tells you what’s happening and qualitative research will help explain why it’s happening. 🖼️ UX research methods by Maze #UX #research #uxresearch

Build it and they will come? 🤔 When product teams launch a highly-requested feature, they tend to expect users to engage with it. But things don’t always work out that way. 😬 This is what Lyft discovered when they launched Women+ Connect. Despite the clear benefits and the demand for the feature, not all drivers who were eligible were opting into it. 🔍 The challenge? Simply telling users about a feature isn’t always enough to drive action. When Irrational Labs partnered with Lyft, here’s what our brilliant behavioral scientist Isabel Macdonald, PhD and team learned, working closely with Robyn Bald and Kirsten M.: 🚀 A simple shift in messaging—based on behavioral science—can massively impact feature engagement. Irrational Labs tested several behaviorally-informed messages and all outperformed the control. The winning message? “Just checking. Looks like you are not opted into Women+ Connect. Is this correct? Tap to review.” What this does: The question creates a desire for resolution and nudges the driver to take action (versus do nothing). The result? Compared to the control group, this approach got 173% more opt-ins from new drivers. 📈 So, what’s the takeaway for product teams? 👉🏼 Product success doesn’t come just from building great features. You have to frame them in ways that resonate with your users, capture their attention, and motivate them to act. 💡 Curious to see how small changes can lead to massive impact? Check out the link in the comments to learn how we helped Lyft get great engagement with a great feature. 👇🏼 #BehavioralScience #ProductManagement #UserEngagement #WomenInTech #IrrationalLabs #Lyft

If you're a UX researcher working with open-ended surveys, interviews, or usability session notes, you probably know the challenge: qualitative data is rich - but messy. Traditional coding is time-consuming, sentiment tools feel shallow, and it's easy to miss the deeper patterns hiding in user feedback. These days, we're seeing new ways to scale thematic analysis without losing nuance. These aren’t just tweaks to old methods - they offer genuinely better ways to understand what users are saying and feeling. Emotion-based sentiment analysis moves past generic “positive” or “negative” tags. It surfaces real emotional signals (like frustration, confusion, delight, or relief) that help explain user behaviors such as feature abandonment or repeated errors. Theme co-occurrence heatmaps go beyond listing top issues and show how problems cluster together, helping you trace root causes and map out entire UX pain chains. Topic modeling, especially using LDA, automatically identifies recurring themes without needing predefined categories - perfect for processing hundreds of open-ended survey responses fast. And MDS (multidimensional scaling) lets you visualize how similar or different users are in how they think or speak, making it easy to spot shared mindsets, outliers, or cohort patterns. These methods are a game-changer. They don’t replace deep research, they make it faster, clearer, and more actionable. I’ve been building these into my own workflow using R, and they’ve made a big difference in how I approach qualitative data. If you're working in UX research or service design and want to level up your analysis, these are worth trying.