A/B Testing in Marketing

4 out of 5 CRO agencies I've worked with usually relied on 'best practices' to increase conversion rate. These practices include: - Adding badges like 'few left', 'bestseller' - Making reviews more prominent - Creating urgency with timers - Adding key product USPs - Leveraging offers While these strategies do give results, many tend to overlook a critical aspect. Which is UX/UI design. That’s likely the least spoken topic at a CRO agency. Despite its significant potential to increase conversion rates. In this example, using Nourish You India's PDP, I've implemented UX/UI and other changes that can increase conversion rates. Below are the 8 changes I recommend a/b testing - 1. Move the product name above the product image along with reviews+price. That way, the space between the images and the add-to-cart CTA is reduced, increasing the chances of adding to cart. 2. The primary product image should highlight key USPs. This would help the user to quickly understand why to buy this product and why from you. 3. Consider adding product image thumbnails. If your product requires education then use the image slider to provide that. Most important in consumables, personal care industry, and tech. 4. Consider adding 3 quick bullet points or USPs about the product before the user goes to add to cart. This way, they are educated about the product before they consciously think about purchasing from you. 5. Motivate users to add more quantity, increasing the AOV. Do this by highlighting savings when they buy in bulk or highlighting the cost per item if they buy a bundle. 6. Optimize the area around the add-to-cart CTA. Highlight the estimated delivery time, free shipping threshold and return policy. 7. Highlight key USPs to differentiate your product and brand from the others. 8. Add accordions that the user can click on to read more. This way they can find the answers to their questions quickly. Other 2 CRO changes I did: 1. Added 'Few left' once the user selected the pack they want to buy. This creates urgency. 2. Re-iterated price near the pack selection so the user doesn't have to scroll back up to see the price. Success lies in attention to detail. Found this useful? Let me know in the comments! P.S. The learning curve for UX/UI design is quite different from that of CRO. Some great resources to explore are Baymard Institute and Nielsen Norman Group to get started. #conversionrateoptimization #uxdesign

Listen, if you say “fail fast” but spend forty minutes in a post-mortem grilling the team on why a test underperformed — you just taught everyone in that room that failure is dangerous. If you say “take risks” but give the bigger raise to the person who hit their number safely rather than the one who tried something ambitious and came up short — you’ve told your team exactly what you actually value. You can’t talk your way to an experimentation culture. Ninety-five percent of the signal comes not from what you say, but from those three decisions. Reward. Promote. Fire.

As more companies embrace A/B testing, the bottleneck is no longer running experiments—it’s ensuring those experiments lead to trustworthy decisions. In this tech blog, the data science team at Booking.com explains how they scaled experimentation quality across the organization. Rather than enforcing rigid rules, they chose to preserve team autonomy while building the supporting systems needed to encourage better experimentation practices. The team’s solution followed a simple but thoughtful progression: process, metric, then tool. They first invested in community initiatives like Experiment Ambassadors and peer experiment reviews to build a shared experimentation culture. They then introduced an Experimentation Quality framework that evaluated every experiment across three dimensions—Design, Execution, and Decision—making experimentation quality measurable and easier to improve. Finally, they embedded those standards directly into their internal experimentation platform through features such as quality checks, power-calculation guidance, and stronger defaults that naturally guided teams toward better decisions. The goal was to maintain flexibility while making good experimentation practices easier to follow. This work highlights an important lesson: improving experimentation at scale requires more than statistical knowledge or individual discipline. Sustainable improvement comes from combining strong organizational processes, meaningful quality metrics, and tooling that reinforces good practices into the everyday workflow. When these pieces work together, teams can make more reliable decisions. #DataScience #MachineLearning #Experimentation #ABTesting #Analytics #SystemDesign #SnacksWeeklyonDataScience – – –  Check out the "Snacks Weekly on Data Science" podcast and subscribe, where I explain in more detail the concepts discussed in this and future posts:    -- Spotify: https://lnkd.in/gKgaMvbh   -- Apple Podcast: https://lnkd.in/gFYvfB8V    -- Youtube: https://lnkd.in/gcwPeBmR https://lnkd.in/gg8eX3Yv 

I watched a client go from taking weeks to launch experiments... to literally a few hours. Here's how I helped them 👇 When I started working with this client, their experimentation program was slow. Every idea had to work its way through a maze of approvals, backlogs, and coordination across teams. By the time something launched, the window of opportunity had already closed. I wanted to help them move faster. MUCH faster. So I started introducing the things that set world-class experimentation cultures apart from everyone else. What, pray tell, might those things be? Well, I love explaining with a particular and very true story that goes around at a former employer: there was a copywriter who had an idea on her bike ride to work, came into the office (this is pre-pandemic, people), made the change in her CMS (which was connected to the experimentation platform), and launched it as a global experiment running before her first coffee. That’s the kind of speed I wanted my client to experience, and we made it happen. How? Sure, we tightened up their tech and data flows so experiments could run smoothly. But this was the minor point in all honesty. The real shifts came from bringing together a cross-functional team who had the skills to deliver autonomously, getting leadership backing for that team to take risks, and setting a clear and focussed goal for the team to rally behind. We removed unnecessary “approvals,” facilitated the essential conversations, created focus, and rewarded pace without compromising rigour (improving it, actually). The team became empowered to make their own decisions and built a culture that normalised risk-taking. The result was night and day. Just weeks earlier, ideas took months to get through approvals, builds, and launches. All before even monitoring, reporting, and decision making (if any) took place. Now? The team could go from an ideation session to launching quick wins in literally hours. And, I can tell you first hand, when a team experiences this shift from moving like a snail to sitting up front of a rocket ship, that acceleration brings creativity, confidence, and energy. The kind that spreads across teams and compounds. It’s infectious. Oh, and did I mention that they started getting more runs on the board too? 😉 So recap, what makes these cultures different? And what did we instil to help this team accelerate that fast? ✅ Leaders who empower and trust their teams. ✅ Teams with genuine ownership and motivation to create impact. ✅ A culture that celebrates learning, not just winning, and treats failure as fuel for improvement. That’s what lets someone move from an idea on a bike ride → to a global experiment before their first coffee. If you want to innovate and experiment at lightning speed, don’t just look at your tech. Look at your culture. Could your org handle that kind of speed? 👇

💡A/B Testing: 8 Essential Tips A/B testing is a powerful method for comparing two versions of a design against each other to determine which one performs better. Here are the top 8 tips for conducting effective A/B tests: 1️⃣ Define clear goals: Know what you want to achieve with your test. Whether it's increasing conversions, click-through rates, or user engagement, having clear goals is crucial. 2️⃣ Test one variable at a time: To understand the effect of a change, test only one variable at a time (i.e., color of a primary call to action button). Multiple changes can confound results. 3️⃣ Randomize your sample: Ensure your sample is randomly selected to avoid biases and ensure the test results are reliable. 4️⃣ Ensure sufficient sample size: Make sure your test runs long enough to gather a statistically significant sample size to make confident decisions. Use sample size calculator: https://lnkd.in/dCXpgv2Z 5️⃣ Segment your audience: Consider segmenting your audience to understand how different groups respond to the change. 6️⃣ Monitor metrics beyond primary goal: Track secondary metrics to ensure that the changes do not negatively impact other important aspects of user experience (i.e., you have a higher conversion rate but a lower user retention rate). 7️⃣ Check for statistical significance—you need to ensure that the data you collect cannot be attributed to pure chance. Use the calculator to check significance: https://lnkd.in/d5jcWa7N 8️⃣ Consider long-term effects: Assess whether the changes have a lasting positive impact or if they might lead to long-term negative consequences (this can happen if you use dark patterns: https://lnkd.in/dtztGgFW) 📕 Introduction to A/B testing for product designers (YouTube): https://lnkd.in/dxuW8-hq #testing #design #research #productdesign #design #abtesting

At Swiggy every product feature goes live via A/B testing. Experimentation is such a goldmine for decision-making, and as someone who didn't even know how to do it right a year back - below is my step-by-step approach to statistical analysis. The problem statement is - You’re working to improve the conversion rate for a product signup flow. You’ve implemented several changes—now, how do you figure out which one really makes a difference? 1️⃣ Define the Hypothesis Before diving into the test, clearly define what you're testing. Example: "Will the new signup flow increase the conversion rate (CVR)?" 2️⃣ Define Clear Metrics What does success look like? Are you aiming to increase the percentage of sign-ups, or reduce drop-offs at a specific funnel stage? Success Metric: Conversion rate or step completion rate. Check Metric: Have a secondary metric to ensure nothing else breaks (e.g., page load times or errors). 3️⃣ Test One Change at a Time Testing multiple changes (e.g., a new form layout and an incentive like a discount) at once won’t help you pinpoint which worked. 4️⃣ Split Your Traffic Determine the sample size you need and split users randomly into two groups: - Group A (Control): Users see the current version - Group B (Test): Users see the new version 5️⃣ Collect Data & Analyze Monitor key metrics like CTR, conversion rates, or churn - choose metrics tied to the business goal. Example: Track how the new signup form impacts the completion rate of the entire signup process. 6️⃣ Analyze Statistical Significance You see a 15% increase in conversions with the new form—great! But is it statistically significant, or could it be due to random variation? Use p-values and Z scores to validate if the changes are meaningful and not just due to chance. 7️⃣ Interpret Results & Take Action Once you’ve confirmed statistical significance, interpret the results in a business context. Example: If the new form significantly increases conversion but doesn’t impact overall user satisfaction, it’s time to implement it at scale. 💡 What’s are some of your go-to strategies for an effective A/B test? Share your insights in the comments! ___________ 🔔 Follow Sanya Swain ♻ Repost to help others find it 💾 Save this post for future reference #businessanalysis #dataanalytics #dataanalyst #analytics #businessinsights #womenintech #product #sql #datascience #abtesting

One of my core leadership tenets is creating a culture of psychological safety - especially when it comes to experimentation - because it fosters innovation, creativity and a growth mindset. At 1Password, we believe that normalizing failure is a crucial part of becoming a stronger person and leader. By doing so, we create an environment that encourages risk-taking without the fear of losing your job or impacting your career trajectory. When experiments do fail, we apply the same blameless methodologies we use for incidents. We learn from them together, and try again as a team.   Steve Won and I recently held a hackathon that brought together over 200 employees from our tech, product, and design teams. Our leadership teams created space for cross-functional collaboration and encouraged our employees to push the boundaries of what our company can be. When you put the skill sets of an engineer, a designer and a product manager together, they jolt each other to think differently about the problem at hand. The "winning" projects often turn into full-fledged features. For example, we recently released 'nearby items' in 1Password. This feature allows users to assign a location to any of their 1Password items. A new dedicated section in the home tab will now display those items that are physically close (e.g. office Wi-Fi passwords, gym locker PIN codes, debit card PIN codes near ATMs, benefits insurance when you’re at the doctor, and more). We're proud of the culture of experimentation we've built at 1Password. If you're interested in learning more about our hackathon, check out our blog post: https://lnkd.in/g_3saCb2

Using insights from tens of thousands of A/B tests, I break down Ramp’s homepage hero ➡️ highlighting both the smart bets and areas for optimization. 1) Social proof at the top of homepages is typically a poor bet. Ramp includes both G2 stars above the header and a logo bar beneath the hero. In our testing, when brands place third-party review stars above headers, these versions consistently lose (I cover this in detail in my LinkedIn article, The Problem with Social Proof). Why? 🔎 Distrust of third-party reviews, often perceived as pay-to-play. 🔎 Key content and CTAs get pushed down, especially on mobile. 🔎 Unintentional signaling, for example "4.8 stars from 200+ reviews" can actually make the brand seem small. 2) Embedded email capture + clear CTA text is a winning combo. About two years ago, Ramp A/B tested an embedded email capture form versus a standard button. The embedded form won. Since then, across dozens of site iterations, they’ve kept it. Brands like Buffer and Rippling have similarly tested into and retained embedded capture forms. Their CTA text, “Get Started for Free,” is also strong: it clearly communicates that it’s a free trial. The only improvement I’d suggest is adding reassurance text below the CTA, clarifying that no credit card is required. We’ve seen this small detail improve conversions in multiple A/B tests (see Twilio’s homepage for a good example). 3) Secondary CTAs are a good bet. Ramp’s secondary CTA, “Explore Product,” beneath the main CTA is smart. We’ve seen extensive testing on one vs. two CTAs in homepage heroes for B2B SaaS and fintech brands. Two CTAs typically win. Why? Most of these companies have both self-service and enterprise buyers, with varied traffic sources (and intent levels). Offering two clear paths lets each group choose their preferred next step. 4) Product imagery works. Across hundreds of tests, product imagery consistently outperforms stock photos, branded graphics, or stylized backgrounds. Prospects want a preview of the actual product. 5) Customer logo bars typically underperform. I’ve written extensively on this, but here’s a quick recap of why logos usually lose: 🚧 Logo blindness: If you’re an industry leader, customers assume you serve top brands, so listing them adds little credibility. 🚧 Logo fit: Irrelevant logos create disconnect. Prospects want proof that companies like theirs trust your product. 🚧 Logos mislead: Many sites display big-brand logos when just a small team or individual used the product, or worse, when that company has already churned. If you do use logos, make them interactive or segmented. Brands like Clay and Hex link logos to case studies, providing depth. Others, like 7shifts, segment logos by industry to improve relevance. Hope this is helpful. Any other brands you would love to see analyzed based on DoWhatWorks's database of tracked tests?

Experimentation at Yelp had huge potential. But teams were measuring things differently. Teams had different definitions of "active user" and different statistical methods. Two teams could test the same feature and reach opposite conclusions - and neither could prove they were right. The first step was to measure the chaos. We audited past experiments across four dimensions: correctness, volume, performance, and infrastructure health. The results confirmed the problem: inconsistent metrics and design practices that led to slow, unreliable decisions. We needed standardization - urgently. We built a unified experimentation platform named Bunsen that gave every team the same foundation: - Central metrics layer so “active user” meant the same everywhere. - Experiment creation tool that cut setup from days to hours and enforced reliable statistical defaults. - Scorecard UI that displayed results and guardrail metrics in a shared language anyone could read. We paired it with mandatory PM training, an experimentation handbook, and on-team “deputies”. The new standard was adopted by the whole organization within 3 months. Within two years, experiment volume scaled 10x, decision accuracy doubled, and teams went from debating metrics to making data-backed decisions. In a world where the cost of building has gone down and dev velocity has gone up, guardrails and standardization are the only way to capture the upside of speed without letting quality slip.

Recently, I heard a manager demand his team "run it again." Those three words destroyed their experimentation culture. Here's what happened: His hypothesis: Faster changeovers would increase throughput by 15%. They ran the experiment for three weeks. Measured everything. Results: 3% improvement. Within normal variation. His response? "Run it again." ❌ Not the values poster in the lobby about "innovation and learning." ❌ Not the strategic plan promising "data-driven decisions." That single sentence told his team everything they needed to know. Then I watched a different leader handle the same situation last month. Her team tested a material substitution she'd been championing. Data showed 4% increase in defect rates. She pulled everyone together: ❓ "What does this tell us about our process constraints?" ❓ "Where else might we be making similar assumptions?" Then she thanked them. Publicly. And shifted budget to test three alternative approaches. THE REAL DIFFERENCE When we treat contradictory data as gold rather than treat it as a threat, → WeThey run experiments again → We find flaws in methodology → We explain why "this time was different" THE MOMENT THAT DEFINES YOUR CULTURE We don't build experimentation when data confirms our genius. We build it when results prove us wrong: • Our pet theory fails • The new engineer's method beats ours • Three weeks of testing reveals we optimized the wrong variable That's when our team is watching. Do we ask "What did we learn?" Or demand "Run it again?" Every time we dismiss contradictory data, we teach our team: "Don't bring me learning. Bring me validation." Then we wonder why: → Experiments slow down → Innovation stalls → People stop challenging assumptions The culture we want lives in how we respond to being wrong. What contradictory data are you sitting on right now? How will you respond? 👇 I help engineering and manufacturing leaders build experimentation systems where contradictory data drives breakthrough insights, not defensive reactions. If your teams keep confirming what you already believe, let's connect.