E-commerce businesses make hundreds of decisions that can affect how customers shop.
Should the Add to Cart button say “Add to Cart” or “Buy Now”?
Should shipping information appear near the price or farther down the product page?
Do customers respond better to product-only photography or lifestyle images?
Would a shorter checkout process reduce abandonment?
Should a free-shipping threshold appear in the announcement bar, on the product page, or in the cart?
You could make those decisions based on instinct.
Or you could test them.
A/B testing allows you to compare different versions of an experience and measure how customers actually respond.
Done well, it can help an e-commerce business improve product pages, checkout flows, landing pages, email campaigns, offers, navigation, and other parts of the customer journey.
But good experimentation is more sophisticated than changing a button from blue to green and waiting to see which version gets more clicks.
The real value of A/B testing is reducing uncertainty.
You identify a problem, develop a hypothesis, test a meaningful change, measure the outcome, and use what you learn to make a better decision.
For Success Brews, I like this framework:
Observe → Hypothesize → Prioritize → Experiment → Measure → Learn → Repeat
Let’s walk through how it works.
1. What Is A/B Testing?
A/B testing — also called split testing — is a controlled experiment in which different groups of users experience different versions of something.
Google Analytics defines an A/B test as a randomized experiment involving two or more variants. The original is generally considered Variant A, while Variant B — or additional variants in an A/B/n test — contains one or more changes. Users are randomly assigned to the variants during the same period.
For example, suppose your current product page has a button that says:
Add to Cart
That’s your control: Variant A.
You create another version:
Add to Bag
That’s Variant B.
Rather than changing the button for everyone and comparing this month’s sales with last month’s sales, an A/B experiment divides eligible users between the experiences.
You can then examine whether customer behavior differs between the groups.
But that simple example leaves out the most important question:
Why are you testing the button in the first place?
That’s where good experimentation begins.
2. What A/B Testing Can—and Can’t—Tell You
A/B testing is useful because it allows you to observe behavior rather than relying entirely on opinions.
Suppose your team is debating whether product reviews should appear higher on a product page.
One person thinks reviews increase trust.
Another thinks they distract shoppers.
A third thinks customers don’t care either way.
You can debate it for three meetings.
Or, if you have sufficient traffic and the question is important enough, you can design an experiment.
A/B testing can help provide evidence about questions such as:
- Did the variation change the primary metric?
- How large was the observed difference?
- Did other important metrics change?
- Is the evidence strong enough to support a decision?
- Is the effect large enough to matter to the business?
What it doesn’t automatically tell you is why something happened.
Suppose Version B increases purchases.
Was it because customers trusted the page more?
Because the offer became clearer?
Because the CTA became easier to find?
Because one element reduced friction?
Quantitative experimentation tells you what happened within the parameters of the experiment.
Customer interviews, usability testing, surveys, session analysis, support conversations, and other research can help you understand why.
SUCCESS BREWS TIP
A/B testing is most useful when it’s combined with customer research. Use research to discover problems and experiments to test potential solutions.
3. Start With a Hypothesis, Not a Button Color
One of the easiest ways to waste an experimentation program is to begin with:
“What should we test?”
That question often produces random ideas:
“Let’s test a red button.”
“Let’s make the headline bigger.”
“Let’s move the reviews.”
“Let’s change the hero image.”
Maybe one of those changes works.
But why are you making it?
A stronger experiment starts with a problem or opportunity.
Start With an Observation
Suppose your analytics show that plenty of customers view products, but relatively few add them to their carts.
Customer-service questions also reveal that shoppers frequently ask about shipping costs.
That’s useful evidence.
Observation
Customers may be hesitating because shipping information isn’t visible until later in the purchase process.
Hypothesis
If we display shipping information near the Add to Cart button, more customers may feel comfortable beginning the purchase process.
Experiment
Variant A: Current product page.
Variant B: Product page with shipping information displayed near the CTA.
Primary Metric
Add-to-cart rate.
Guardrail Metrics
Purchase conversion rate, revenue per visitor, cart abandonment, or other metrics important to the business.
Now you aren’t randomly moving text around a page.
You’re testing a theory about customer behavior.
A useful hypothesis can follow this structure:
Because we observed [problem], we believe [change] may result in [outcome] because [reason].
4. Choose One Primary Metric
Before starting the experiment, decide what success means.
Your primary metric should connect directly to your hypothesis.
For an e-commerce business, possible primary metrics include:
purchase conversion rate,
add-to-cart rate,
checkout completion rate,
revenue per visitor,
average order value,
email signup rate,
or another meaningful customer action.
Don’t simply collect every metric available and declare whichever one improved the “winner.”
Decide what matters before the test.
Add Guardrail Metrics
You should also identify metrics that help you detect unintended consequences.
Suppose you’re testing a more aggressive discount.
Your primary metric is purchase conversion.
Variant B produces more purchases.
Great.
Except revenue per visitor falls dramatically because customers are purchasing lower-value orders with a much larger discount.
If you only looked at conversion rate, you might conclude that the experiment was successful.
Guardrails help provide context.
Depending on the experiment, useful guardrail metrics might include:
revenue per visitor,
average order value,
refund rate,
cart abandonment,
page performance,
customer support contacts,
or another important business measure.
The metric should fit the experiment.
5. Calculate Your Sample Size Before Testing
The original version of this article suggested aiming for at least 1,000 visitors per variation.
That’s too simplistic.
There is no universal number of visitors that makes an A/B test reliable.
The sample required depends on factors such as:
your baseline performance,
the size of the change you want to detect,
the statistical method you’re using,
your chosen significance/confidence threshold,
metric variability,
number of variations,
and traffic allocation.
One particularly important concept is minimum detectable effect, or MDE.
Your MDE represents the smallest change you want your experiment to be capable of detecting.
Smaller changes generally require much larger samples.
For example, Optimizely’s current documentation illustrates how dramatically sample requirements can increase as the desired detectable effect becomes smaller.
This is why:
“Run the test until you have 1,000 visitors.”
isn’t a reliable experimentation strategy.
Use the planning tools and statistical methodology supported by your experimentation platform.
For traditional fixed-horizon experiments, determine your required sample before starting and establish your analysis plan in advance.
6. Don’t Stop a Test Because Today’s Results Look Exciting
You launch an experiment on Monday.
By Tuesday morning:
Variant B is winning by 34%!
Fantastic.
Time to end the experiment?
Not necessarily.
Early experiment results can fluctuate considerably.
If you’re using a traditional fixed-horizon statistical method, repeatedly checking the results and stopping as soon as one variation looks significant can increase the risk of false positives. Fixed-horizon approaches therefore rely on predetermined sample sizes and analysis rules.
Other experimentation platforms may use sequential or Bayesian methods that are designed differently and allow different approaches to monitoring results.
That’s an important distinction.
There isn’t one universal rule for every A/B testing platform.
Follow the statistical methodology used by your experimentation tool rather than inventing your own stopping rule.
And account for normal business cycles and unusual events.
A test run during Black Friday may tell you something about Black Friday shoppers — but that behavior may not represent an ordinary week in March.
7. Product Pages: What Should You Test?
Product pages are excellent candidates for experimentation because they’re close to the purchasing decision.
The original article correctly identified product layout, images, descriptions, reviews, and CTA elements as useful testing areas.
But don’t test something merely because it’s visible.
Look for actual customer friction.
Potential experiments include:
Product Images
Test meaningful differences such as:
product-only vs. lifestyle lead image,
different image sequences,
more prominent image galleries,
video demonstrations,
or additional product-detail photography.
Product Information
You might experiment with:
short vs. expanded descriptions,
specifications higher on the page,
size information,
compatibility details,
care instructions,
FAQs,
or comparison information.
Social Proof
Test whether legitimate customer reviews, ratings, testimonials, user-generated content, or other forms of proof are more useful in different positions.
Shipping and Returns
If customers regularly ask about shipping or returns, experiment with making that information more prominent.
Product Variants
Size, color, quantity, configuration, and other selectors can create friction when poorly designed.
Experiment with ways to make options easier to understand.
Calls to Action
CTA wording, hierarchy, size, position, and surrounding information may all be worth testing when you have a reason to believe the current design is creating friction.
Notice what’s missing from this advice:
“Make the button orange because orange converts better.”
There is no universally perfect button color, headline, page layout, or CTA.
Context matters.
8. Test Your Checkout Experience
Getting someone to add an item to the cart doesn’t guarantee they’ll complete the purchase.
Checkout friction can appear through:
unnecessary form fields,
forced account creation,
unexpected shipping charges,
confusing error messages,
limited payment options,
poor mobile design,
unclear delivery information,
or an unnecessarily complicated checkout flow.
The original article appropriately identified form length, guest checkout, and progress indicators as potential test areas.
But don’t automatically assume “shorter = better.”
Some information may be necessary.
Instead ask:
Which steps are creating unnecessary friction?
Potential checkout experiments include:
- guest checkout availability;
- field requirements;
- address-entry experience;
- order-summary placement;
- shipping information;
- delivery estimates;
- coupon-code presentation;
- payment-option presentation;
- progress indicators;
- error messaging;
- trust information; and
- mobile checkout design.
The closer an experiment gets to revenue, the more carefully you should watch guardrail metrics and technical implementation.
9. Test Offers and Pricing Presentation Carefully
Pricing experiments deserve more care than simply trying different numbers.
The original article suggested testing discount displays, price anchoring, and free shipping.
Some of those remain useful areas for experimentation, but pricing presentation should always be accurate and transparent.
Don’t manufacture a fictional “original price” merely to make the current price appear more attractive.
Instead, legitimate experiments might examine:
percentage vs. dollar discount presentation,
free-shipping thresholds,
product bundles,
quantity discounts,
subscription options,
legitimate sale messaging,
or where offer information appears.
For example:
Variant A: Free shipping on orders over $75.
Variant B: Free shipping on orders over $60.
Your primary metric might be purchase conversion rate.
But your guardrails should probably include:
average order value,
revenue per visitor,
shipping expense,
and contribution margin.
More orders don’t automatically mean a more profitable experiment.
10. Test Navigation and Product Discovery
One major area missing from the original article is how shoppers find products once they’re on your website.
This becomes particularly important as a catalog grows.
Potential experiments include:
navigation menus,
category organization,
filters,
sorting options,
site search,
related products,
recommended products,
recently viewed products,
breadcrumbs,
and category-page layouts.
Imagine an online clothing store with 2,000 products.
Improving how shoppers filter those products by size, color, price, and availability may have far more impact than changing the color of a CTA.
Ask:
Can customers find what they’re looking for quickly?
Product discovery itself can be part of conversion optimization.
11. Test Landing Pages
Landing pages remain another strong experimentation opportunity.
The original article identified headlines, imagery, and form placement as useful test candidates.
I’d expand that considerably.
You can experiment with:
headline and value proposition,
hero imagery,
product presentation,
offer clarity,
social proof,
page length,
content hierarchy,
CTA placement,
FAQs,
trust information,
and the relationship between the ad or link and the landing page.
But remember:
The goal isn’t simply to get more clicks on a button.
The goal is to improve the outcome the landing page is supposed to produce.
12. Test Email and Lifecycle Marketing
A/B testing isn’t limited to your website.
Email can also be tested.
The original article correctly identified subject lines, content, and CTA elements as possible experiments.
Depending on your email platform and audience size, you might test:
subject lines,
preheader text,
sender presentation,
email format,
offers,
product recommendations,
CTA wording,
send timing,
abandoned-cart messaging,
welcome sequences,
or re-engagement campaigns.
Just be careful about defining the metric.
A subject line might increase opens but produce fewer purchases.
An email might receive fewer clicks but generate more revenue.
The “winning” metric should reflect the actual objective of the campaign.
13. Mobile Deserves Special Attention
Your desktop store can be beautiful while your mobile experience quietly destroys conversions.
Don’t assume responsive design automatically means good mobile usability.
Test the experience on actual smaller screens.
Pay attention to:
navigation,
product galleries,
variant selectors,
sticky Add to Cart controls,
forms,
pop-ups,
payment methods,
site search,
filters,
page speed,
and checkout.
A product page that works perfectly with a mouse and 27-inch monitor may be frustrating when someone is shopping with one thumb.
If your analytics show meaningful differences between desktop and mobile performance, that’s worth investigating.
But don’t automatically create endless device-specific experiments.
Start with evidence of an actual problem.
14. Use GA4 to Measure E-Commerce Behavior
Your experimentation platform tells you about the experiment.
Your analytics system can help you understand the broader customer journey.
Google Analytics 4 supports recommended e-commerce events such as:
view_item
add_to_cart
view_cart
begin_checkout
add_shipping_info
add_payment_info
purchase
and refund.
That allows you to think about e-commerce behavior as a funnel:
Product View → Add to Cart → Checkout → Purchase
Suppose an experiment increases add-to-cart activity dramatically but purchases don’t change.
That’s useful information.
The experiment may have improved one stage of the journey without addressing friction farther down the funnel.
Or it may have encouraged lower-intent customers to add products without ultimately buying.
This is why looking at the entire customer journey is often more useful than celebrating a single engagement metric.
And remember: the relevant e-commerce events need to be implemented correctly for GA4 to receive them.
15. Choose an Experimentation Tool
If you’ve been working in digital marketing for a while, you may remember Google Optimize.
Don’t build a new testing strategy around it.
Google Optimize and Optimize 360 were discontinued on September 30, 2023. Google Analytics now supports third-party experimentation integrations instead. Google’s current documentation identifies integrations with providers including AB Tasty, Optimizely, and VWO, while its APIs allow other experimentation tools to integrate as well.
When evaluating an experimentation platform, consider factors such as:
traffic volume,
technical skill required,
statistical methodology,
GA4 integration,
e-commerce platform compatibility,
server-side capabilities if needed,
personalization features,
reporting,
cost,
and implementation complexity.
A small WooCommerce store and a multinational retailer don’t necessarily need the same experimentation infrastructure.
Choose a tool that fits the maturity and traffic of your business.
16. Analyze Results Without Declaring Every Test a Winner
Not every A/B test produces a winner.
That’s normal.
Broadly, an experiment might produce one of three useful outcomes.
The Variation Performs Better
The evidence supports implementing the change, assuming the effect is meaningful and important guardrails remain acceptable.
The Control Performs Better
That’s useful too.
You learned that your proposed change probably shouldn’t replace the existing experience under the tested conditions.
The Result Is Inconclusive
This isn’t failure.
You may not have collected enough evidence to distinguish the versions reliably.
The true effect may be smaller than the experiment was designed to detect.
Or the change may simply not matter much.
Don’t force every experiment into:
WINNER!
Sometimes the most valuable conclusion is:
“We don’t have enough evidence that this change improves the outcome.”
That can save you from implementing unnecessary changes.
17. Segment Carefully
After an experiment ends, you may want to understand whether different groups behaved differently.
Perhaps the overall result was neutral, but mobile customers appear to have responded differently from desktop customers.
Potential segments might include:
mobile vs. desktop,
new vs. returning customers,
traffic source,
geography,
customer type,
or product category.
Segmentation can reveal useful patterns.
It can also create a statistical trap.
If you slice an experiment into enough groups and examine enough metrics, eventually something may appear interesting purely by chance.
Use segments when there’s a meaningful reason to investigate them—not simply because you’re searching for a positive result.
Interesting segment patterns can also become hypotheses for future experiments rather than conclusions you immediately implement.
18. Document Every Experiment
A good experimentation program builds organizational knowledge.
Don’t run a test, look at the dashboard, make a change, and forget everything six months later.
Create an experiment record.
A simple template might include:
Problem:
What customer or business problem are we investigating?
Observation:
What evidence suggests the problem exists?
Hypothesis:
What do we believe will happen, and why?
Control:
What is the current experience?
Variation:
What are we changing?
Primary Metric:
What determines success?
Guardrails:
What other important outcomes must we monitor?
Sample Plan:
How much data does the methodology require?
Duration:
How long should the experiment run?
Result:
What happened?
Decision:
Implement, reject, iterate, or investigate further?
What We Learned:
What should influence future experiments?
Over time, that library becomes extremely valuable.
You may discover recurring patterns in what customers respond to—or assumptions your team repeatedly gets wrong.
SUCCESS BREWS TIP
Document losing and inconclusive experiments too. Knowing what didn’t improve customer behavior can be just as valuable as remembering what did.
The A/B Testing Framework
Putting everything together gives you a repeatable experimentation cycle:
Observe
Use analytics, customer feedback, usability research, support questions, reviews, and business data to identify problems or opportunities.
Hypothesize
Develop a reasonable explanation and a potential solution.
Prioritize
Not every idea deserves an experiment.
Prioritize based on factors such as:
potential business impact,
customer impact,
available evidence,
traffic,
implementation effort,
risk,
and your ability to measure the outcome.
Experiment
Create a controlled variation and expose eligible users according to your experimentation design.
Measure
Evaluate the predefined primary metric and guardrails according to the statistical methodology you’re using.
Learn
Ask what the experiment actually taught you.
Repeat
Use what you learned to identify the next worthwhile question.
Observe → Hypothesize → Prioritize → Experiment → Measure → Learn → Repeat
That’s a much stronger experimentation strategy than continuously changing random page elements.
Common A/B Testing Mistakes
Even a sophisticated testing platform can’t rescue a poorly designed experiment.
Watch for common problems such as:
- testing without a hypothesis;
- choosing metrics after seeing the results;
- stopping fixed-horizon tests early because one variation looks good;
- testing insignificant details while ignoring larger customer problems;
- running experiments without enough traffic;
- changing multiple unrelated things and then not knowing what caused the result;
- ignoring technical problems between variations;
- treating statistical significance as the same thing as business importance;
- ignoring revenue or profitability while optimizing clicks;
- over-segmenting results;
- running tests during unusual periods without considering seasonality;
- failing to document experiments; and
- assuming a successful test will produce the same result forever.
The objective isn’t to run the greatest number of experiments.
It’s to run experiments that help you make better decisions.
What If Your Store Doesn’t Have Enough Traffic for A/B Testing?
This is especially important for small businesses.
A/B testing sounds appealing, but some e-commerce websites simply don’t have enough traffic or conversions to run certain experiments efficiently.
If your store receives relatively few purchases, detecting a small change in purchase conversion rate may require an impractically large sample.
That doesn’t mean you can’t improve the website.
Use other forms of evidence.
You can:
conduct usability tests,
talk to customers,
review customer-service questions,
study search queries,
analyze checkout abandonment,
review session behavior where appropriate and privacy-compliant,
survey customers,
analyze product reviews,
or investigate obvious usability problems.
You don’t need an A/B test to fix a broken checkout button.
And you don’t need statistical significance to correct an obvious factual error.
Use experimentation when there’s genuine uncertainty worth resolving and enough data to learn something meaningful.
A Simple E-Commerce A/B Testing Checklist
Before launching an experiment, ask:
- Do we know what problem we’re trying to solve?
- Do we have evidence the problem exists?
- Have we written a clear hypothesis?
- Is the change meaningful enough to test?
- Have we defined the control and variation?
- Have we chosen one primary metric?
- Have we selected appropriate guardrail metrics?
- Does our experimentation method require a predetermined sample size?
- Do we understand the stopping rules for our statistical method?
- Are tracking and e-commerce events working?
- Have we checked the experiment on desktop and mobile?
- Could seasonality or promotions distort the result?
- Do we know how we’ll analyze the outcome?
- Are we prepared for an inconclusive result?
- Do we have somewhere to document what we learn?
If you can’t answer those questions, the experiment probably isn’t ready.
Final Thoughts: A/B Testing Is About Learning, Not Finding Winners
The purpose of A/B testing isn’t to turn every experiment into a conversion increase.
It’s to reduce uncertainty.
Sometimes Variant B wins.
Sometimes the original performs better.
Sometimes there isn’t enough evidence to confidently choose either one.
All three outcomes can teach you something.
The strongest experimentation programs don’t randomly test button colors hoping to stumble across additional sales.
They identify meaningful customer problems.
They create thoughtful hypotheses.
They decide what success means before looking at the results.
They measure carefully.
And they document what they learn.
Over time, those individual experiments can create something more valuable than a single conversion lift:
a better understanding of how your customers actually shop.
And that knowledge can influence much more than your website.
It can improve your product presentation, offers, email campaigns, checkout experience, customer service, content, and broader marketing strategy.
SUCCESS BREWS TIP
Don’t ask, “What should we A/B test?” Start with, “What customer problem are we trying to solve?” The best experiments begin with a reason — not a variation.
