How to Test CRO Changes on a Low-Traffic Shopify Store

Quick answer: You can test CRO changes on a low-traffic Shopify store, but you usually cannot A/B test small changes. Work in this order: fix what is plainly broken without a test, measure moderate changes before and after against guardrails you wrote down first, and keep true A/B tests for big swings on pages and metrics where your traffic can detect them. Put your own numbers into Evan Miller’s sample size calculator before you commit: at an illustrative 2% conversion rate, detecting a 20% relative lift needs 19,784 visitors per variation.

Some advice on this topic gives a single visitor threshold, such as 10,000 a month or 1,000 per variation, and stops there. No single threshold fits every store. The traffic a test needs depends on four inputs: your current conversion rate, the smallest lift you would care about, how often you will accept a false alarm, and how often you will accept a miss. Change the lift alone and the answer moves by a factor of ten or more.

This guide shows those inputs, runs the sums on a named calculator, and then gives you a path for when the sums say “not enough traffic”: fix the obvious without a test, measure moderate changes before and after with guardrails, and test bigger swings properly. Underneath all three sits a no-test method, funnel drop-off and session recordings, which tells you what to fix. Every worked example with a store in it is hypothetical and labelled as such. None of it promises what any change will do for your store.

Why low traffic breaks ordinary A/B testing

An A/B test splits visitors into two groups, shows each a different version, and asks whether the gap in results is bigger than chance would produce. The number of visitors you need is the sample size. Evan Miller publishes a free sample size calculator, and Shopify’s own partner guidance points to it. Its inputs are the whole argument of this post.

The four inputs

• Baseline conversion rate. The share of visitors who buy today. Take it from your own store, not a benchmark. Shopify’s Conversion rate over time report shows the percentage of sessions that resulted in a purchase, and our guide to a good Shopify conversion rate explains how to measure your own baseline.
• Minimum detectable effect. The smallest change you want the test to see. Enter it as an absolute change (2.0% to 2.4% is 0.4 percentage points) or a relative one (the same move is a 20% relative lift).
• Significance level. How often the test flags a difference when none exists. The calculator’s default is 5%.
• Statistical power. How often the test catches a real effect of the minimum size. The default is 80%.

The answer comes as visitors per variation, so double it for the total. When the page loads, the baseline and effect boxes hold placeholder values (20% and 5 percentage points), so replace them with your own. Every figure below uses the 5% and 80% defaults. The calculator lets you move significance between 1% and 10% and power between 60% and 95%.

What the sums look like

Here is the calculator’s answer for an illustrative store converting at 2%. The baseline is a round number, not a benchmark and not a client figure.

These figures come from the calculator’s own published code, and the 20% row matches the live page (checked 30 October 2026). A 2017 Shopify Partners blog post gives the same figures for the 5% and 20% cases at a 2% baseline (309,928 and 19,784 per variation), a useful cross-check.

Halve the lift and you need roughly four times the visitors: 19,784 becomes 78,039, then 309,928. That is why button colours and headline tweaks are hard to test at modest traffic. Only large effects are visible.

What your traffic can detect

Turn the question around: given the visitors in each version, how big must a lift be before the test can see it? Same 2% baseline, same defaults.

“Reliably” means 80% of the time at 5% significance. A real lift smaller than the table’s figure will often show up as “no difference”.

Shopify’s A/B testing guide, published on 14 May 2026, says each variation should have at least 1,000 visitors to give meaningful data. Read that as a floor for having any data. At a 2% baseline, 1,000 visitors per variation can only reliably detect a near doubling of conversion.

Orders matter as much as visitors. In the 20% case, 19,784 visitors per variation at 2% is about 396 orders per variation. The 2017 Shopify Partners post says purchases, not just traffic, set the sample size: about 250 purchases per variation can sometimes suffice and 500 or more is more common, with “no magic number”.

A hypothetical store

Suppose a store gets 24,000 sessions a month and converts at 2%, which is 480 orders (made-up numbers). Split traffic 50/50 and each version sees 12,000 sessions a month. The calculator says that reliably detects a lift of about 26% relative, so 2.0% would have to reach roughly 2.5%. Run two months, 24,000 sessions per version, and the detectable lift falls to about 18%.

A lift of that size takes a big change, so this store should save A/B tests for big swings and not spend them on headlines.

Why there is no magic visitor number

The authors of a widely cited reference do not give one either. Ron Kohavi, Diane Tang and Ya Xu, in Trustworthy Online Controlled Experiments, recommend “thousands” of experimental units, say more units detect smaller effects, and note that smaller companies should start by looking for big effects. A 2014 KDD paper by Kohavi and colleagues mentions one writer’s suggestion of 10,000 monthly visitors and says the guidance should be refined to the metrics of interest.

So do your own sums. It takes five minutes:

1. Take your last 30 to 90 days of sessions and orders from Shopify and work out the conversion rate.
2. Enter it as the baseline, choose relative, and try 10%, 20% and 30% lifts.
3. Compare each answer with the sessions your page or funnel step gets in the time you will wait.
4. Write down the smallest lift your traffic can detect. That decides which rung of the path you are on.

Pick the cheapest honest metric

A yes/no metric, such as whether a session ended in a purchase, is the cheapest to measure. Money metrics cost more: in the 2014 paper, revenue per user at Bing was heavily skewed, and the authors’ rule of thumb for the average to behave normally was about 114,000 users per variant for it, against about 4,700 for sessions per user. Those are Bing’s numbers and a normality rule, not a power calculation for a Shopify store, but they show why revenue per visitor tends to need more traffic than a yes/no metric.

Metrics higher in the funnel have higher baselines, which makes them cheaper: at an illustrative 8% add-to-cart rate, a 20% relative lift needs 4,633 visitors per variation, against 19,784 for a 2% purchase rate. The catch is the 2014 paper’s fifth rule: reducing abandonment is hard and shifting clicks is easy, so local improvements are easy and overall ones much harder. A better add-to-cart rate is a lead. Purchases are the judge.

The decision path

Rung 1: Fix the obvious, and do not test it

“Obvious” has a narrow meaning: something fails for a visitor who is trying to do what you want them to do. A button or form that does not work in one browser. A payment method or discount code that errors at checkout. A promoted product that is sold out. A banner covering the add-to-cart button on a small phone. A JavaScript error when a visitor taps something.

Find these with test orders on a real phone, support emails, and Clarity’s dead click, rage click and JavaScript error filters. Our checkout leakage audit covers checkout.

Why not test a bug fix? First, waiting costs money: every day the bug stays live, visitors who hit it leave. Second, the sums defeat you. A bug affects a share of sessions, so its effect on your overall conversion rate is diluted, probably far below what your traffic can detect, and you would spend weeks unable to prove that a broken button should work. A plain defect is the clearest case where a test is not worth the wait.

Fix it responsibly:

1. Write down how to reproduce it: device, browser, page, steps.
2. Fix it.
3. Repeat the steps on the same device and confirm it is gone.
4. Log the date and what changed.
5. Watch the relevant funnel step over the following weeks as a check, not a trophy.

One caution: if a reasonable person could argue the change either way, it is not a bug. Colour, copy, layout and offer are opinions, and belong on rung 2 or 3.

Rung 2: Measure before and after, with guardrails

A before-and-after comparison is not an experiment. Nobody was randomly assigned, so it cannot prove your change caused the movement. In Kohavi, Tang and Xu’s hierarchy of evidence, randomised controlled experiments sit near the top (below only reviews of several experiments) and observational studies sit below them. The book recommends the less trustworthy methods, including observational studies, when a controlled experiment is not possible. Make it as trustworthy as you can and say plainly what it cannot tell you.

1. Write it down first. Before you touch anything, record the change, why you expect it to help, the one metric that decides it (usually purchases per session), the start date and a review date. A spreadsheet row stops you moving the goalposts afterwards.

2. Use equal windows of whole weeks. Shopify’s A/B testing guide recommends letting a test run for two weeks, and says it should run long enough to cover normal variability such as weekdays versus weekends. The 2017 Shopify Partners post says at least a full week, ideally two. Your calculator sums will show whether two weeks is nowhere near enough.

3. Make sure the baseline is ordinary. Do not compare across a sale, a launch, a price change, a big email send or a swing in ad spend. Our Black Friday checklist explains why a sale weekend should be reported separately, and the 2017 post warns that promotions and holidays skew results.

4. Change one thing. No app installs, theme edits or new campaigns during the window. Two changes at once cannot be separated afterwards.

5. Set guardrails before you look. A guardrail is a number that must not get worse, plus a rule for what you will do if it does:

• Traffic mix: the share of sessions by channel, new versus returning, and mobile versus desktop. If the mix shifts, conversion can move for reasons unrelated to your change.
• Average order value. More orders from smaller baskets may not be a win.
• Checkout starts. More people reaching checkout but fewer finishing suggests a problem pushed downstream.
• Stock. A sold-out hero product in one window and not the other will swamp most changes.
• Page speed and errors. A slower page can cancel a change’s benefit.

Set the rollback line before you see data, for example “if mobile checkout starts fall two weeks running, revert”.

6. Use a comparison group when you can. Change one collection template or product group and leave a similar one alone, then compare how both moved. It is not randomised, but it handles seasonality and traffic shifts better than a single before-and-after line. Our product page CRO checklist lists changes to try first.

7. Check the tracking before you celebrate. In the 2014 paper, Kohavi and colleagues describe Bing metrics that improved for months and were partly an artefact of better click tracking, not better user behaviour. Their general lesson is Twyman’s law: a surprisingly good figure is more likely an error than a breakthrough. On Shopify, compare orders in your analytics with orders in the admin first. Our guide to GA4 versus Shopify Analytics explains why they differ.

8. Read the result against what your traffic can detect. If the movement is smaller than the lift your traffic could detect, the honest summary is “unclear”, not “it worked”.

An example with hypothetical numbers

A store changes how it shows shipping costs on the cart page and compares two equal two-week windows. The numbers are invented.

Conversion rose 10% in relative terms. But each window holds about 12,000 sessions, and the calculator says that reliably detects a lift of about 26%. A 10% move sits inside what chance can produce, and the mobile share also rose. The honest write-up: “No harm seen and conversion was higher, but this traffic cannot tell a 10% change from noise. Keep the change if it is sensible and cheap, keep watching, and do not count it as a win.”

Before and after is not cheaper statistically than an A/B test: treat the windows as two groups and the same sums apply, as a best case, because the windows also differ in season, mix and promotions. It saves effort, not sample size.

Rung 3: Test a bigger swing properly

When your sums say your traffic can detect a large lift, and the change could plausibly deliver one, a real A/B test is worth running. A good candidate meets three conditions:

• It is a big swing: a restructured product page, a different offer structure, a new value proposition at the top of the page, not a tweak.
• The page is busy and the metric high enough in the funnel for the calculator to give a number you can reach.
• The result will change a decision. If you will make the change anyway, you are not testing.

An honest limit comes first. Big wins are rare. In the 2014 paper, Kohavi and colleagues report that at Bing most experiments fail, and those that succeed improve key metrics by 0.1% to 1.0% once diluted to overall impact. Bing is not a Shopify store, but the lesson carries: a low-traffic test will often come back flat. A flat result is still useful if you record it properly: “no lift of 26% or more was visible” is information, while “no effect” overstates it.

Running it in Shopify

Shopify’s built-in tool is Rollouts, under Markets in the admin. This is what Shopify’s Help Center said as at 30 October 2026:

• An Experiment compares a treatment against a control. Experiments need the Grow plan or higher.
• The default split is 50/50, and you can change it before saving. The default end date is 90 days after the start.
• Experiments can cover changes to your theme (editing or replacing it), a checkout and accounts configuration, and discount changes. You cannot change Liquid templates as part of a rollout, vintage themes are not supported, and headless storefronts cannot be tested.
• Theme experiments report conversion rate over time, bounce rate, reached checkout rate and add to cart rate; checkout experiments report checkout conversion rate.
• You cannot filter results by device or country. Bot traffic is not counted, and only visitors who consent are counted.

The Rollouts pages in the Help Center do not say which significance method or confidence level Rollouts uses, so I would not treat a “winner” label as the end of the story. Set the sample size and end date yourself from the calculator and judge the result then.

Five rules for the test itself

1. Fix the sample size and end date in advance, and do not stop early. Evan Miller’s article “How Not To Run an A/B Test” explains why. If you check repeatedly and stop at the first “significant” reading, the reported significance is meaningless. In his worst-case example, where significance is tested after every single observation, a change that did nothing was declared significant 26.1% of the time, more than five times the 5% you thought you were running, and peeking ten times turns what looks like 1% significance into about 5%. His advice: decide a sample size in advance and wait until the experiment is over. Peeking is fine, he says, so long as you can resist stopping.

2. Run whole weeks, so every weekday and weekend counts equally, and stay away from sales and holidays.

3. Name one primary metric. Purchases per session decides. Add-to-cart and checkout starts are information, because shifting clicks is easier than changing outcomes.

4. Decide the action for each outcome beforehand: win, flat or loss.

5. Do not slice afterwards. Splitting a finished test by device, channel or country turns one test into several smaller ones, each with the sample-size problem from the first table.

Loosening the settings

You can shrink the sample by accepting more risk. At the same 2% baseline and 20% relative lift, moving significance to 10% cuts the requirement from 19,784 to 15,640 visitors per variation, and dropping power to 60% as well cuts it to 8,940. The price is more false alarms and more missed real effects. In my view that suits only changes that are cheap to reverse. Evan Miller’s article also points to sequential and Bayesian designs for people who need to stop early; do not improvise those in a spreadsheet.

The no-test alternative: funnel drop-off and recordings

You do not have to test to learn a lot. Funnel data shows where visitors leave, and recordings suggest why.

Funnel drop-off

Shopify’s Conversion rate breakdown report shows the path through your store and checkout in four steps: all sessions, sessions with cart additions, sessions that reached checkout, and sessions that completed checkout. Shopify defines “reached checkout” as a session with user input during checkout, such as a key press or mouse click. An open funnel counts visitors who enter at any point; a closed funnel counts only those who complete the steps in order. GA4’s funnel exploration does the same job with up to 10 steps of your own choosing, open or closed, and up to four segments to compare.

An illustrative month for a store with 20,000 sessions:

The overall rate is 2.0%. The question is not “is 43.8% good?”, because there is no honest benchmark to hold it against. It is whether a step has weakened compared with your own earlier months, or loses far more people than the others. If the weak step looks like phones, our mobile conversion guide is the next cut to make.

Later steps have higher rates, so they need fewer people to test: at an illustrative 43.75% baseline, a 20% relative lift needs about 506 people per variation. Those people must have reached that step, so count them, not total sessions.

Session recordings with Microsoft Clarity

Microsoft Clarity is a free behavioural analytics tool. Its FAQ says it is free forever, with no traffic limits and no sampling, and separately that it records up to 100,000 sessions per project per day. You install it through the Microsoft Clarity app on Shopify, and Shopify Plus checkout pages are tracked through Shopify’s pixel. Allow a few hours for data to appear.

What to know before you rely on it:

• Recordings are reconstructions, not video. Clarity builds each session from captured HTML and user actions. A session ends after 30 minutes of inactivity.
• Retention is short. Recordings are kept for 30 days, then Clarity keeps 1% of them or 10 a day, whichever is higher, for up to 9 months. Favourited recordings are kept for up to 9 months, so favourite the ones you learn from.
• Filters do the finding. Rage clicks (repeated clicks in a clustered area), dead clicks (a click that gets no response), quick backs, excessive scrolling, JavaScript errors, click errors, scroll depth, device and funnel steps are all filters. The docs list a purchases filter for Shopify sites and a checkout abandonment filter for Shopify Plus only.
• Funnels link to recordings. Clarity builds funnels without code, shows the share of sessions that progressed or did not at each step, and lets you open recordings for either group.
• Shopify stays the source of truth. Clarity’s docs say its purchase conversion rate can differ slightly from Shopify’s.
• Privacy matters. Clarity can mask content, and Microsoft publishes sample privacy-policy wording. Check your policy covers it (general information, not legal advice).

A way to watch recordings that produces decisions instead of impressions:

1. Pick one question, for example “Why do mobile visitors who add to cart not reach checkout?”
2. Filter to it: device, funnel step, last 30 days.
3. Watch with inactivity skipped and log the session, device, what the visitor tried and what stopped them.
4. Tally. Problems that repeat across sessions are candidates; one-offs are anecdotes. Keep watching until new sessions stop adding new problems.
5. Size each candidate. A filter count shows how many sessions hit the dead click or error.
6. Route it. A bug goes to rung 1, a moderate change to rung 2, a big idea your traffic can test to rung 3.

Recordings show what visitors did, not why, and a handful of watched sessions sits nearer the anecdote end of the evidence ladder than the experiment end. Our guide to reading session recordings covers what to look for, and the Shopify CRO audit guide shows where they fit in a wider review.

A four-week plan for a low-traffic store

An illustrative sequence. Week 1: compare analytics orders with Shopify orders, record your baseline, install Clarity and write down the smallest lift your traffic can detect. Week 2: compare funnel step rates with your own earlier months and watch recordings of the weakest step. Week 3: fix anything plainly broken. Week 4: run one moderate change as a before-and-after or, if your sums allow, one big swing as a Rollouts experiment. Then repeat, one change at a time, logging date, change, method and verdict (“worked”, “no harm seen”, “unclear” or “reverted”).

Key takeaways

• Low traffic does not stop you improving a store. It stops you detecting small improvements by A/B test.
• Sample size depends on baseline conversion rate, minimum detectable effect, significance and power, so use your own numbers in Evan Miller’s calculator.
• At an illustrative 2% baseline, a 20% relative lift needs 19,784 visitors per variation, and halving the lift roughly quadruples that.
• Fix plain defects without a test. For moderate changes, compare equal whole-week windows, change one thing and write guardrails first.
• Keep true A/B tests for big swings, with sample size and end date fixed in advance.
• Funnel data and Clarity recordings find the problems worth fixing. Judge changes on purchases, not clicks.

FAQs

Can you A/B test a Shopify store with low traffic?

Only for large changes. At an illustrative 2% conversion rate, Evan Miller’s calculator (5% significance, 80% power) needs 19,784 visitors per variation to detect a 20% relative lift and 78,039 to detect a 10% lift. If your traffic cannot reach that, use a before-and-after comparison with guardrails, or fix defects directly.

How many visitors do I need for an A/B test on Shopify?

It depends on your conversion rate and the smallest lift you want to detect, so there is no single number. Shopify’s guide says at least 1,000 visitors per variation, but at a 2% baseline that only detects a lift of roughly 93%. Run your own inputs through the calculator.

How long should a Shopify A/B test run?

Long enough to reach the sample size you calculated, in whole weeks, ending on a date you set in advance. Shopify’s guide suggests two weeks as a starting point, and Rollouts experiments default to a 90-day end date, but the sample size decides, not the calendar. Do not stop early because the numbers look good.

Is a before-and-after comparison a valid way to measure a change?

It is weaker evidence than a randomised test, because nothing separates your change from seasonality, traffic mix, promotions or stock. It improves with equal whole-week windows, one change at a time, written guardrails and a tracking check. Report it as “the numbers moved this way”, not “the change caused this”.

Does Shopify have built-in A/B testing?

Yes. Rollouts, under Markets in the admin, runs Experiments that compare a treatment against a control, with a default 50/50 split, on the Grow plan or higher. The Help Center’s Rollouts pages do not say what significance method it uses, so set your own sample size and end date.

Not sure whether to test, fix or just watch?

If you want a second opinion on which rung of this path your store is on, start with the free diagnostic. We share your Shopify store on screen and name two or three conversion leaks live, and tell you whether they are worth fixing. No pitch. You can book a free 30-minute diagnostic, tell us about your store, or see how our Shopify CRO service works, which includes A/B testing where traffic supports it.