How to Choose a Shopify CRO Agency (and Spot the Fluff)

Quick answer: Choose a Shopify CRO agency on its method, not its portfolio. A real one checks your tracking first, researches your store before it proposes a test, shows the evidence behind each hypothesis, and tells you plainly when your traffic is too low for A/B testing to give a reliable answer. If the first deliverable is a new design instead of a diagnosis, you are buying a design refresh, not CRO. Score every agency you speak to on the seven-point scorecard below, including us.

A familiar way to get burned: you hire a “conversion rate optimisation” agency. Three weeks in, you are looking at mood boards for a new homepage. Three months in, you get a report full of percentages with no baseline, and nobody can say which change caused what. The work was not necessarily bad. It was just not CRO.

This guide is for owner-operators of Shopify stores about to talk to CRO agencies. It covers what to ask, what a good answer sounds like, and what a bad one gives away. It will not tell you what results to expect, because nobody honest can promise that before looking at your store.

One disclosure first: many of the pages that rank for this question are written by agencies that sell CRO, and so is this one. Mark us down as hard as anyone else.

What you are actually buying

CRO is a method, not a deliverable. The loop is: measure how the store behaves now, find out why shoppers stop, form a specific hypothesis about a fix, make the change, and measure again. A redesign is a deliverable. It can be a good one, but it is a different thing to buy.

Optimizely’s A/B testing glossary page lists six steps for running experiments. Collecting data comes first, and the hypothesis step says to base predictions on existing data. Designing variations is step four, and a plan that jumps straight to it skips the part that makes the rest worth doing.

The simplest test of a proposal is to ask: what happens in week one? A good answer is access to your GA4 and Shopify data, a tracking check, a baseline conversion rate by device, and a review of how real shoppers move through the store. A weak answer is a brand workshop, wireframes, or a test calendar with dates already filled in. If week one is about design, the engagement is a design project, whatever the proposal calls it.

The scorecard

Score each criterion 0, 1 or 2 for each agency (0 = fails or dodges, 1 = partial answer, 2 = clear and specific). The maximum is 14. The rest of this guide explains each row.

A decision rule: a 0 on row 1, 3 or 5 is a reason to stop talking to that agency, however well they present. Those three decide whether you will ever be able to tell what worked.

1. Do they check your tracking before they touch your store?

Every report an agency produces depends on your numbers being right, so checking them should be the first piece of work.

Shopify and GA4 count differently, and some gap is normal. Shopify defines conversion rate as the percentage of sessions that resulted in a purchase (Shopify Help Center, behaviour reports). Its page on analytics discrepancies lists why other tools disagree with it: tools define sessions differently, Google Analytics only counts visitors with JavaScript and cookies enabled, browser extensions can block it, reporting time zones can differ, and each service uses its own tracking methods. A gap is expected. An unexplained gap is the problem. In our view, comparing order counts is the cleanest check, and a good agency can tell you roughly how big the order gap is and why. Our guide to GA4 versus Shopify Analytics shows how to compare them yourself.

A GA4 key event is recorded when its trigger fires. Google’s documentation defines a key event as an event that measures an action important to your business, says any collected event can be marked as one, and says it is recorded and shown in your reports once someone triggers it. If the trigger is set up wrongly, the count is wrong, and every conversion figure built on it is wrong too.

There is also a dated point. Shopify’s discrepancies page says its Analytics session measurement rollout ran from 21 to 23 September 2026, and that sessions, conversion rate and other session-based metrics might look different once it reached a store, which does not necessarily mean traffic, orders or shopper behaviour changed. A before-and-after comparison that straddles that window needs a note.

Ask: “How do you confirm a purchase is counted once? How will you reconcile GA4 with Shopify? What do you do if the tracking is wrong?” The right answer to the last one is “we fix it first and tell you what the old numbers really meant.”

Fluff: “We’ll set up conversion tracking in week two.” Tracking is the base for everything else, not a step to tick off later.

2. Do they research before they test?

Research is the unglamorous part of CRO and the easiest to fake. A credible research phase covers five things:

1. Quantitative analysis: funnel steps by device and channel, and where sessions drop between product page, cart and checkout.
2. Qualitative analysis: session recordings and heatmaps. Microsoft’s Clarity documentation, for example, lists session recordings and heatmaps (click maps and scroll maps) among its features. Our guide to reading Shopify session recordings explains what to look for.
3. Voice of customer: reviews, support tickets and pre-purchase questions.
4. Competitor and usability review: how close competitors answer the same buyer questions, and where your pages break usability basics.
5. Copy review: whether pages answer the objections that actually stop people.

An agency does not need these exact five, but you should see something equivalent. Ask for a sample research output with another store’s details removed. If an agency cannot show what its research looks like, it may not produce any.

Then ask: “What would make you tell me CRO is the wrong thing to spend on right now?” Sometimes the honest answer is too few visitors, or traffic quality (see whether to fix SEO or CRO first). Our guide to a Shopify CRO audit lists what a full diagnostic should contain. An agency that never says “not yet” is selling a package, not a diagnosis.

3. Can they show the evidence behind each hypothesis?

This is the best single filter in the guide, because it is hard to fake on the spot. A proper hypothesis has four parts: what was observed, what it probably means, what will change, and how success is measured. For example (illustrative, not from a real store):

Mobile recordings show shoppers opening the size guide on a product page, scrolling, then leaving without adding to cart. The funnel shows mobile add-to-cart rate on that template well below desktop. Hypothesis: a short fit summary directly above the add-to-cart button will raise mobile add-to-cart rate on that template. Measure: mobile add-to-cart rate first, then orders per mobile session.

The fluff version is “We’ll add trust badges and a sticky add-to-cart button because they usually work.” That is a list of tactics, and nobody has shown it fits your store.

Ask what evidence sits behind each proposed change. From strongest to weakest: your own quantitative data; observed behaviour on your store; what your customers say; what competitors do; general best practice; a hunch. Expect the first three behind the biggest changes. Best practice is a fair tiebreaker, but as the only reason it means the research was thin.

Ask to see three proposed hypotheses with their evidence, and why this one comes first. A good agency has a prioritisation method it can explain in a sentence. One honest limit: evidence reduces guesswork, it does not remove it. A well-evidenced hypothesis can still lose, and you want an agency that reports losses as readily as wins.

4. Are they honest about low traffic and statistical significance?

This is where agencies are most tempted to stay vague, so ask precise questions.

The terms. Optimizely’s glossary describes statistical significance as a measure of how unusual your experiment results would be if there were no real difference between the variation and the baseline. In plain terms, it checks whether a gap is likely to be real or just luck. The same page says larger samples generally give more reliable results.

What a test costs in visitors. Optimizely gives two formulas: total visitors needed equals the sample size per variation times the number of variations, and estimated days equals total visitors needed divided by average daily visitors. An illustrative example, with made-up numbers to show the arithmetic:

Suppose a sample-size calculator says you need 15,000 visitors per variation to detect the improvement you care about. With the current page plus one variation, that is 30,000 visitors. At 500 visitors a day, the test needs about 60 days. At 250 a day, about 120 days. At a 2% conversion rate, 15,000 visitors is about 300 orders per variation.

That last line is the real constraint. Because you are comparing conversion rates, conversions, not visits, give a test its power. A store with plenty of traffic but few orders can still be unable to run a quick test.

What the platforms suggest. Optimizely’s page says to run tests for at least one business cycle, which it gives as 7 days. Shopify’s A/B testing guide (published 14 May 2026) suggests two weeks, at least 1,000 visitors per variation, and 95% or higher confidence before declaring a winner. Treat those as floors. Our view, a judgement and not a statistic, is that sample size should come from your baseline and the smallest improvement worth detecting, not a round number.

What an honest agency does with low traffic. If the arithmetic says a clean test would take many months, it says so and moves to one of these:

• Fix clear, well-evidenced problems directly (a mis-firing event, a broken mobile menu) and measure before and after, with caveats.
• Test only larger changes, where a bigger effect needs a smaller sample.
• Put more effort into research, so each change is more likely to matter.

A before-and-after comparison is weaker than a test, because anything else that changed (a sale, a new campaign, a tracking change) is mixed in. That is acceptable if the agency says so and logs what else changed, and not acceptable if it is presented as a controlled test. Our guide to A/B testing on a low-traffic Shopify store goes deeper.

Fluff: “We’ll run three tests a month,” said before anyone has looked at your order volume.

5. Do you agree how results will be judged before launch?

The classic way to fool yourself is to stop a test the first time it looks like a winner. Evan Miller’s article “How Not To Run An A/B Test” explains why: the usual significance calculation assumes the sample size was fixed in advance. In one of his worked examples, with a 50% baseline conversion rate and a change that does nothing, stopping at the first moment the result reached 5% significance (or after 150 observations) produced a false positive rate of 26.1%, more than five times the nominal 5%. His advice is to fix the sample size in advance and not stop early on significance, unless you use a sequential or Bayesian design built for interim looks.

You do not need the maths. You need answers to four questions before launch: what is the primary metric (for most stores, orders or revenue per session, not clicks); what are the minimum run time and sample size; what counts as a win, a loss and “no clear result”; and who decides when to stop, on what rule. An agency that answers “we’ll see how it looks” has left itself room to call anything a win.

6. How to spot a design refresh sold as CRO

A redesign can be the right move. A theme that is genuinely broken, or cannot carry the changes the research calls for, is a fair reason to rebuild. The problem is a redesign that is labelled CRO, billed as an ongoing retainer, and judged by taste instead of shopper behaviour. These questions separate the two:

1. What problem in our data does the new design solve? A real answer points to a finding: “mobile shoppers drop at the product page because of X”. A weak answer talks about “modernising” the brand.
2. Can the new design be tested against the current one? Shopify’s changelog (5 June 2026) says its Rollouts feature lets you run an A/B test between two entirely different themes or checkout setups, which is one way to compare a rebuilt store with the existing one. The announcement does not say which plans include it, so confirm that in your admin (it sits under Markets, then Rollouts). An agency that cannot say how a redesign will be compared with the current theme has not planned to measure it.
3. What stays the same? If everything changes at once, you cannot tell which change helped or hurt.
4. What is the rollback plan? If conversion drops after launch, how fast can the old version return?
5. Is the scope fixed or ongoing? If a rebuild is billed as monthly “CRO”, ask what the monthly work is once the design ships.

More warning signs: the first meeting is about brand and visual references, not your funnel, and the deliverables are screens, not findings. The logic works in reverse too: an agency that says a redesign is never worth doing deserves the same scrutiny. The research decides.

7. Read their proof like an analyst

A percentage on its own tells you very little. For each result an agency shows you, ask for five things:

1. The baseline: the rate before, not just the change.
2. The window: which dates were compared.
3. The source: which tool produced the numbers (Shopify Analytics, GA4, a testing tool).
4. What else changed: sales, ad spend, seasonality, a product launch, a tracking fix.
5. Test or before-and-after: they are not equally strong, and the agency should say which it was.

A result with all five is worth reading. A result with none is decoration. Our guide to what results a CRO agency can deliver on Shopify covers why results vary.

Never accept a guaranteed conversion rate. Nobody can promise one before seeing your store, traffic and product. Whoever does is guessing, or planning to define “conversion” so that they hit it.

Access, ownership and terms

• You own the accounts. GA4, any recording tool and your Shopify store stay in your name, with the agency given access.
• Know who builds and who works. Some agencies hand a list to your developer, others build in your store. Either can work, but insist on approving each change before it goes live, and ask whether the person on the sales call is the person doing the research.
• Reports should tie to money. Orders per session, revenue per session and average order value do. Impressions and page views mostly do not.
• Prefer terms you can leave. In our view, a long lock-in is hard to justify when you can check the method as you go.

Where we sit on this scorecard

You should score us the same way. Row by row:

• Tracking and research (rows 1 and 2): our CRO audit has six phases: quantitative analysis (starting with a tracking check), qualitative analysis (recordings, heatmaps, dead-click maps, reviews), a review of three of your closest competitors, a usability review, a copy review, and a recommendations report. It runs over 14 days for most stores.
• Evidence (row 3): each recommendation is scored on potential, importance and ease (PIE).
• Traffic (row 4): we run A/B tests where your traffic supports them, and multivariate tests only where traffic is high enough. Otherwise we measure before and after and say so.
• Fixes and terms (row 7): we build the approved fixes in your Shopify store, you approve each change first, and engagements are month to month. I run the work myself, so you are not handed to a junior.
• Rows 5 and 6: put these to us exactly as you would to anyone else, and do not take our word for the answers.

Where we may not fit: we do CRO, SEO and GEO for Shopify stores. We are not a design studio and we do not manage paid ads or email, so if you need those, we are the wrong choice.

Want to see how we score? Tell us about your store.

Key takeaways

• Choose on method: tracking, then research, then hypotheses with evidence, then changes and measurement.
• Ask “what happens in week one?” If the answer is design work, you are buying a design project.
• Every proposed change needs the observation behind it. “Best practice” is a tiebreaker, not evidence.
• Conversions, not visits, limit what you can test. An honest agency says when a clean A/B test is unrealistic and how it will measure instead.
• Agree the metric, minimum run time and stopping rule before launch.
• A redesign sold as CRO should still say which problem in your data it solves, how it is tested against the current theme, and how you roll back.
• Ask for the baseline, window, source and context behind every result. Never accept a guaranteed conversion rate.

FAQs

What does a Shopify CRO agency actually do?

It measures how your store converts now, finds out why shoppers stop, proposes evidence-backed changes, makes or specifies them, and measures the effect. If the description is mostly about design or ad management, it is selling something else.

How much traffic do I need before hiring a CRO agency?

There is no universal number, because conversions, not visits, limit a test. An agency should work it out from your baseline conversion rate, the smallest improvement worth detecting and your daily sessions on the page. If that points to many months, the right plan is direct fixes with before-and-after measurement, not a test calendar.

Should a CRO agency guarantee a conversion rate increase?

No. Nobody can know in advance how your store and traffic will respond. A guarantee usually means the agency controls how the number is defined or measured. Look for a clear method and honest reporting instead.

What is the difference between a redesign and CRO?

CRO is a method: research, hypothesis, change, measurement. A redesign is one change the research might recommend. It counts as CRO if it is tied to findings and tested or measured against the current store, not if it is driven by taste and judged by opinion.

How long should a CRO test run?

Long enough to reach a sample size agreed before launch, and at least one full business cycle. Optimizely gives that minimum as 7 days, and Shopify’s guide suggests two weeks with at least 1,000 visitors per variation. Those are floors, and a test should not be stopped early because it looks like a winner.

Should I pay for a CRO audit before committing to a retainer?

A defined audit with a written output is a reasonable first step, as long as you can take the findings to anyone, including another agency or your own developer. Check that it delivers findings with evidence and priorities, not a generic checklist.

Want to run this scorecard on your own store?

The quickest way to test it is on a live call: tell us about your store or book a free 30-minute diagnostic. We share your Shopify store on screen and name two or three conversion leaks live, and tell you whether they are worth fixing. No pitch. Score us on the same seven points afterwards, or see how our Shopify CRO service works if you want the detail first.