What a low-order Shopify store should A/B test, and what to measure it by

That's the math Conal Brady laid out on X on October 21, in a post about test velocity in conversion rate optimization (CRO).

Low-order stores can still A/B test, but they have to pick their tests carefully. A test judged on orders runs for months, so it's only worth spending on big changes. Product page changes can be judged earlier on add-to-cart events. Whatever the test is judged on, roll out the winner only once real orders back it up.

"Traffic doesn't decide whether CRO works. It decides how fast you learn." (Conal Brady)

Divide monthly orders by 200 to see how many tests you can run each month

Conal Brady's math rests on one benchmark: each variant in a test needs roughly 100 conversions before the result starts to mean much. A test with two variants (the original and the change) therefore needs about 200 orders.

At 10,000 monthly visitors and a 2% conversion rate, a store gets 200 orders, enough for one clean test a month. At the same rate with only 1,000 visitors, it gets 20 orders, and the same test stretches to about ten months.

From there, Conal suggests dividing monthly orders by 200 to see how many tests a store can run per month.

By that math, a store doing 60 orders a month gets only three or four tests a year. The 100 figure is an estimate, so use it to picture your store's testing pace rather than as a hard threshold.

Save order-judged tests for offers and landing pages

When every test takes months, even a small change uses up a whole slot. A post promoting the A/B testing tool Visually describes a Shopify brand that lifted conversion rate 7.5% by adding a "$XX value" badge next to the discounted price on its sale product pages.

For a store doing 20 orders a month, proving a small change like that on orders alone would take most of a year.

One CRO writer on X argues the big difference lies elsewhere. Sharing the dashboard of a high-volume store, Josh Thompson wrote that a weak offer and one generic landing page for every ad angle don't get a store to that scale. The point Josh makes to readers: if your best month doesn't look like this yet, another badge test probably isn't the move that changes things.

JJosh Thompson@joshtCRO

Last 30 days: $4,731,973 in sales from 26.81K orders.

One spike on the graph gets close to $400,000 in a single day.

Sales were up 74%.

Orders were up 75%.

Conversion rate reached 3.63%, up 22%.

The same dashboard also shows what happens when that kind of volume hits the operation:

- 50+ orders waiting to be fulfilled
- 21 high-risk orders
- 8 chargebacks

I’m not naming the store, and I’m not going to pretend…

Oct 21, 2026 · View on X

When you think about the offer, start from how customers actually buy each product type.

Paddy, who does CRO and websites for Shopify stores, described a clothing brand their team worked with for three months. Instead of one sitewide discount, they built a separate offer for each category:

• T-shirts: quantity discounts, with savings growing as customers buy 1, 2, 3 or 4
• Hoodies: a hoodie plus sweatshirt bundle at one price, with customers free to mix colors and sizes

According to Paddy, the brand now has £10,000 revenue days, and average order value (AOV) is up 44% to £126.54. The team rebuilt the landing page first and then the offer, and these numbers come from both changes rather than from an A/B test, so the offer's share can't be separated out.

Still, the case points to one kind of offer worth testing: a different way to buy for each product group.

Any offer you test has to be built on real prices. Jonas Staben, a CRO specialist who audits landing pages regularly, lists discounts off an "original price" the product never actually sold at among the tactics Jonas considers illegal in most jurisdictions. The post points to EU price-indication rules, which require proof of the price before the discount, and to the FTC (US) calling the practice "fictitious pricing".

If you test savings badges or strikethrough prices, make sure the store really did sell at that original price and still has the records to prove it.

Product page changes can be judged early on add-to-cart

Besides choosing the right thing to test, you can shorten the wait by changing the metric you judge it on.

Conal Brady suggests judging product page tests on add-to-cart rate, because in Conal's experience add-to-cart events usually run 3 to 8 times higher than purchases. The same test that would take months on orders can reach a readable sample in weeks on add-to-cart rate.

For the 20-order store from the opening, 3 to 8 times means roughly 60 to 160 add-to-carts a month. Applying the same benchmark of about 100 per variant, the wait drops from around ten months to somewhere between just over one month and about three months.

The catch: add-to-cart is a proxy, and more customers adding to cart doesn't necessarily mean more customers buying.

That's why Conal recommends confirming with orders or revenue per session once enough orders have built up, before rolling the change out to the whole store.

In our view, this metric fits product page tests best, which is exactly the kind of example Conal gives. Offer tests like bundles or quantity discounts usually aim to change order value, and add-to-cart doesn't show how much a customer ends up buying.

Tests judged on new customers need checking against your Shopify customer list

Confirming with orders also protects you from another error that sits inside the testing tool itself. Arijan, who works on a team that audits and runs tests for brands, wrote on X on October 22 about a test the team had just run to lift new-customer conversion rate. According to the A/B testing tool Intelligems, the variant won with a lift of around 10%.

When the team matched the test data against the customer list exported from Shopify, they found 52% of the "new customers" in the test had bought from the store before, averaging 7 orders each. The real lift among new customers was only about 3%. Arijan also noted that this store has a lot of returning customers. So for stores with fewer returning customers, the gap may be smaller.

As Arijan explains it, an A/B testing tool counts someone as a new customer when their browser has no previous cookie. Cookies disappear when people clear their cache, switch browsers or switch devices, so a returning customer who clicks through from an email or searches for the brand name can still be counted as new.

Arijan's process has four steps:

1. Export the test data from the A/B testing tool
2. Export your full customer list from Shopify
3. Match email, name and address across the two files (Arijan uses AI for this step)
4. Keep only the people who have never placed an order. Those are your true new customers

Every test slot needs a question worth asking

The 20-order store from the opening gets about one order-judged test a year. That slot belongs to a question that could change revenue outright, such as whether customers buy more with a different offer, while product page tweaks can be read early through add-to-cart.

And however early the tool calls a winner, roll it out store-wide only when real orders in Shopify show the same result.