GemNews

All stories · Article

Your product page test won on conversion rate. Is it BFCM-ready?

Not yet. Returns land weeks after an experiment ends, so a product page test that wins on conversion rate can still cost more than the report shows once they arrive.

By Janie Nguyen · · 7 min read

Share: WhatsApp · Telegram · X · Facebook · LinkedIn · Email

Karim Zitouni, co-founder and CEO of the Shopify A/B testing app Elevate A/B Testing, made that point on LinkedIn on October 26, 2026.

Your test report never sees the returns

Picture a common setup. A product page variant wins a few weeks before Black Friday. More visitors buy, the testing tool calls it significant, and the team ships it to everyone. Weeks later, returns come in as refunds in Shopify, in a report someone else reads, and nobody connects them to the test.

Zitouni put the risk in one line: "A test can lift your conversion rate and raise your return rate at the same time, and nothing in the result will tell you." By the time the returns arrive, the winner is already live. He adds that the page that sold more may also have sold more of the wrong thing to the wrong people.

The result you read on test day is only half the result.

A conversion win can hide a loss in more than one place

Returns are one way a conversion win goes wrong. Practitioners shared two others in October 2026. Neither is about returns, but both show conversion rate picking the wrong winner. Goldi Kambale (@Goldikam), a self-described Shopify CRO expert on X, ran a discount test that lifted conversion rate by 572%, while average order value (AOV) dropped 24.7% on the same test, so the store ended the week with more orders and less revenue. Kambale calls this the trap most brands walk into at BFCM. Arijan Janes, a CRO practitioner posting on LinkedIn, saw Intelligems, the testing tool, report about a 10% lift in new-visitor conversion rate. After matching the test data against Shopify customer data, 52% of those "new visitors" had bought before, with 7 prior orders on average and 44 for one of them, and the true new-customer lift was about 3%.

Late data changes the picture too. An anonymous seller on r/ecommerce, u/bayouski, described campaigns that looked profitable, then looked noticeably worse when they checked the same dates a few weeks later: Meta and Google had backfilled conversions, and more refunds had come through in Shopify. By then they no longer had the numbers they had used to decide.

You decide on the numbers you have that day.

Why does gift season make the gap wider?

Zitouni's reason for raising this now is the buyer. In gift season, many shoppers choose for somebody else, with much less information than someone buying for themselves.

Shopify's returns guide on its Enterprise Blog cites the 2025 Retail Returns Landscape report from the National Retail Federation (NRF) and Happy Returns, and says retailers expect 17% of holiday sales to be returned. That survey covers US businesses with more than $500 million in revenue, so treat it as a big-retailer benchmark, not a figure for your store. The same guide says apparel and footwear run above average because fit is hard to judge online.

The return rate section of Shopify's returns guide, with the line about 17% of holiday sales highlighted, on Shopify, captured October 2026

The holiday returns benchmark in Shopify's returns guide, captured in October 2026.

Sell clothing, shoes or anything with a size? This risk is mostly yours. Stores selling items that rarely come back, or running tests that do not change who buys, may see a smaller gap, but only your own numbers will show it.

Size and length options with a size guide link on the Rhiona denim dress page, on Unfolded, captured October 2026

Every choice here is a guess the gift buyer makes for someone else: the size guide (1) and three lengths (2) on Unfolded, captured October 2026.

What to check before the winner runs through BFCM

Treat these as checks to run, not rules:

  1. Write down your current return rate before the test starts. This is Zitouni's own advice for anyone testing a product page before November. Without it, a later rise has nothing to compare against.
  2. Match each variant's orders against returns: export the orders from your testing tool and look up their refunds in Shopify. A commenter under Zitouni's post suggested tracking return rate by product and by test variant. Borrow the export-and-compare approach Janes used to find returning customers: export each variant's orders from your testing tool, export refunds from Shopify, and match them. If the winning variant returns more than your baseline, the win is not confirmed.
  3. Save the numbers you decided on, and book a review date after your own return window closes for orders placed during the test. Then compare each variant's revenue after refunds against the saved numbers, and roll back if returns wiped out the gain.

Buy box of Hillman's insulated hunting boots with a size guide and a 90-day returns line, on Hillman, captured October 2026

The return window sets your review date: boots on this page can come back within 90 days (2), even with a size guide (1) on the page, on Hillman, captured October 2026.

A win counts once the returns are in.

Frequently asked questions

Should I keep the test running until returns come in, or review it after 30 days?

A commenter under Zitouni's post asked exactly this. It depends on your return window. If you have to ship before BFCM, ship, then review once the return window closes on the test orders. Another commenter put it simply: if returns arrive a month after the test, the team needs to revisit the result.

Does a higher conversion rate always mean more returns?

No. Zitouni's point is that it can, and that the test result will not show it. The only way to know for your store is to measure returns for each variant.

Sources

Discuss on GemCommunity

esc