Sample Ratio Mismatch (SRM) Checker
If you set a 50/50 split and one side ends up with noticeably more sessions, the test may be measuring which shoppers got counted rather than which experience works better. Enter the counts and planned split to find out in seconds.
Control: 40,700 vs 40,000 expected (50.88% vs 50%)
Variant B: 39,300 vs 40,000 expected (49.13% vs 50%)
p < 0.0005: assignment or tracking is broken. Do not read this test's conversion results; find the cause and rerun.
Talk to us about your measurementA planned 50/50 test with 40,700 control and 39,300 variant sessions gives chi-square ≈ 24.5 and p < 0.0001: a clear mismatch. That result cannot be trusted, however good the conversion numbers look.
Where mismatches come from in e-commerce
Common culprits: a variant that loads slower and loses impatient mobile shoppers before the analytics tag fires; redirect tests that break on some browsers or ad-click URLs; bot filtering applied after assignment; and theme or app conflicts that stop the testing script running on one version.
Each one removes a non-random slice of shoppers from one arm. The remaining groups are no longer comparable, so the conversion difference is contaminated in a direction you cannot know.
How the check works
The checker compares observed sessions with the counts your planned split implies, using a chi-square goodness-of-fit test. It flags a mismatch at p < 0.0005, the strict level large experimentation platforms use, and warns between 0.0005 and 0.01. Run it before reading any conversion metric.
Frequently asked questions
How much imbalance is normal?
It depends on volume. 5,050 against 4,950 is ordinary noise; 505,000 against 495,000 is a serious mismatch. Percentages alone mislead, which is why the test matters.
Can I fix a test with SRM by reweighting?
No. You do not know which shoppers went missing, so no adjustment removes the bias. Find the cause, fix it and rerun.
Does my testing tool check SRM automatically?
Some enterprise platforms do. Many lightweight tools and Shopify apps do not, or bury the warning. Check it yourself on every test.
More free tools
- Break-Even ROASThe ROAS and CPA your paid media must clear before a single order makes money.
- A/B SignificanceWhether your variant really won, how big the lift plausibly is, and whether you had the traffic to know.
- Sample SizeHow many sessions and how many weeks your test needs before you launch it.
- Stop or ContinueA straight answer on whether your test can end, and what checking it daily has cost you.
Ready to prove your marketing ROI?
Book a free 30-minute consultation. Bring your ad account numbers; leave knowing which channels earn their budget.
Free. No commitment.