Free tool · Experimentation

Sample Ratio Mismatch (SRM) Checker

If you set a 50/50 split and one side ends up with noticeably more sessions, the test may be measuring which shoppers got counted rather than which experience works better. Enter the counts and planned split to find out in seconds.

Verdict
SRM detected
p-value
< 0.000001
Chi-square (df 1)
24.50

Control: 40,700 vs 40,000 expected (50.88% vs 50%)

Variant B: 39,300 vs 40,000 expected (49.13% vs 50%)

p < 0.0005: assignment or tracking is broken. Do not read this test's conversion results; find the cause and rerun.

Talk to us about your measurement
Worked example

A planned 50/50 test with 40,700 control and 39,300 variant sessions gives chi-square ≈ 24.5 and p < 0.0001: a clear mismatch. That result cannot be trusted, however good the conversion numbers look.

Where mismatches come from in e-commerce

Common culprits: a variant that loads slower and loses impatient mobile shoppers before the analytics tag fires; redirect tests that break on some browsers or ad-click URLs; bot filtering applied after assignment; and theme or app conflicts that stop the testing script running on one version.

Each one removes a non-random slice of shoppers from one arm. The remaining groups are no longer comparable, so the conversion difference is contaminated in a direction you cannot know.

How the check works

The checker compares observed sessions with the counts your planned split implies, using a chi-square goodness-of-fit test. It flags a mismatch at p < 0.0005, the strict level large experimentation platforms use, and warns between 0.0005 and 0.01. Run it before reading any conversion metric.

Frequently asked questions

How much imbalance is normal?

It depends on volume. 5,050 against 4,950 is ordinary noise; 505,000 against 495,000 is a serious mismatch. Percentages alone mislead, which is why the test matters.

Can I fix a test with SRM by reweighting?

No. You do not know which shoppers went missing, so no adjustment removes the bias. Find the cause, fix it and rerun.

Does my testing tool check SRM automatically?

Some enterprise platforms do. Many lightweight tools and Shopify apps do not, or bury the warning. Check it yourself on every test.

More free tools

Currently accepting new clients

Ready to prove your marketing ROI?

Book a free 30-minute consultation. Bring your ad account numbers; leave knowing which channels earn their budget.

Review your current attribution and measurement setup
Identify the highest-priority evidence gaps
Leave knowing which channels earn their budget

Free. No commitment.

No obligation. We reply within one business day.