You cannot A/B test SEO the way you test a landing page
You want to test an SEO change properly instead of shipping it and hoping the line moves.
Short answer
A conventional A/B test splits users randomly, which is impossible for SEO because the unit being evaluated is the page, not the visitor, and there is one search engine seeing one version. Valid SEO testing splits pages instead: apply a change to a matched group, hold back a comparable control group, and compare the two over time.
Why the usual method breaks
Random assignment requires showing different versions to different visitors. Search engines see one version, and showing them something different from users is cloaking. So the visitor-level split, which is the entire mechanism of conventional testing, is unavailable.
The unit of analysis is the page
Split pages into a test group and a control group. Apply the change only to the test group. Compare how each group moves over the following weeks. This is a quasi-experiment rather than a true one, and it is the best available design outside a lab.
Matching the groups is most of the work
The groups must be comparable before you start: similar traffic levels, similar page types, similar positions, similar trend. Splitting your best pages into test and your worst into control guarantees a result and teaches you nothing.
You need enough pages
Page-level testing needs page-level sample size. Testing a change on four pages produces a result you cannot distinguish from noise. This is why the technique suits templated sets and why it is largely unavailable to sites with thirty pages.
Everything else keeps moving
Seasonality, algorithm updates, competitor changes and your own other work all land during the test window, and they land on both groups. That is exactly why the control group exists: it absorbs the shared movement so the difference between the groups is the part you can attribute.
The timescale is unforgiving
Recrawling, re-evaluation and reranking take weeks. A test read after five days is reading crawl noise. Decide the window before you start and do not read the result early, because reading early and stopping when it looks good is how noise gets published as a finding.
Check the change against the baseline noise
Before believing a difference, compare it against the range each group moves in anyway. If the effect is smaller than the swing the control group produced on its own, you have not measured anything, whatever the percentage looks like.
Questions
- Can I test on a single page?
- Not reliably. A single page has no control and its own variance is large. You can use it to check that a change deployed correctly, which is a different and useful thing.
- How long should an SEO test run?
- Weeks, and ideally a period that covers whole weeks so weekday cycles balance. Decide the window in advance and hold to it.
- What if I cannot hold back a control group?
- Then you are not testing, you are shipping and observing, which is legitimate as long as you describe it that way. Compare against the baseline variance before claiming the change did anything.
Measured, not asserted
The noise is not small. On ihatepdf.cv, one month shipped zero commits and traffic still grew 61%, from 4,409 to 7,115 clicks. Any test running that month with no control group would have credited whatever was shipped elsewhere with an increase that had nothing to do with it.
Free tool for this: Noise or Signal. No account, nothing uploaded.
Where this goes deeper
Every number on this page comes from one complete dataset: one product taken from zero to 100K+ users on search alone, with nothing spent on advertising. The full argument is Chapter 15 of the book. Five chapters are free to read.