How to run an SEO experiment you can actually believe
You want a repeatable way to test changes rather than shipping and arguing about the graph afterwards.
Short answer
Write down the hypothesis, the metric, the window and the success threshold before you ship. Split comparable pages into a test group and a control group, change only the test group, wait the full window, then compare the difference between groups against the variation each shows on its own.
Write the prediction down first
One sentence: what you are changing, on which pages, what you expect to move, by roughly how much, and by when. Written in advance this is a test. Written afterwards it is a story, and the difference is the entire value of the exercise.
Choose a metric that can actually move
Clicks on low-traffic pages are too noisy to read in a month. Impressions move earlier and more smoothly. Click-through rate is the right metric for a title change because it isolates the snippet. Match the metric to what the change plausibly affects.
Build the two groups carefully
Similar traffic, similar page type, similar position band, similar recent trend. The easiest way to fake a result is to put your improving pages in the test group. Sort by a relevant metric and alternate assignment rather than splitting by category.
Set the window before you look
Four weeks minimum for most changes, in whole weeks. Write the end date down. The single most common way an honest test becomes a dishonest one is checking daily and stopping when the line looks good.
Change one thing
If you rewrite titles and add internal links and update the content, you will learn that something worked. That is not a finding you can apply anywhere. One variable per test is slower and it is the only way to accumulate knowledge rather than anecdotes.
Read it against the noise
Compare the difference between groups, then compare that against how much each group moved internally during the same window. If the between-group difference is smaller than the within-group variation, you have not measured an effect.
Record the result either way
Especially the failures. A documented negative result stops the same idea being retried every six months, which on a small team is worth as much as a positive one. Most organisations only write down the wins, which is why they keep relearning the same lessons.
Questions
- How many pages do I need per group?
- Enough that one outlier page cannot dominate the group, which in practice means at least ten or fifteen a side and preferably more. Below that the design cannot distinguish an effect from a single lucky page.
- Can I test on a small site?
- Page-level testing needs page-level volume, so often no. On a small site, prefer sequential comparison with an explicit noise check, and be honest that it is weaker evidence.
- What if the result is inconclusive?
- That is a real and common outcome. Record it, and either extend the window if the effect direction looks consistent, or accept the change is too small to matter at your size.
Measured, not asserted
This discipline exists because the alternative is documented: across 224 days on ihatepdf.cv, 160 pages produced 42,320 clicks, and 75.4% of those came from five pages. Without controlled comparison, every change shipped during the growth of those five would have looked like it caused the growth.
Free tool for this: Noise or Signal. No account, nothing uploaded.
Where this goes deeper
Every number on this page comes from one complete dataset: one product taken from zero to 100K+ users on search alone, with nothing spent on advertising. The full argument is Chapter 15 of the book. Five chapters are free to read.