Statistical significance in SEO, without the false confidence
Traffic moved after you changed something and you want to know whether the movement means anything.
Short answer
Significance testing answers one narrow question: how likely is a gap this large if nothing had really changed. It does not tell you what caused the change, and on a large enough sample it will flag differences too small to care about. Pair it with effect size and with the baseline’s own worst swing before believing anything.
What the test is actually doing
Comparing two periods and asking whether their difference is larger than the variation each shows internally. A p-value below five percent conventionally means "unlikely to be chance". That is a statement about chance, not about your change, and the distinction is where most SEO claims break.
Why a p-value alone misleads here
With enough daily observations, a trivial difference becomes statistically significant. A one percent lift can clear the bar and be worth nothing. Significance says a difference is probably real; it says nothing at all about whether it is worth the work.
Effect size is the second number
Effect size measures how large the difference is relative to the variation, independent of how many days you measured. A large sample with a tiny effect is a finding you should ignore. A small sample with a large effect is a finding worth confirming.
The third check nobody runs
Compare your change against the largest swing the baseline period made against itself with nothing shipped. If your result is smaller than something the metric already did on its own, no p-value should persuade you. This single check kills most published case studies.
Use the right test
Search periods rarely have equal variance: a growing site’s later window is both larger and more volatile. Welch’s t-test does not assume equal variance and costs nothing extra, which makes assuming equal variance a choice with no upside.
Whole weeks, always
Search traffic has a weekly cycle. Comparing windows with different weekday composition builds a difference into the comparison before you measure anything. Multiples of seven eliminate it.
What it can never give you
Causation. There is no holdout: the same site is measured before and after while season, competitors, demand and algorithm updates all move. Significance makes a change worth investigating. It is not evidence that your change produced it, and any tool implying otherwise is overselling.
Questions
- What p-value should I use?
- Five percent is the convention and it is arbitrary. Treat it as a threshold for looking harder rather than a verdict, and always read it next to the effect size.
- How much data do I need?
- Seven days each side is a floor and a poor one. Twenty-eight each side is where answers start being stable for most sites. Below that, variance dominates.
- Can I test CTR and position this way?
- You can, and be careful: both are averages over a query set that changes as you grow. A worsening average position alongside rising clicks is expansion, not decline, and a test will happily report it as a significant negative.
Measured, not asserted
The reason to check against baseline noise is a real month: ihatepdf.cv shipped zero commits in May and grew 61%, from 4,409 to 7,115 clicks, on the way to 100,542 users. Any test attributing that month to a change would have been significant, large, and wrong.
Free tool for this: Noise or Signal. No account, nothing uploaded.
Where this goes deeper
Every number on this page comes from one complete dataset: one product taken from zero to 100K+ users on search alone, with nothing spent on advertising. The full argument is Chapter 15 of the book. Five chapters are free to read.