Noise or Signal

Drop two Search Console exports, or one and the date you shipped, and find out whether the change is bigger than the noise this metric makes anyway. Runs a Welch t-test against the period’s own variance, checks the windows for uneven weekdays, and refuses to call causation.

How to use it

  1. In Search Console, open Performance, set a range that covers well before and well after the thing you shipped, and click Export → Download CSV. You want Dates.csv from the zip.
  2. Drop it here and pick the day you shipped. Or drop two separate exports and it compares them directly.
  3. Read the verdict, then read the warnings under it. The warnings are the part that usually changes the answer.

What the numbers mean

Noise
The range a metric moves in while nothing is happening. Every site has one and almost nobody measures it, which is why so many case studies are describing weather. This computes yours from the period before your change and compares the change against it.
p-value
The chance of seeing a gap this big if nothing had really changed. Below 5% is the usual bar. It is a necessary test and not a sufficient one: a change can clear it and still be smaller than a swing the baseline already made on its own, which is why that swing is reported next to it.
Weekday balance
Search traffic runs on a weekly cycle. A 30-day window against a 28-day one contains two extra weekdays, and that alone can manufacture a few percent of “growth”. Use multiples of seven, or read the warning.
Effect size
How large the difference is in standard deviations, independent of how many days you measured. A large enough sample makes tiny differences statistically significant; effect size is what stops you from acting on one.

Questions

Does this prove my change caused the difference?

No, and nothing built on one site’s Search Console can. There is no holdout group: the same site is measured before and after while season, competitors, demand and algorithm updates all move underneath. A verdict of Real means the difference is bigger than this metric’s own noise, which makes it worth investigating. Any tool that claims more than that is selling something.

Why does it warn me about average position and CTR?

Both are averages over a query set that changes as you grow. Start ranking for a hundred new terms at position 40 and your average position gets worse while your clicks rise. Site-wide CTR does the same thing: in the reference dataset it fell from 6.02% in April to 2.72% in July while clicks went from 4,409 to 16,410. Neither number was lying, and neither was a problem.

How much data do I need?

Seven days each side is the floor and it is a low one. Twenty-eight each side is where the answers get stable. The tool estimates how many more days would settle an inconclusive result.

Is my Search Console data uploaded?

No. The file is parsed and analysed in your browser. There is no server, no account and no storage. Load the page, disconnect from the internet, and it still works.

Guides that use this tool

Other free tools