Averages Describe Nobody

A short chapter about a number I stared past for seven months.

Three quarters of this site's impressions come from desktop. Desktop is also, by both available measures, the worst-performing surface it has:

ImpressionsClicksCTRAvg position
Desktop757,08724,2533.20%9.39
Mobile252,63712,4754.94%8.19
Tablet11,0237686.97%6.03

Mobile ranks 1.2 positions better and converts at 1.5× the click-through rate, on a third of the volume. Tablet is better again on both, on almost no volume at all.

Why the aggregate hides this

Every headline number in this book (the 3.44% overall CTR, the ~8.5 average position) is a blend of these three, weighted by desktop because desktop is 74% of impressions.

That is the same error as Chapter 16's CTR trap and Chapter 9's brand-vs-non-brand split, in a third costume: an average across populations that behave differently is a number describing nobody.

Before you interpret any aggregate, ask what populations got averaged into it.

Three so far in this book: brand vs non-brand, position band, and now device. Each one, split out, said something the blended number concealed.

What the split actually means

Two readings, and I cannot fully separate them with this data.

Reading one: the results pages are different. Mobile results have fewer visible slots, more aggressive vertical stacking, and often a different competitive set. Being eighth on mobile can mean something quite different from being eighth on desktop.

Reading two: mobile intent is more urgent. Someone on a phone with a document problem tends to want it solved now, not compared across five tabs. Higher intent, higher click-through.

The tablet row is where I stop trusting my own explanation. 6.97% CTR and position 6.03 is the best performance on the site, on 11,023 impressions, about 1% of the total. That is a small enough sample that I would not build anything on it, and I am flagging it rather than constructing a story about tablet users being more decisive, which is the kind of thing that sounds insightful and is usually noise.

What to do with it

1. Segment every report by device before drawing conclusions. Search Console does this in one click and almost nobody uses it. If your desktop and mobile positions differ by more than a point, you effectively have two sites.

2. Check whether your page is a different page on mobile. Not the design, the content. Collapsed accordions, hidden depth sections, and lazy-loaded FAQ blocks are all fine for humans and all reduce what is present for a crawler. If your depth sections (Chapter 7, Field 8) are behind a "read more" on mobile, you have a mobile-specific content problem that looks like a ranking mystery.

3. Fix the interaction budget on mobile first. Chapter 19's 296ms INP is measured across all devices, and mobile hardware is where that number is worst. The device that converts best is also the device most punished by a slow interface.

4. Do not over-optimise for the minority surface. Desktop is 74% of impressions and 65% of clicks. It is the worst-performing surface and the one that matters most in absolute terms. The correct response to this chapter is to segment your reporting, not to redesign around tablets.

The general method, since the habit is the point

The chapter is a worked example of a habit, so here is the habit as a procedure you can run on any number in your own reporting.

1. List the populations hiding inside the metric. For search data there are five you can split in one click, and each of them has produced something in this book: query type (brand vs non-brand), position band, device, country, and page. If a number matters, it should never stay blended.

2. Split, then compare the extremes rather than the average of the parts. The finding is almost always at the ends. Desktop against tablet, not the mean of the three. Position 1–3 against 8–10, not the overall 8.5.

3. Ask whether the split is a difference in the population or in the measurement. This distinction is the whole of the interpretation, and getting it backwards produces confident nonsense. Mobile converting better could mean mobile users are more decisive, a difference in people, or that mobile results pages have fewer slots, so position 8 means something different, a difference in the ruler. This chapter cannot separate them, and says so.

4. Check the sample before you believe the split. This is the step people skip when the finding is flattering.

The sample-size floor

The tablet row above is the best-performing surface on the site and it is 1% of impressions. That is why it gets flagged rather than acted on, and it is worth making the rule explicit because segmentation multiplies this problem: every split you add divides your data again, and the fourth split is frequently down to noise.

Three checks before you act on a segment:

A segment that fails all three is where the most confident wrong conclusions in analytics come from. It is small, so it moves a lot; it moves a lot, so it produces a dramatic number; the dramatic number is memorable, so it becomes a belief.

The transferable point

This chapter exists mostly as a worked example of a habit rather than a finding you can copy, your device split will look nothing like mine, because it is a document tool and people edit documents at desks.

The habit is the thing: whenever a number in this book is a single figure, go and look at what is inside it. Every time I have done that in this dataset, whether brand, position, device or country (Chapter 21), the inside was more interesting than the outside.

About this book

Averages Describe Nobody is chapter 20 of How to Grow Your SaaS to 100K Users Without Spending on Ads, a playbook on taking one product from zero to 100K+ users on search alone, with nothing spent on advertising. Every claim in it is checked against the real data, including the three findings that contradicted the author.

Continue reading

Free tools that implement this book