How to tell real Googlebot from something pretending
Your logs are full of crawler traffic and you want to know how much of it is actually the search engine.
Short answer
A user agent string is a claim, not an identity: anyone can send "Googlebot" and many scrapers do. The definitive check is a reverse DNS lookup on the requesting IP, confirming it resolves to a Google domain and that the hostname resolves back to the same IP. The practical alternative is matching against the published IP ranges.
Why anyone bothers to fake it
Sites treat crawlers generously: higher rate limits, no bot challenge, sometimes full content where a human would hit a paywall. Impersonating a crawler is the cheapest way past all three, so scrapers do it as a matter of course.
The definitive method
Reverse DNS on the IP, which should resolve to a googlebot.com or google.com hostname, then a forward lookup on that hostname to confirm it returns the original IP. Both directions matter, because reverse DNS alone can be configured by whoever controls the address block.
The practical method
Google, Bing and several others publish their crawler IP ranges as machine-readable lists. Matching against those is weaker than reverse DNS, because ranges change and lists lag, and it is enormously stronger than believing the string. Report unmatched requests as unverified rather than as fake.
What the fakes cost you
Bandwidth and server load, first. Then analysis: any crawl-budget calculation that counted them is measuring strangers. And sometimes content, because a scraper with crawler privileges is being handed the thing you would have rate-limited.
Do not block on user agent
Blocking anything claiming to be Googlebot from an unverified IP will eventually block the real one, because ranges change. Verify first, then throttle. Blocking the real crawler is a far more expensive mistake than serving a scraper.
The other bots in your logs
Link databases, AI retrieval crawlers, uptime monitors, security scanners, feed readers. Many are legitimate and none of them are search crawl. Separate search engines from everything else before drawing any conclusion about crawl budget.
What to do with the finding
If a meaningful share of your apparent crawl traffic is unverified, your crawl analysis was wrong and your server has been carrying load for nothing. Rate-limit the unverified, keep serving the verified, and recount.
Questions
- Can I do reverse DNS from a browser?
- No, browsers cannot perform DNS lookups directly, which is why browser-based tools fall back to published IP ranges and should say so rather than implying a stronger check.
- Should I block fake crawlers?
- Rate-limit rather than block, and never on user agent alone. The cost of accidentally blocking the real crawler is much higher than the cost of serving a scraper a few pages.
- Does Google penalise sites that block scrapers?
- No. Blocking abusive traffic is normal operations. The risk is misidentification, not the blocking itself.
Measured, not asserted
This is why the log analyser on this site verifies by IP range rather than by user agent. The habit came from ihatepdf.cv, taken to 100,542 users, where the 31-day Bing blackout made one thing obvious: if you cannot tell which requests are really the search engine, you cannot tell an outage from a change in your own traffic.
Free tool for this: Log File Analyzer. No account, nothing uploaded.
Where this goes deeper
Every number on this page comes from one complete dataset: one product taken from zero to 100K+ users on search alone, with nothing spent on advertising. The full argument is Chapter 17 and Chapter 22 of the book. Five chapters are free to read.