Measuring crawl budget from your own logs
You want to stop guessing about crawl budget and get an actual number from your own server.
Short answer
Take a week of logs, filter to verified search crawlers, and compute four numbers: requests per day, distinct URLs crawled, the share going to redirects, errors and assets, and which of your important pages never appear. Those four settle whether crawl is your constraint and, if it is, where it is going.
Step one: get the right log
A CDN or edge log if you have one. An application server behind a CDN often never sees the crawler, and analysing it produces a confident conclusion drawn from a fraction of the traffic. If your numbers look implausibly low, this is usually why.
Step two: filter to real crawlers
Match on user agent to identify candidates, then verify against published IP ranges before counting anything. Unverified requests carrying a crawler user agent are not crawl budget, and including them inflates every number that follows.
Step three: requests per day against URL count
Divide verified crawler requests by days covered, then compare against your total URL count. If the engine fetches more URLs per day than you have, crawl budget is not your constraint and no optimisation will change your indexing.
Step four: the waste ratio
Classify each request: page, redirect, error, asset. Assets must be fetched and are not automatically waste. Redirects and errors are, because each is a request that could have been a page. A high ratio here is the one crawl finding that reliably justifies work.
Step five: the absence list
Take the pages you care about and check which never appear in the log. That list is more actionable than everything above it, because a page that is never crawled has not been found, and the fix is internal links rather than crawl tuning.
What the frequency distribution tells you
Pages crawled daily are considered important. Pages crawled monthly are tolerated. The gap between how you rank a page internally and how often it is crawled is a direct measure of whether your internal linking communicates your priorities.
When to stop
If requests per day comfortably exceed your URL count and the waste ratio is modest, you are done. Crawl is not your problem, and the next hour is better spent on the pages that get crawled and still do not rank.
Questions
- What is a good crawl waste ratio?
- There is no published benchmark worth citing. Compare against yourself over time, and treat anything above roughly a third going to redirects and errors as worth an afternoon.
- How often should I do this?
- Once when you first want to rule crawl in or out, and again after a migration, when redirect chains and error rates change dramatically in a short window.
- Do I need special software?
- No. A log file and a tool that parses Combined Log Format is enough, and doing it in the browser avoids sending visitor IP addresses to a third party.
Measured, not asserted
On ihatepdf.cv, 160 pages meant crawl could never be the binding constraint, and it never was: across 224 days and 100,542 users, every indexing problem traced to a page nothing linked to or a query nobody searched. The log was how that got ruled out rather than assumed.
Free tool for this: Log File Analyzer. No account, nothing uploaded.
Where this goes deeper
Every number on this page comes from one complete dataset: one product taken from zero to 100K+ users on search alone, with nothing spent on advertising. The full argument is Chapter 17 of the book. Five chapters are free to read.