Server logs: the only record of what crawlers actually did
You want to know what search engines really do on your site rather than what a dashboard summarises days later.
Short answer
Search Console reports what happened in the results page. Server logs record every request a crawler actually made: which URLs, how often, what status came back, and how much of the crawl went to things that cannot rank. It is the only source that answers what was crawled, as opposed to what was indexed.
What logs contain that nothing else does
Every request, with timestamp, URL, status code, size, referrer and user agent. That means you can see a page being crawled, see it return a 500 that day, and see the crawler stop coming back. No aggregated report reconstructs that sequence.
The first three numbers to pull
Requests from verified search crawlers per day. Distinct URLs crawled. Share of requests that went to redirects, errors and assets. Those three tell you whether crawl is a constraint at all and where it is going.
Absence is the most useful finding
Sort your important pages against the crawled list. Any page you care about that never appears has not been found, which is a linking problem and not a content one. This is the fastest way to diagnose the pages that never get indexed and never explain why.
Verify crawlers, do not trust them
A user agent is a claim. Anyone can send "Googlebot" and plenty of scrapers do to bypass rate limits. Check the IP against the published ranges, or do a reverse DNS lookup for the definitive answer, before counting a request as crawl budget.
Status codes over time
A page returning 200 today and 404 last week explains an indexing gap that Search Console will report as a mystery. Logs are the only place you can see the sequence, which makes them the diagnostic of choice after any incident.
Which log to use
A CDN log if you have one, because an application server behind a CDN may never see the crawler at all. This is the single most common reason a log analysis comes back showing almost no crawler traffic.
Why almost nobody offers this online
Logs contain visitor IP addresses, which is personal data in most jurisdictions. Uploading them to a third-party website is a genuine risk, which is why the established tools in this category are desktop software and why a browser-based analyser is the only shape that should be acceptable.
Questions
- How much log data do I need?
- A week is enough for a first read of crawl distribution. A month is better for spotting which pages are crawled rarely, and anything shorter than a few days will mislead on frequency.
- What format do I need?
- Combined Log Format is the nginx and Apache default and is what most tooling reads. JSON and custom formats need converting first, and a parser that silently drops unmatched lines will report a clean-looking result from half your data.
- Is this worth it for a small site?
- Less so for crawl budget, which is rarely binding below a few thousand URLs. Still worth it once, to see which of your pages the crawler never visits, because that list is usually surprising.
Measured, not asserted
That privacy constraint is why the log analyser on this site parses everything in your browser and transmits nothing. The underlying discipline came from ihatepdf.cv, which reached 100,542 users and 42,320 clicks: when Bing went silent for 31 consecutive days, the request-level record was the only way to tell an outage from a penalty.
Free tool for this: Log File Analyzer. No account, nothing uploaded.
Where this goes deeper
Every number on this page comes from one complete dataset: one product taken from zero to 100K+ users on search alone, with nothing spent on advertising. The full argument is Chapter 17 of the book. Five chapters are free to read.