Log File Analyzer

Reads your access log in the browser and reports what search crawlers really did: which URLs they fetched, how much of the crawl went to redirects, errors and assets, and which requests carried a crawler user agent from an IP that does not belong to that crawler.

How to use it

  1. Get the access log from your server or CDN. A CDN log is usually the one you want, because an app server behind a CDN may never see the crawler.
  2. Decompress it if it is gzipped, then drop the plain file here. Combined Log Format is the nginx and Apache default and is what this reads.
  3. Start with the waste percentage, then the path list. Pages missing from that list entirely have not been found.

What the numbers mean

Crawl waste
Requests that went to something that cannot rank: redirects, errors, and assets. Assets have to be fetched and are not automatically waste, but every redirect and every 404 is a request that could have been a page.
Unverified crawler
A request sending a crawler user agent from an IP outside that crawler’s published ranges. A user agent is a claim, not an identity. Some are researchers and proxies, some are scrapers dodging rate limits, and none of them are the search engine.
Most-crawled URLs
Frequently crawled means considered important. The useful reading is the absence: a page you care about that never appears has not been found, which is a linking problem rather than a content one.

Questions

Am I really uploading my server logs to a website?

No, and you should not do that anywhere. Access logs contain visitor IP addresses, which is personal data in most of the world. This file is read in your browser tab and never transmitted: load the page, disconnect from the internet, and it still works. That is the only arrangement under which anyone should ask you for a log file, and it is why the paid tools in this category are all desktop software.

How reliable is the crawler verification?

It compares the request IP against the ranges Google, Bing and others publish. The definitive check is a reverse DNS lookup, which a browser cannot perform, so this is weaker than rDNS and far stronger than believing the user agent string. Published ranges change, so an unverified result is reported as unverified rather than as fake.

My log format is different.

This reads Combined Log Format, the nginx and Apache default. JSON logs and custom formats will not match and it will tell you how many lines it could not parse rather than silently reporting zero.

Guides that use this tool

Other free tools