How many pages before crawl budget matters?

As a rule of thumb, tens of thousands of URLs. Below that a crawler can typically read your whole site faster than you can change it. Measure it from your logs rather than trusting the rule: pages crawled per day against total URLs is a two-minute calculation and it settles the question for your site specifically.

Does blocking pages in robots.txt save crawl budget?

It stops those URLs being fetched, so mechanically yes. It also stops them being indexed and stops any link equity passing through them, and it does not remove them from the index if they are already there. Use it for genuinely worthless URL patterns, not as a budget tactic.

Do redirects waste crawl budget?

Each hop is a separate fetch, so a three-hop chain costs three requests to deliver one page. On a small site that is invisible. On a large one it is one of the few crawl-efficiency fixes worth doing, and it is worth doing anyway because chains slow real users too.

Crawl budget, and whether yours is actually the problem

You read that crawl budget matters, and now you are wondering whether it is why your pages are not indexed.

Short answer

Crawl budget is the number of URLs a search engine will fetch from your site in a given period. It is a real constraint on sites with hundreds of thousands of URLs and almost never the binding constraint below a few thousand. If your pages are not indexed and you have under ten thousand URLs, the cause is far more likely to be that nothing links to them.

What the phrase actually covers

Two things get bundled under one name. Crawl rate limit is how fast a crawler is willing to fetch without hurting your server, and it adjusts to how quickly you respond. Crawl demand is how much the engine wants your pages at all, which follows from popularity and freshness. A small site is almost never rate limited; it is demand limited, which is a polite way of saying the engine does not consider most of your URLs worth returning to.

The size at which it starts to matter

Google has said plainly that crawl budget is not something most publishers need to think about, and the threshold it names is in the thousands of URLs, not the hundreds. If you have a few hundred pages, a crawler can read your entire site in a few minutes. Spending a week on crawl efficiency at that size is optimising a resource you are not close to exhausting.

How to check instead of guess

Take a server or CDN log covering at least a week and count requests from verified search crawlers, then divide by days. That gives you pages crawled per day. Compare it against your total URL count: if the engine fetches more URLs per day than you have, budget is not your problem and no amount of tuning will change your indexing.

Where budget genuinely leaks

Redirect chains, because every hop is a fetch that returns no content. Error pages, because a 404 costs the same request as a page. Faceted URLs and session parameters, because they multiply one page into thousands. Infinite calendars. Assets fetched repeatedly. None of these matter at small scale; all of them compound past a certain size.

The fix that is usually the real fix

Pages get crawled because something links to them. A page in your sitemap with no internal link pointing at it is a suggestion; a page linked from a page that is crawled daily is an instruction. If a URL you care about is missing from your log entirely, it has not been found, and adding it to a sitemap again will not change that.

Why the advice persists anyway

Crawl budget is satisfying to work on. It produces a tidy list of technical fixes, each one verifiable, none of which requires you to admit that nobody links to your site. That makes it a comfortable substitute for the harder work, and the comfort is exactly why it stays in circulation long after it stopped applying to the sites reading about it.

Questions

How many pages before crawl budget matters?: As a rule of thumb, tens of thousands of URLs. Below that a crawler can typically read your whole site faster than you can change it. Measure it from your logs rather than trusting the rule: pages crawled per day against total URLs is a two-minute calculation and it settles the question for your site specifically.
Does blocking pages in robots.txt save crawl budget?: It stops those URLs being fetched, so mechanically yes. It also stops them being indexed and stops any link equity passing through them, and it does not remove them from the index if they are already there. Use it for genuinely worthless URL patterns, not as a budget tactic.
Do redirects waste crawl budget?: Each hop is a separate fetch, so a three-hop chain costs three requests to deliver one page. On a small site that is invisible. On a large one it is one of the few crawl-efficiency fixes worth doing, and it is worth doing anyway because chains slow real users too.

Measured, not asserted

On ihatepdf.cv, the browser tool taken from zero to 100,542 users on search alone, the whole site was 160 pages. At that size Googlebot could read every URL many times over in a day. Not one indexing problem across 224 days turned out to be crawl budget: every page that failed to rank either had nothing linking to it or answered a query nobody was searching.

Free tool for this: Log File Analyzer. No account, nothing uploaded.

Where this goes deeper

Every number on this page comes from one complete dataset: one product taken from zero to 100K+ users on search alone, with nothing spent on advertising. The full argument is Chapter 10 and Chapter 17 of the book. Five chapters are free to read.