The robots.txt mistakes that quietly cost you indexing
You want to know whether your robots.txt is helping or silently blocking the pages you care about.
Short answer
robots.txt controls crawling, not indexing. Blocking a URL stops crawlers fetching it, but a blocked URL can still appear in results with no description, and blocking a page that already ranks will not remove it. To keep a page out of the index you need a noindex tag, which requires the page to be crawlable in order to be read.
The distinction that causes most of the damage
Crawling is fetching the page. Indexing is storing it and being willing to show it. robots.txt only touches the first. This means the most common instinct, blocking a page you want removed, produces the opposite of what you wanted: the crawler can no longer fetch the page, so it can no longer see the noindex tag telling it to drop the page, so the page stays.
Mistake one: blocking your own assets
Disallowing /assets/ or /static/ made sense when crawlers ignored CSS and JavaScript. Now the renderer needs them to see your page the way a user does, and blocking them means the rendered version can differ from what you serve. If your page depends on JavaScript at all, blocking the bundle is blocking the content.
Mistake two: a staging rule that shipped
Disallow: / in a staging robots.txt that gets deployed to production is the single most expensive line in SEO. It is also invisible, because the site works perfectly for humans. Check production directly rather than trusting the repository, because the file that matters is the one being served.
Mistake three: blocking a page to hide it
A blocked URL that has links pointing at it can still be listed, showing the URL with no title or description. If the goal is to keep something out of search, the answer is a noindex tag on a crawlable page, or authentication if it is genuinely private. robots.txt is a request about crawling, not a privacy control.
Mistake four: assuming it is a security boundary
robots.txt is public and lists exactly what you would rather people did not look at. Anyone can read yours in a browser. If a path must not be reached, it needs authentication; listing it in robots.txt advertises it while doing nothing to protect it.
Mistake five: no sitemap line
A Sitemap: directive in robots.txt is the one place every crawler looks without being told. It costs one line and it removes an entire class of discovery problem. Point it at an absolute URL, and make sure that URL actually returns the sitemap rather than a redirect to a login.
How to check in under a minute
Load yours in a browser, then take three URLs you care about most and confirm none of them match a Disallow. Then confirm the pages you want out of the index are crawlable and carry noindex, which is the combination that actually works.
Questions
- Will robots.txt remove a page from Google?
- No, and it can prevent removal. A page already indexed stays indexed, now without a description because the crawler can no longer read it. To remove a page, allow crawling and serve a noindex tag, or use the removal tool in Search Console for something urgent.
- Should I block my search results pages?
- Usually yes, because they generate near-infinite URL combinations with no unique value. Prefer noindex on those pages if you have a manageable number, so link equity still flows; use robots.txt when the URL space is genuinely unbounded.
- Does robots.txt affect rankings directly?
- No. It affects whether a page can be fetched, which affects whether it can rank at all. There is no ranking bonus for having one and no penalty for a permissive one.
Measured, not asserted
ihatepdf.cv, the tool taken from zero to 100,542 users with $0 spent on advertising, served 160 pages and blocked almost nothing in robots.txt. That was deliberate: at 160 pages there is no crawl pressure to relieve, and every Disallow is a chance to hide something you meant to rank.
Free tool for this: Crawler View. No account, nothing uploaded.
Where this goes deeper
Every number on this page comes from one complete dataset: one product taken from zero to 100K+ users on search alone, with nothing spent on advertising. The full argument is Chapter 10 of the book. Five chapters are free to read.