How to fix robots.txt errors
Four lines of text that can remove a whole website from search.
Check this page now
Free, no signup. We fetch the page live and read what a search engine would see.
robots.txt is the first file a crawler requests and the easiest place on a site to do catastrophic damage. A single Disallow: / left over from staging is the most expensive four characters in SEO.
This is how to read the file you have, what each rule really does, and how to fix the three problems that account for almost every case.
What usually causes it
Disallow: / shipped from staging
Development environments block crawlers, and the file goes live with the site. Nothing looks broken; traffic simply stops arriving, usually noticed weeks later.
Blocking CSS and JavaScript
Old hardening advice that is actively harmful now. Google renders a page before judging it, so blocked assets mean it sees a broken layout and cannot confirm the page works on mobile.
Using robots.txt to keep pages out of the index
It does not do that. It stops crawling, not indexing — a blocked URL with links pointing at it can still appear in results, with no description. To remove a page from the index, allow crawling and use noindex.
A 500 error on the file itself
Worse than a missing file. Crawlers treat a server error on robots.txt as "try later" and may back off from the entire site.
How to fix it
Read your live file
The robots.txt checker above fetches it, parses the groups, and says in plain words what each rule does — including whether anything blocks the whole site or the assets Google needs.
Remove any rule you cannot explain
If nobody knows why a Disallow line is there, it is almost always copied from a tutorial. The default for most sites is: block nothing except admin and checkout paths.
Unblock assets
Delete rules covering /wp-includes/, plugin folders, /assets/, /static/, .css and .js. Rendering needs them, and there is no security benefit — the files are public either way.
Declare your sitemap
Add a Sitemap: line with the full URL. It costs nothing and it is how crawlers that never see Search Console find your sitemap.
Use noindex for the pages you want hidden
Allow crawling, add a robots meta noindex, and let the page be read and dropped. Blocking it in robots.txt guarantees it stays in whatever state it is in now.
What the audit checks
- Rules that block the whole site or a major section
- Blocked CSS, JavaScript and asset folders
- Whether a sitemap is declared and resolves
- Syntax that no crawler understands, and rules in the wrong group
Check part of it right now, free
These read a live page and tell you what a search engine sees. No signup, no email.
robots.txt checker
One file, read carefully — because the expensive mistakes all live in it.
XML sitemap checker
Find your sitemap, prove it parses, and see what it is actually telling Google.
The issues behind this
Each has a full entry in the issue library: what it is, what it costs a business, and step-by-step fixes for WordPress, Shopify, Webflow and custom builds.
robots.txt blocking resources
The site blocks the files needed to render its own pages.
Noindex on an important page
A page tells search engines to hide it from results.
Missing XML sitemap
No sitemap to tell search engines what exists.
Questions
Related
XML sitemap errors
A sitemap is a claim about your site. This is how to make the claim true.
Noindex pages
The fastest way to lose a page is to tell Google to forget it.
Crawl the site and see which of your real pages the rules are blocking
The free audit crawls the site, groups every finding, and returns the five things worth fixing this month in plain English. No card, no call.
Run a free audit