One file, read carefully — because the expensive mistakes all live in it.
Free · no signup · reviewed
Free, no signup. We fetch the page live and read what a search engine would see.
robots.txt is four lines of text that can remove an entire website from search. A single Disallow: / left over from a staging environment is one of the most common catastrophic SEO mistakes, and it is invisible from inside the CMS.
This fetches the file, parses the groups, and tells you in plain words what each rule does — including the subtle one almost everybody gets wrong: blocking the CSS and JavaScript Google needs to render the page before it can judge it.
It is also the first file a crawler requests, before anything else on your site. That makes it the highest-leverage four lines you will edit all year, in both directions.
Nothing else on a site can do this much damage this quietly. There is no error, no warning and no visible change — traffic simply stops arriving, and by the time someone connects it to a deploy from three weeks ago the recovery takes longer than the mistake did.
A short file: allow everything you want crawled, disallow admin, cart and checkout paths, never block CSS or JavaScript, and declare your sitemap. If you cannot explain a line, it probably shouldn't be there.
Each of these has a full entry in our issue library: what it is, what it costs a business, and step-by-step fixes for WordPress, Shopify, Webflow and custom builds.
The site blocks the files needed to render its own pages.
A page tells search engines to hide it from results.
A checker that hands you a list and walks away has done half the job. Here is what each kind of finding means and what it is worth doing about it.
Disallow: / under User-agent: * tells every search engine to stay off the entire site. If this shows up on a live site, fix it now and request indexing for your key pages in Search Console afterwards — recovery is measured in weeks, not hours.
Google renders pages before judging them. A rule covering your stylesheet, your JavaScript bundle or an asset folder means it sees a broken page and cannot confirm your mobile layout works. There is no security benefit — the files are public either way.
A missing file means everything may be crawled, which is what most sites want. A 500 error on that URL is a different matter: crawlers read it as "come back later" and can back off from the whole site.
They stop crawling. A blocked URL with links pointing at it can still appear in results with no description, and because the page cannot be fetched, any noindex you add is never seen. Allow crawling and use noindex instead.
Disallowing /wp-includes/ and plugin folders was advice a decade ago and is actively harmful now. If nobody at your company knows why a Disallow line exists, that is the strongest argument for deleting it.
robots.txt is public, readable by anyone, and lists exactly the paths you would rather people didn't visit. Private content needs authentication, not a polite request to well-behaved bots.
The two cancel out. The crawler cannot fetch the page, so it never reads the directive, and the URL lingers in results indefinitely.
Crawl-delay is ignored by Google. Scrapers ignore the file entirely. It is a set of instructions for crawlers that already agreed to follow them.
The setting is in a different place on every platform, and that is usually what stands between knowing and fixing.
WordPress serves a virtual robots.txt unless a real file exists in the web root. Check Settings → Reading first: "Discourage search engines from indexing this site" changes it and adds a site-wide noindex, and it is the single most common cause of a new site never appearing.
Editable through robots.txt.liquid on the theme. Useful and easy to overdo — a rule that looks narrow in Liquid can match far more than you intended, so re-check the live file after every change.
Project settings → SEO holds the file. Most sites need nothing beyond a sitemap line; the setting people actually need there is disabling indexing on the webflow.io subdomain.
You cannot edit it. Crawl control happens per page in each page's SEO settings, which is enough for most sites but means the auto-generated tag and category pages need attention instead.
app/robots.ts generates it. Keep the disallow list to real private paths, and if preview deployments are public, give them their own rules or a noindex header rather than relying on obscurity.
More on your platform: SEO audits by platform.
Find your sitemap, prove it parses, and see what it is actually telling Google.
Score one page against the checks that actually move rankings.
The tag that decides which URL gets the credit — checked properly.
Or see all ten free tools.