SEO Small

Publish a valid robots.txt

robots.txt is a plain-text file at the root of a site that tells crawlers which paths they may fetch, grouped by User-agent, and usually where the sitemap lives. This check asks whether the site serves a real one, and lists what it blocks: the paths blocked for search crawlers (the Googlebot group if there is one, otherwise User-agent: *) and any crawler shut out of the whole site. It fails when the file is missing, is not a robots.txt at all, or blocks the whole site for search crawlers. Whether this particular page is blocked is reported separately by the Indexable check.

Find out whether your site has this problem, and every other check, in one free scan.

Check your site free

Why it matters

robots.txt is the first file search engines and AI crawlers ask for. Without a real one they get no crawl rules and no pointer to your sitemap, and a site that answers with an HTML page instead looks misconfigured to them.

How to fix

1 Create a text file named robots.txt at the root of the site, served as text/plain with a 200 status.
2 Start with a User-agent: * group and a Disallow: line for each path crawlers should skip (an empty Disallow: allows everything).
3 Add a Sitemap: line with the full URL of your sitemap.
4 If it blocks the whole site (Disallow: / for User-agent: * or Googlebot), remove that line unless the site really should stay out of search.
5 Make sure your framework or host does not answer /robots.txt with the app page instead of the file.

Common mistakes

Letting a single-page app answer /robots.txt with its HTML shell and a 200 status, so crawlers receive a web page instead of rules.
Writing Disallow: / under User-agent: * during development and shipping it, which blocks the whole site for every crawler.
Using robots.txt to hide private pages. It is public and only a request: anyone can read it, and it does not stop a page being indexed from links elsewhere.

How to verify

1 Open /robots.txt on the site in a browser and confirm it shows plain text, not your site design.
2 Check that each Disallow line is a path you really want crawlers to skip, and that the Sitemap line points at a URL that loads.
3 Use the robots.txt report in Google Search Console to confirm Google fetched and parsed it without errors.

Ready to see where you stand?

Scan your site and get your Engineering Score with a prioritized roadmap in under a minute, no signup required.

Scan your site