EngineeringScoreAllow the page to be indexed
Three separate things can keep a page out of search, and this check reads all three. A noindex directive (via the robots meta tag or the X-Robots-Tag response header) explicitly tells search engines to exclude the page from results, overriding everything else on the page; content="none" is defined as exactly equivalent to noindex, nofollow, and a directive addressed to a single crawler (name="googlebot", name="bingbot") removes the page from that engine. A Disallow rule in robots.txt works differently: it stops the page being fetched at all, so its content never enters the index. Where directives conflict, search engines apply the most restrictive one. Each is a deliberate tool for pages that should stay out of search (internal tools, thank-you pages), but each is also often left over accidentally from a staging configuration.
Why it matters
A noindex directive tells search engines to leave the page out of results entirely, and a robots.txt Disallow stops them fetching it in the first place. If either isn't intentional, the page and any traffic or leads it could generate are invisible to search.
How to fix
- 1 Check the robots meta tag and the X-Robots-Tag response header for noindex or none, which mean the same thing.
- 2 Check crawler-specific meta tags too, such as name="googlebot" or name="bingbot": a directive aimed at one crawler still removes the page from that engine.
- 3 Open robots.txt at the root of this site and look for a Disallow rule covering this URL, under either User-agent: * or a Googlebot group. The finding above names the exact rule and group we matched.
- 4 Remove it if the page is meant to be discoverable. Where directives conflict, search engines apply the most restrictive one, so a single leftover noindex or Disallow is enough to keep the page out.
Common mistakes
- Promoting a staging or pre-launch build to production without removing the noindex flag, or the blanket robots.txt Disallow, it had for testing.
- Setting noindex in the meta tag but not realizing an X-Robots-Tag header from the same response can independently block indexing (or vice versa).
- Adding a permissive <meta name="robots" content="index, follow"> to override a noindex elsewhere on the page, when the most restrictive directive wins regardless of which one comes first.
- Removing noindex from the generic robots tag while a crawler-specific tag, or the shorthand content="none", still carries it.
- Blocking the URL in robots.txt at the same time as removing noindex. The two are not interchangeable: a Disallow prevents the page being crawled, so a crawler never sees the permissive meta tag at all.
- Assuming a Disallow rule under User-agent: * does not apply to Googlebot. It does, unless robots.txt names a Googlebot group of its own, in which case that group replaces the wildcard one rather than adding to it.
How to verify
- 1 View page source and confirm no robots meta tag carries noindex or none, including crawler-specific ones such as <meta name="googlebot" content="noindex"> (unless intentional).
- 2 Inspect the response headers for an X-Robots-Tag header and confirm it is not set to noindex or none, including a crawler-scoped form such as "googlebot: noindex".
- 3 Open robots.txt at the root of the site and confirm no Disallow rule under User-agent: * or a Googlebot group covers this URL path.
- 4 Confirm robots.txt actually loads. If the request times out, errors, or returns your app shell instead of a text file, we report this check as not determined rather than guessing, and search engines will be reading something unintended too.
Related checks
Ready to see where you stand?
Scan your site and get your Engineering Score with a prioritized roadmap in under a minute, no signup required.