robots.txt decides who may crawl. The noindex meta decides who may index. Disallow a URL that currently serves noindex and the crawler may never return to see that noindex: the page can sit in the index indefinitely from stale data.

The four cases worth knowing

  • Cheap pages that should be crawled but not indexed: allow crawling, serve <meta name="robots" content="noindex">. This is the search-page case.
  • Never fetch at all (drafts behind auth, exports): robots.txt Disallow. Works when nothing else links to them.
  • Already-indexed URLs that must go: allow crawling so the noindex gets read; de-indexing then takes days, quietly.
  • Private or stolen content that needs to vanish now: that is a removal request handled at the search engine, not a robots rule.

The footguns

  1. A Disallow that matches nothing: /search/ rules against URLs that live at /en/search/. Test the rule against a REAL URL, once.
  2. Blocked pages still named in the sitemap: the sitemap says go fetch, the Disallow says do not; you are filing contradictory evidence.
  3. A meta tag the CDN or prerenderer stripped: verify the OUTPUT URL, not your source file.

One habit catches all three: in Search Console, run URL Inspection on the real production URL and read both halves: what the fetch got and what the index holds.