Definition
Robots.txt is a plain text file at the root of a host telling crawlers which URLs they may request. It is defined by RFC 9309, published in September 2022 and written largely by Google engineers. The thing to hold onto: it controls crawling, not indexing.
Example. A rule of Disallow: /search asks crawlers not to fetch your internal search pages. It does not remove a page that is already in the index.
You can test any URL against your own rules with the free Robots.txt Tester.
What does robots.txt actually control?
Robots.txt controls which URLs a crawler is allowed to request, and nothing else. Google's documentation puts it plainly: robots.txt is not a mechanism for keeping a web page out of Google, and a page disallowed there can still be indexed if other sites link to it.
That produces the result people find baffling: a URL in the results with no description, because Google knows the address from links but was never allowed to read the page. To keep a page out of the index, Google's stated remedy is a noindex rule or a password.
Why does noindex fail on a disallowed page?
Noindex fails on a disallowed page because the crawler has to fetch the page to see the tag, and robots.txt just told it not to. The two directives cancel each other out. Allow the crawl, let the noindex be read, and only then consider blocking the path.
What are the rules on placement and size?
The rules are strict about location. The file sits in the top level directory of the host, and it applies only to the host, protocol and port it is served from, so https://example.com/robots.txt says nothing about https://shop.example.com.
| Rule | What the source says |
|---|---|
| Location | Top level directory of the site, on a supported protocol (Google) |
| Scope | The host, protocol and port where the file is hosted (Google) |
| Size | Google enforces a 500 kibibyte limit. RFC 9309 sets 500 KiB as a minimum parsing limit, which is the opposite of a cap |
| Compliance | Voluntary. Well behaved crawlers obey it; nothing forces a bad one to |
How do you check your own robots.txt?
Check it by requesting the file at the exact host and protocol you care about and confirming it returns plain text, then testing the URLs you actually worry about against the rules. Google retired the robots.txt tester that used to live in Search Console, so a standalone tester is the practical replacement.
Sources
- RFC 9309: Robots Exclusion Protocol (September 2022)
- Google Search Central: Introduction to robots.txt
- Google Search Central: How Google interprets robots.txt
Last checked against these sources on 22 September 2026.