Definition
Training data is the text a model learns from before it answers anything. For large language models that includes a great deal of the public web, which is why each AI company documents a crawler you can block to signal that you do not want your content used.
Example. A model that describes your product without citing a source is drawing on training data. A model that links your page fetched it at the time of the question.
You can see which training and search crawlers your robots.txt currently allows with the free AI Crawler Access Checker.
How do you keep your content out of training data?
You opt out by disallowing each company's training crawler in robots.txt, and the tokens are documented by the companies themselves.
| Token | Company | What blocking it signals |
|---|---|---|
GPTBot | OpenAI | Your content should not be used to train their foundation models |
ClaudeBot | Anthropic | Future materials should be excluded from training datasets |
Google-Extended | Content may not be used for training Gemini models. Does not affect inclusion in Google Search | |
CCBot | Common Crawl | Exclusion from an open crawl archive that many datasets are built from |
Does blocking a crawler remove you from a model?
Blocking a crawler does not remove you from a model. It applies to future training, not to a model that has already learned from your pages. This is the single most common misunderstanding about the whole subject, and no opt-out on the list above is retroactive. Anthropic also notes that blocking by IP address may not work reliably as an opt-out.
What do you lose by opting out?
What you lose depends entirely on which crawler you block, because training bots and search bots are different. Block the training crawler and you stay in AI search results. Block everything and you remove yourself from answers people are asking right now, which is usually the opposite of the intent. Our AI crawlers page has the full list by job.
Mentioned in
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Google Search Central: Google's common crawlers
- Common Crawl: CCBot
Last checked against these sources on 23 September 2026.