Definition
AI crawlers are bots run by AI companies that fetch web pages for one of three jobs: collecting training data, powering AI search results, or fetching a page because a user asked an assistant to. Each company runs separate bots for separate jobs, so you can allow one and block another in robots.txt.
Which AI crawlers should you know?
The AI crawlers to know are the ones each vendor documents, grouped by job. This table uses each company's own documentation.
| Name in robots.txt | Company | Job | What the company says about robots.txt |
|---|---|---|---|
GPTBot | OpenAI | Training data | Controlled through robots.txt |
OAI-SearchBot | OpenAI | ChatGPT search results | Controlled through robots.txt |
ChatGPT-User | OpenAI | Fetches a page a user asked for | Robots.txt rules may not apply |
ClaudeBot | Anthropic | Training data | Anthropic says its bots honour robots.txt |
Claude-SearchBot | Anthropic | Search result quality | Anthropic says its bots honour robots.txt |
Claude-User | Anthropic | Fetches a page a user asked for | Anthropic says its bots honour robots.txt |
PerplexityBot | Perplexity | Perplexity search results, not model training | Controlled through robots.txt |
Perplexity-User | Perplexity | Fetches a page a user asked for | Generally ignores robots.txt |
Google-Extended | A control token, not a separate bot: covers Gemini training use | Does not affect inclusion in Google Search |
Checked against the vendor pages listed below on the date at the foot of this page. Vendors add and rename bots, so recheck before you rely on a name.
Why do AI crawlers matter for SEO?
AI crawlers matter because an AI engine cannot cite a page its bot was not allowed to fetch. The split between jobs is what makes this manageable. Blocking GPTBot opts you out of OpenAI's training collection; ChatGPT search results come from OAI-SearchBot, which is a separate rule. Likewise, Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search.
How do you control AI crawlers?
You control AI crawlers with a separate robots.txt group for each bot you want to treat differently. This example keeps a site in AI search while opting out of training:
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /Hawk Academy makes the opposite choice. Our own robots.txt lets every crawler in, and a Content Signals line states that search, AI answers and AI training are all welcome, because being read by AI is the point of a free education site.
Sources
- OpenAI: Overview of OpenAI crawlers
- Anthropic: Does Anthropic crawl data from the web?
- Perplexity: Perplexity crawlers
- Google Search Central: Google's common crawlers
- Google Search Central: AI features and your website
Last checked against these sources on 17 September 2026.