AI companies send different types of bots to your website, unless you block them. Some collect content to train AI models, some index pages for AI search, and some fetch a page when someone asks a question. Blocking the wrong AI crawler can remove your site from AI answers.
This is a pressing concern these days. Cloudflare, which is at the back of a large share of the web, blocks training and agent crawlers by default on pages that display ads for new domains, a change that took place in September 2026. This means a lot of sites may be blocking AI bots without the owners making that choice themselves.
This guide talks about the main types of AI crawlers, what each one does, how to decide which one to allow, and an example of robots.txt you can use.
What Are AI Crawlers?
AI crawlers are automated bots that AI tools use to read web pages. They generally fall into one of three categories.
Training crawlers: They collect content that may help train future AI models. Blocking them keeps your content out of future training data but usually doesn’t affect whether AI tools can cite you today.
Search crawlers: They index pages so an AI assistant can find and cite them in search-style answers. Blocking them can remove you from those answers.
User-triggered fetches: They visit a specific page when a person asks an AI tool to look something up or open a link. They act on behalf of a user.
OpenAI, Anthropic, and Perplexity all use separate bots for these jobs. This lets you make a more precise choice, for example, blocking the training data bot while allowing the search visibility one to crawl your site.

Which AI Crawlers Should You Know About?
Let’s talk about the specific AI crawlers from the major companies, OpenAI, Anthropic, Google, and Perplexity. Each company has its own bots.
- GPTBot: This one’s used for training by OpenAI to make its generative AI foundation models more useful and safe.
- OAI-SearchBot: This one is for search. It surfaces websites in ChatGPT’s search features.
- ChatGPT-User: It’s user-triggered. OpenAI says it isn’t used for automatic crawling, and robots.txt rules may not apply because a user starts the visit.
ClaudeBot: Training. Blocking it signals that your future content should be excluded from Anthropic’s training data.
Claude-SearchBot: Search. Disabling it may reduce your visibility in Claude’s search results.
Claude-User: User-triggered. It fetches pages when Claude users ask questions.
- ClaudeBot: training. Blocking it signals that your future content should be excluded from Anthropic’s training data.
- Claude-SearchBot: search. Anthropic says disabling it may reduce your visibility in Claude’s search results.
- Claude-User: user-triggered. It may fetch pages when Claude users ask questions.
- Googlebot: Its main search crawler. It also feeds AI Overviews and AI Mode.
- Google-Extended: A robots.txt control token, not a separate crawler. It decides whether Google can use your content to train future Gemini models and for grounding in Gemini Apps.
- PerplexityBot: Search. It surfaces and links websites in Perplexity’s results. The company uses it to train its foundation models.
- Perplexity-User: User-triggered. It generally ignores robots.txt because a user requested the fetch.
Does Blocking Google-Extended Remove You From AI Overviews?
No, Google-Extended controls Gemini training and grounding, not Google search, and AI Overviews and AI Mode are part of search.
Google says that Google-Extended does not impact a site’s inclusion in Google Search, nor is it used as a ranking signal.
No. Google-Extended controls Gemini training and grounding, not Google Search, and AI Overviews and AI Mode are part of Search. And SEO consultant Marie Haynes says that blocking it should not have an impact on your inclusion in AI Mode answers.
Should You Block AI Crawlers?

Not if you want your business or brand to be found in AI search. But what about the training bots? The sensible middle ground would be to allow search and user-triggered bots and make a separate choice for training bots.
Reasons You Might Block Training Crawlers:
- You publish original content, such as research, journalism, or paid material, and don’t want AI platforms to train their AI with your data.
- You have licensing or legal concerns about how your content gets reused
- You want to reduce server load from heavy crawling
Reasons to Allow AI Search Crawlers:
- Blocking OAI-SearchBot, Claude-SearchBot, or PerplexityBot can remove you from those tools’ search answers
- AI citations can bring visibility and referral traffic, especially for businesses that rely on being found
- Your competitors may stay visible while you disappear from the answers
Check what’s happening at your hosting or CDN level as well, not just in robots.txt. Cloudflare might treat multi-purpose crawlers by the strictest rule by default. If you use Cloudflare, review your AI bot settings so you don’t block search engines by accident.
How Do You Block AI Crawlers in robots.txt?
You block an AI crawler by adding its user agent to your robots.txt file with a Disallow rule. The file sits at the root of your domain, for example, yoursite.co.uk/robots.txt.
Example 1: Block AI training, stay visible in AI search
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Example 2: Block GPTBot from one folder only
User-agent: GPTBot
Disallow: /members/
Keep these points in mind:
- Robots.txt is a request, not a lock: It doesn’t technically stop a bot. Firewall or CDN rules give you better control.
- Add rules for each subdomain: Anthropic, for example, asks site owners to add the rule for every subdomain they want to opt out.
- Changes take time: Updates could take 24 hours to take effect.
- Never block Googlebot by mistake: Blocking it removes you from Google Search, including AI Overviews.
If you’re also thinking about how AI tools read your site, our guide to llms.txt files explains a related, but separate, idea.

How Can AI SEO Help You Manage AI Crawlers?
We help businesses with AI search optimization. If you’re unsure which bots to allow and which to block, we can assist based on your goals. As part of our AI SEO services. If you’d like a clear crawler policy for your website, get in touch with our team.




