Should you block AI crawlers from your website?
Quick answer
Most businesses that want to be recommended by AI assistants should allow the crawlers that power AI search and citations on their marketing pages. Blocking training-only crawlers is a separate, reasonable choice that usually does not affect visibility in AI answers. Block AI access only for content that is itself the product, and make the decision per crawler and per path in robots.txt.
Three kinds of AI crawler
"AI crawler" covers bots with very different purposes. The distinction matters because blocking one kind has different consequences from blocking another.
- Training crawlers collect public pages to train future models. Blocking them keeps your content out of future training data.
- Search and retrieval crawlers index pages so an assistant can find and cite them when answering. Blocking them reduces your chance of being cited.
- User-initiated fetchers load a page when a person asks an assistant to read a specific URL. They behave more like a browser than a crawler.
Which user agents do what
The main user agents as documented by each vendor at the time of writing. Names and behaviour change, so check each vendor's documentation before relying on this list.
| User agent | Operator | Purpose |
|---|---|---|
| GPTBot | OpenAI | Model training |
| OAI-SearchBot | OpenAI | ChatGPT search results and citations |
| ChatGPT-User | OpenAI | Fetches pages a user asks about |
| ClaudeBot | Anthropic | Model training |
| Claude-SearchBot | Anthropic | Search results and citations |
| Claude-User | Anthropic | Fetches pages a user asks about |
| PerplexityBot | Perplexity | Search index for answers |
| Perplexity-User | Perplexity | Fetches pages a user asks about |
| Google-Extended | Control token for Gemini training and grounding — not a separate crawler, and does not affect Google Search | |
| Applebot-Extended | Apple | Control token for Apple AI training; Applebot still powers Siri and Spotlight |
| CCBot | Common Crawl | Open dataset widely used for training |
| Bingbot | Microsoft | Bing Search, which also grounds Copilot |
What happens if you block them
If an assistant's search crawler cannot read your pages, it has less first-hand information about your business. It may still mention you from other sources — directories, reviews, news — but you lose influence over how you are described and whether you are linked.
Two important exceptions:
- Google AI Overviews and AI Mode use Google's regular Search index. Blocking Google-Extended does not remove you from them; only blocking Googlebot would, and that removes you from Search entirely.
- Copilot relies on Bing. Blocking Bingbot affects both Bing Search and Copilot answers.
When blocking makes sense
- Paid courses, research, reports or datasets that are part of your product
- Member-only or gated resources
- Proprietary pricing or data you do not publish elsewhere
- Content under licensing restrictions
- A deliberate policy decision not to contribute to model training
None of these apply to your service pages, About page or blog articles written to attract customers.
Recommended robots.txt setups
Option A — maximum AI visibility. Allow everything public; block only private areas.
User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xmlOption B — visible in AI search, opted out of training. The common middle ground.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: *
Allow: /
Disallow: /admin/Option C — protect premium content only. Block AI crawlers from specific paths while leaving marketing pages open.
User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: CCBot
Disallow: /members/
Disallow: /reports/
User-agent: *
Allow: /A crawler follows the most specific group that names it, so a bot listed in its own group ignores the * group — repeat any rules it should still obey.
What robots.txt cannot do
- It is voluntary. Reputable operators respect it; scrapers may not. Truly private content needs authentication, not robots.txt.
- It does not remove existing data from models already trained.
- User-agent strings can be spoofed. Major vendors publish IP ranges so you can verify real requests in server logs or firewall rules.
- It is not a ranking tool. Allowing crawlers does not guarantee citation; it only makes it possible.
How to decide in five minutes
- List the sections of your site: marketing pages, blog, resources, members area.
- For each, ask: is this content meant to attract customers, or is it the product?
- Attract customers → allow search and retrieval crawlers.
- It is the product → block AI crawlers on that path, or put it behind a login.
- Decide separately whether to opt out of training crawlers.
- Check your server logs or CDN analytics after a month to confirm bots behave as expected.
Want a second opinion? Our AI search optimization audit includes a robots.txt and crawler access review, and our SEO service covers the indexing foundations assistants depend on. For background, read GEO vs SEO vs AEO.
To see how your current robots.txt treats 13 search and AI user agents, paste it into the AI Search Readiness Checker. Crawler policies are shown for information only and do not affect the readiness score.
Frequently asked questions
Does blocking GPTBot affect Google rankings?
No. Google Search uses Googlebot. GPTBot is OpenAI's training crawler and has no effect on Google.
Does blocking Google-Extended remove me from AI Overviews?
No. AI Overviews use Google's regular Search index. Google-Extended controls use of content for Gemini training and grounding.
Where do I change these settings?
In the robots.txt file at the root of your domain. Many CDNs, including Cloudflare, also offer AI bot controls.
Should I add an llms.txt file?
It is a proposed convention, not a confirmed standard among major assistants. It does no harm, but it is not a priority.
How often should I review this?
Every few months. Vendors add and rename user agents regularly.