Skip to content

robots.txt template for AI search crawlers

· Famed

Most AI companies run separate crawlers for search and for training, so you can stay in AI search answers while opting out of training. Below are two ready files: one that lets everything in, and one that blocks training crawlers only.

Replace the Sitemap line with your own address. robots.txt is a public request, not a lock: well-behaved crawlers follow it, and visits that a user asks for may not.

2. Stay in search, opt out of training

# Stay in AI search answers, but opt out of AI model training.
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: Applebot
Allow: /

# Training only. Blocking these doesn't remove you from search answers.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

This keeps the search and user-request crawlers and blocks the ones their makers describe as collecting training data.

Google-Extended is left out on purpose. Google says it controls training and also grounding in the Gemini app, so blocking it can keep your content out of Gemini app answers. It doesn't affect Google Search or AI Overviews.

Which crawler does what

CrawlerCompanyUsed for
OAI-SearchBotOpenAIChatGPT search
ChatGPT-UserOpenAIPages a ChatGPT user asks for
GPTBotOpenAIModel training
PerplexityBotPerplexityPerplexity search results, not training
Perplexity-UserPerplexityPages a Perplexity user asks for
Claude-SearchBotAnthropicClaude search results
Claude-UserAnthropicPages a Claude user asks for
ClaudeBotAnthropicModel training
ApplebotAppleSiri, Spotlight and Safari search
Applebot-ExtendedAppleOpt-out from Apple model training
Google-ExtendedGoogleGemini training and Gemini app grounding, not Search

Check the firewall too

A crawler allowed in robots.txt can still be blocked by Cloudflare or another firewall's bot protection. OpenAI and Perplexity both publish IP ranges and recommend allowing them. Famed's free check tests this for the main AI crawlers.

See whether AI names your business

The free check reads your site, your Google listing and the directories AI cites, and asks AI about your business. About 20 seconds, no card.

Run the free AI check

Sources

AI search changes often. We check these sources and update the date above when something changes.

Related guides