robots.txt template for AI search crawlers
· Famed
Most AI companies run separate crawlers for search and for training, so you can stay in AI search answers while opting out of training. Below are two ready files: one that lets everything in, and one that blocks training crawlers only.
Replace the Sitemap line with your own address. robots.txt is a public request, not a lock: well-behaved crawlers follow it, and visits that a user asks for may not.
1. Show up everywhere (recommended for local businesses)
# Show up in AI answers: let the search and answer bots read your site. User-agent: Googlebot Allow: / User-agent: Bingbot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: Applebot Allow: / User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml
This is right for most businesses that want customers to find them. Note that the last group already allows every other crawler.
2. Stay in search, opt out of training
# Stay in AI search answers, but opt out of AI model training. User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: Applebot Allow: / # Training only. Blocking these doesn't remove you from search answers. User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: * Allow: / Sitemap: https://www.example.com/sitemap.xml
This keeps the search and user-request crawlers and blocks the ones their makers describe as collecting training data.
Google-Extended is left out on purpose. Google says it controls training and also grounding in the Gemini app, so blocking it can keep your content out of Gemini app answers. It doesn't affect Google Search or AI Overviews.
Which crawler does what
| Crawler | Company | Used for |
|---|---|---|
| OAI-SearchBot | OpenAI | ChatGPT search |
| ChatGPT-User | OpenAI | Pages a ChatGPT user asks for |
| GPTBot | OpenAI | Model training |
| PerplexityBot | Perplexity | Perplexity search results, not training |
| Perplexity-User | Perplexity | Pages a Perplexity user asks for |
| Claude-SearchBot | Anthropic | Claude search results |
| Claude-User | Anthropic | Pages a Claude user asks for |
| ClaudeBot | Anthropic | Model training |
| Applebot | Apple | Siri, Spotlight and Safari search |
| Applebot-Extended | Apple | Opt-out from Apple model training |
| Google-Extended | Gemini training and Gemini app grounding, not Search |
Check the firewall too
A crawler allowed in robots.txt can still be blocked by Cloudflare or another firewall's bot protection. OpenAI and Perplexity both publish IP ranges and recommend allowing them. Famed's free check tests this for the main AI crawlers.
See whether AI names your business
The free check reads your site, your Google listing and the directories AI cites, and asks AI about your business. About 20 seconds, no card.
Run the free AI checkSources
- OpenAI: overview of OpenAI crawlers
- Perplexity: Perplexity crawlers
- Anthropic: how site owners can manage Anthropic's crawlers
- Apple Support: about Applebot
- Google: common crawlers (Google-Extended)
- Google Search Central: AI features and your website
AI search changes often. We check these sources and update the date above when something changes.