Free tool
robots.txt generator for AI crawlers
Quick answer
AI companies run separate crawlers for separate jobs: some collect training data, some build the index their assistant searches, and some fetch a page because a person asked. Blocking the training crawlers keeps your content out of model training while your site still appears in AI answers; blocking the search and user crawlers removes it from those answers. Pick a preset or choose crawler by crawler, then copy the file.
# AI crawlers kept out of the whole site User-agent: GPTBot User-agent: ClaudeBot User-agent: Google-Extended User-agent: Applebot-Extended User-agent: CCBot User-agent: Meta-ExternalAgent User-agent: Bytespider Disallow: / User-agent: * Allow: /
The three kinds of AI crawler
- Training: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Meta-ExternalAgent, Bytespider.
- Search: OAI-SearchBot, Claude-SearchBot, PerplexityBot. They build the index an assistant answers from.
- User requests: ChatGPT-User, Claude-User, Perplexity-User. They fetch a page because someone asked the assistant to.
Check the result
Publish the file at /robots.txt, then run the agent readiness checker on this site, which tests the same crawlers against it.
Frequently asked questions
How do I block GPTBot in robots.txt?
Add a group for it: "User-agent: GPTBot" followed by "Disallow: /". That stops OpenAI's training crawler; OAI-SearchBot and ChatGPT-User are separate user agents with their own rules.
Should I block AI crawlers?
Block the ones you do not want and keep the rest. Training crawlers take content for model training. Search and user crawlers are how assistants such as ChatGPT, Claude and Perplexity find and quote your site, and blocking them takes you out of those answers.
Does robots.txt actually stop AI crawlers?
It stops the ones that follow it, which the major operators say theirs do. It is a request, not a lock: a crawler that ignores it is not stopped, and that needs blocking at your server or CDN.