Which AI crawlers can read your site? robots.txt for GPTBot, ClaudeBot and the rest

A growing share of what reads your website is not a person and not a search engine. It is a crawler collecting text to train a model, an indexer building an AI search product, or a fetcher grabbing a page because someone asked an assistant about it. Whether you want any of these is your call, but you can only make it if you know what your robots.txt currently says.

Three kinds of AI crawler

The first thing to understand is that a single company often runs several crawlers with different jobs, and each has its own robots.txt token.

• Training crawlers collect content to train or improve models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl), Bytespider (ByteDance) and Meta-ExternalAgent (Meta) are in this group.
• Search crawlers build an index that an AI search product answers from. OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic) and PerplexityBot (Perplexity) are here. Blocking them can mean your pages are not cited in those products.
• User triggered fetchers retrieve a page because a person asked an assistant to read it. ChatGPT-User, Claude-User and Perplexity-User are the examples. Some operators document that rules in robots.txt may not apply to fetches a user explicitly requested, so check each operator's own documentation.

Two more tokens are not crawlers at all. Google-Extended and Applebot-Extended are control tokens: you disallow them to tell Google and Apple not to use your content for their AI models, and doing so does not affect how your site appears in Google Search or Apple's search.

That split is why "block AI" is not one decision. Many site owners block training and allow search, because being cited is useful while being used as training data may not be.

How robots.txt decides

The rules come from the Robots Exclusion Protocol, standardised as RFC 9309. They are simple, but a few details catch people out.

A crawler obeys one group. A group starts with one or more User-agent lines followed by rules. A crawler uses the group that names it and ignores the User-agent: * group completely. It does not combine them. So if you write rules under * and then add a group for GPTBot, GPTBot follows only its own group and none of the rules you wrote for everyone.

The longest matching rule wins. Given Disallow: /docs and Allow: /docs/public, a request for /docs/public/page is allowed because the Allow rule is longer. When two rules match with equal length, Allow wins.

Patterns support * and $. Disallow: /*.pdf$ blocks URLs ending in .pdf. A trailing $ anchors the end, so /file.pdf?download=1 is not matched.

An empty Disallow allows everything. Disallow: with no value is a valid way to say nothing is blocked.

Here is a real shape worth knowing. GitHub's file, as read on 4 November 2026, gives five AI crawlers a single shared group, then blocks Bytespider outright, then falls back to a wildcard group for everyone else:

User-agent: GPTBot
User-agent: OAI-SearchBot
User-agent: ClaudeBot
User-agent: anthropic-ai
User-agent: PerplexityBot
Crawl-delay: 1
Disallow: /gist/
Disallow: /*/raw/
...

User-agent: Bytespider
Disallow: /

User-agent: *
Disallow: /account-login
...

Several user agents can share one group, and a Crawl-delay line in the middle does not end it. The five named bots follow that group, Bytespider is blocked from everything, and every other crawler follows the wildcard group. The AI Crawler Access Checker shows exactly this split: for each crawler, whether it has its own rule or follows the * rule, and which paths are restricted.

What a missing or broken file means

The status code of robots.txt matters as much as its content.

• A 404 means there is no file, and crawlers treat everything as allowed.
• A server error (5xx) means the rules are unknown. Well behaved crawlers pause rather than assume permission, so a flaky robots.txt can quietly stop crawling.
• A redirect is followed, but crawlers are only expected to follow a handful of hops.

Common mistakes

Blocking everything by accident. A leftover Disallow: / under User-agent: * from a staging environment blocks every well behaved crawler, search engines included. This is the most expensive mistake on the list.

Writing rules for * and expecting them to apply to named bots. As above, a named group replaces the wildcard group. Repeat the rules you still want inside each named group.

Blocking the wrong token. GPTBot blocks OpenAI's training crawler, not its search crawler, and not the fetcher that acts on a user's request. If you want to opt out of one, you need to name that one.

Guessing at a token. Tokens are exact strings published by each operator. A misspelled name matches nothing, and the crawler falls through to your wildcard rules.

Examples

To opt out of training but stay available to AI search and user requests:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

To allow one crawler into a section only:

User-agent: OAI-SearchBot
Allow: /docs/
Disallow: /

The checker reads the live file from your site. It does not test text you paste, so the quickest loop is to deploy the change, run the check, and confirm each crawler shows the status you intended.

robots.txt is a request, not a lock

It is worth being straight about this. robots.txt works because reputable crawlers choose to honour it. It does nothing to stop one that does not. If you need to enforce a block, deny the crawler at your firewall or CDN using the address ranges its operator publishes, and treat robots.txt as the polite layer on top.

What about llms.txt?

llms.txt is a proposed file, a plain text or Markdown summary of your site meant to help language models. The checker reports whether you have one, but be realistic about it: no major AI provider has said that it reads the file, so it does not control crawling and it should not be treated as a substitute for robots.txt. Having one is harmless, and it may be useful if a tool you use does read it.

A short routine

Run the checker on your domain. Decide, for training crawlers, search crawlers and user fetchers separately, which you want. Compare that to what the report shows, fix the file, and check again. The related tools for the rest of your site's front door are the security headers scanner and the redirect tracer, and for the bigger picture see how toolhq helps you work with AI.