AI Crawler Access Checker
See whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and other AI crawlers are allowed by a site’s robots.txt.
About the AI Crawler Access Checker
AI Crawler Access Checker reads a site's robots.txt file and tells you what each well known AI crawler is permitted to do. It covers the crawlers behind OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, ByteDance, and Common Crawl, and separates those that collect training data from those that fetch pages to answer a user's question.
The evaluation follows the Robots Exclusion Protocol standard. A crawler obeys the group that names it and ignores the generic group completely, the longest matching rule wins, and a tie goes to allow. So a site that disallows a few private paths for everyone shows as partially restricted, while a site with a dedicated block for one bot shows that bot as blocked and the others as unaffected. The report says whether each result comes from the crawler's own rule or from the generic one.
Keep in mind what robots.txt is: a request that well behaved crawlers honour, not an enforcement mechanism. It also checks for an llms.txt file, with the caveat that no major AI provider has confirmed it reads one. For the rest of a site's crawl and security setup, see the Security Headers Scanner and Redirect & HSTS Tracer.
Reach the developers and engineers who use these tools every day.
Frequently asked questions
Does blocking GPTBot stop ChatGPT from seeing my site?
Not entirely. OpenAI uses separate crawlers: GPTBot for training data, OAI-SearchBot for search results, and ChatGPT-User for pages a user asks it to open. Each has its own robots.txt token, which is why the report lists them separately.
What are Google-Extended and Applebot-Extended?
They are control tokens, not separate crawlers. Disallowing them tells Google and Apple not to use your content to train their AI models, without affecting how your site appears in Google Search or Apple search.
Is robots.txt enough to keep AI companies out?
It is a polite request that reputable crawlers follow, and some crawlers ignore it. If you need to enforce a block, use your firewall or CDN rules to deny the crawler by its published address ranges as well.
What does it mean when robots.txt is unreachable?
If the file returns a server error, the rules are unknown rather than allow all, and well behaved crawlers pause until it recovers. A missing file (404) is different: it means no restrictions, so everything is allowed.
Do AI companies read llms.txt?
No major provider has said that it does. The file is a proposal, so its presence is reported for information only and should not be treated as a way to control crawling.