Agent Skill for AI Crawler Checks

Whether AI assistants can read your site is now a routine question, and it is poorly served by general knowledge. The crawler names change, the controls are not what people assume, and a lot of the published advice is either out of date or invented. That combination is a strong case for writing the facts down in a skill.

What it depends on

Checking a live robots.txt and real response headers needs network access:

claude mcp add --transport http toolhq https://toolhq.io/api/mcp

That provides http_headers and dns_lookup.

The skill

Save as .claude/skills/ai-visibility/SKILL.md:

---
name: ai-visibility
description: Audit whether AI assistants and AI search crawlers can read
  a site, and whether its content is usable once fetched. Use when asked
  about GPTBot, ClaudeBot, PerplexityBot, AI crawlers, llms.txt, blocking
  AI training, appearing in AI search results, or why an assistant cannot
  read a page.
---

# AI visibility audit

Fetch the real files before concluding anything. Never describe a
site's robots.txt from memory or from how similar sites are configured.

## 1. Separate the two questions

People conflate these and the answers differ:

- **Can a crawler fetch the page.** Governed by robots.txt, response
  status, authentication and blocking at the edge.
- **Can the fetched page be understood.** Governed by whether content
  is in the HTML or requires JavaScript.

Establish which one the user is actually asking about. A site can be
fully allowed and still unreadable, and that is the more common fault.

## 2. Fetch robots.txt

Get `https://<domain>/robots.txt` and read the real contents.

Record, per user agent token, whether it is allowed or disallowed. Note
that matching is on the token and is case insensitive, and that a
`User-agent: *` block applies only where no more specific block matches
that crawler.

Check these tokens specifically:

| Token | Operator | What it governs |
| --- | --- | --- |
| `GPTBot` | OpenAI | Crawling for training |
| `OAI-SearchBot` | OpenAI | Indexing for ChatGPT search |
| `ChatGPT-User` | OpenAI | Fetches made when a user asks |
| `ClaudeBot` | Anthropic | Crawling |
| `PerplexityBot` | Perplexity | Indexing |
| `Google-Extended` | Google | Gemini and Vertex AI training |
| `Bingbot` | Microsoft | Bing index, which feeds Copilot |
| `CCBot` | Common Crawl | A corpus many models are trained on |
| `Applebot-Extended` | Apple | Apple Intelligence training |

## 3. Facts to state correctly

These are the points that are most often got wrong. Do not restate the
common misconceptions.

- **`Google-Extended` is not a crawler.** It is a token in robots.txt
  that controls whether content already crawled by Google is used for
  Gemini and Vertex AI training. Disallowing it does not remove the
  site from Google Search and does not stop Googlebot.
- **Blocking Bingbot affects Microsoft Copilot**, because Copilot draws
  on the Bing index. People block Bingbot for crawl budget reasons and
  are surprised by the result.
- **The user initiated fetchers are separate from the training
  crawlers.** `ChatGPT-User` is the fetch made when a person pastes a
  link. Blocking it means a user who explicitly asks about the page
  gets nothing, which is usually not the intent.
- **robots.txt is voluntary.** It is a request. Well behaved crawlers
  honour it. If the requirement is enforcement, the answer is blocking
  at the edge by user agent and address, not a robots.txt line.
- **`llms.txt` is a proposal, not a standard.** No major search engine
  has committed to reading it. It is harmless to publish and must not
  be presented as something that affects ranking or inclusion.

## 4. Is the content actually readable

Fetch the page and check whether the main text is in the returned HTML.

Most crawlers do not execute JavaScript. A page that renders its
content client side returns an empty shell to them, so the site is
allowed, reachable, and invisible. This is the most common real cause
of an assistant being unable to read a page.

Also check:
- Response status on the canonical URL, after redirects.
- Whether a redirect chain ends where it should. More than two hops is
  worth reporting.
- Whether the page requires cookies or a session to show content.

## 5. Report back

1. Allowed or blocked, per crawler, from the real file.
2. Whether the content survives without JavaScript.
3. Anything that contradicts what the user believed, stated plainly.
4. Changes to make, separating "stops crawling" from "stops training
   use", because people routinely ask for one and mean the other.

Why this skill is mostly corrections

Unusually, the most valuable part of this file is the list of things not to say.

AI crawler advice has accumulated fast and much of it is wrong. The Google-Extended misconception is the clearest case: it reads like a crawler name, so advice confidently describes blocking it as a way to keep a site out of Google's AI features, or warns that it will hurt search rankings. Neither is right. It is a usage control applied after crawling, and Googlebot is unaffected.

llms.txt is the second. It is a reasonable proposal and people write about it as though publishing one gets you cited. There is no commitment from any major engine to read it. A skill that says "harmless, not a ranking factor, do not oversell it" produces a more useful audit than one that treats it as a checklist item.

Writing corrections down works because it is exactly what general knowledge gets wrong. The model has read all the overconfident posts too.

The failure nobody checks for

Step 4 is the one that finds real problems.

Teams spend their time on robots.txt and almost none on whether the page has content in it. A site built as a client rendered application returns a near empty document to any crawler that does not run JavaScript, which is most of them. Every permission is correct and the content is still invisible.

The check is simple: fetch the URL without a browser and look for the text. If it is not there, no robots.txt change will help.

Verifying the findings

Check the assistant's reading against exact tools:

Per crawler allow and deny status: AI crawler checker
The raw response, status and headers: HTTP headers
Where a redirect chain actually ends: redirect tracer
What a shared link will show: OG meta generator

For the background on which crawlers exist and what each one does, which AI crawlers can read your site is the longer treatment.

Extending it

ai-visibility/
  SKILL.md
  crawlers.md    the full token list, kept current, with operators
  robots.md      precedence, wildcards, and how matching really works
  edge.md        your own blocking rules, if you enforce rather than ask

Keep crawlers.md separate deliberately. It is the part that goes stale, and having it in one file next to the skill makes updating it a small job rather than an edit to your main instructions.

AI Crawler Access Checker — See whether GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and other AI crawlers are allowed by a site’s robots.txt.