Save as .claude/skills/ai-visibility/SKILL.md in a repository, or under ~/.claude/skills/ai-visibility/SKILL.md to have it everywhere. The copy button takes the whole file, frontmatter included.

---
name: ai-visibility
description: Audit whether AI assistants and AI search crawlers can read a site, and whether its content is usable once fetched. Use when asked about GPTBot, ClaudeBot, PerplexityBot, AI crawlers, llms.txt, blocking AI training, appearing in AI search results, or why an assistant cannot read a page.
---

# AI visibility audit

Fetch the real files before concluding anything. Never describe a site's
robots.txt from memory or from how similar sites are configured.

## Requires

The toolhq MCP server, for `http_headers` and `dns_lookup`:

    claude mcp add --transport http toolhq https://toolhq.io/api/mcp

## 1. Separate the two questions

People conflate these and the answers differ:

- **Can a crawler fetch the page.** Governed by robots.txt, response status,
  authentication and blocking at the edge.
- **Can the fetched page be understood.** Governed by whether the content is in
  the HTML or requires JavaScript.

Establish which the user is actually asking about. A site can be fully allowed
and still unreadable, and that is the more common fault.

## 2. Fetch robots.txt

Get `https://<domain>/robots.txt` and read the real contents.

Record, per user agent token, whether it is allowed or disallowed. Matching is
on the token and is case insensitive, and a `User-agent: *` block applies only
where no more specific block matches that crawler.

Check these tokens specifically:

| Token | Operator | What it governs |
| --- | --- | --- |
| `GPTBot` | OpenAI | Crawling for training |
| `OAI-SearchBot` | OpenAI | Indexing for ChatGPT search |
| `ChatGPT-User` | OpenAI | Fetches made when a user asks |
| `ClaudeBot` | Anthropic | Crawling |
| `PerplexityBot` | Perplexity | Indexing |
| `Google-Extended` | Google | Gemini and Vertex AI training |
| `Bingbot` | Microsoft | The Bing index, which feeds Copilot |
| `CCBot` | Common Crawl | A corpus many models train on |
| `Applebot-Extended` | Apple | Apple Intelligence training |

## 3. State these correctly

These are the points most often got wrong. Do not restate the common
misconceptions.

- **`Google-Extended` is not a crawler.** It is a robots.txt token controlling
  whether content already crawled by Google is used for Gemini and Vertex AI
  training. Disallowing it does not remove the site from Google Search and does
  not stop Googlebot.
- **Blocking Bingbot affects Microsoft Copilot**, because Copilot draws on the
  Bing index. People block Bingbot for crawl budget reasons and are surprised.
- **User initiated fetchers are separate from training crawlers.**
  `ChatGPT-User` is the fetch made when a person pastes a link. Blocking it
  means a user who explicitly asks about the page gets nothing, which is
  usually not the intent.
- **robots.txt is voluntary.** It is a request. Well behaved crawlers honour
  it. If the requirement is enforcement, the answer is blocking at the edge by
  user agent and address, not a robots.txt line.
- **`llms.txt` is a proposal, not a standard.** No major search engine has
  committed to reading it. It is harmless to publish and must not be presented
  as something that affects ranking or inclusion.

## 4. Is the content actually readable

Fetch the page and check whether the main text is in the returned HTML.

Most crawlers do not execute JavaScript. A page that renders its content client
side returns an empty shell, so the site is allowed, reachable and invisible.
This is the most common real cause of an assistant being unable to read a page.

Also check:

- The response status on the canonical URL, after redirects.
- Whether a redirect chain ends where it should. More than two hops is worth
  reporting.
- Whether the page requires cookies or a session to show content.

## Report back

1. Allowed or blocked, per crawler, from the real file.
2. Whether the content survives without JavaScript.
3. Anything that contradicts what the user believed, stated plainly.
4. Changes to make, separating "stops crawling" from "stops training use",
   because people routinely ask for one and mean the other.