Agent Skill for Incident Triage

"The site is down" is the least useful sentence in operations, and it is how almost every incident starts. The work of the first five minutes is not fixing anything. It is narrowing the fault to a layer, so the next step is informed rather than hopeful.

That narrowing is a procedure, which makes it a good skill. An assistant given this job without one tends to do the opposite: pick the most interesting hypothesis and investigate it thoroughly, whether or not it is the layer at fault.

What it depends on

Triage is entirely live checks:

claude mcp add --transport http toolhq https://toolhq.io/api/mcp

That covers dns_lookup, dns_propagation, reverse_dns, ip_lookup, ip_reputation, ssl_check and http_headers, which is every check below.

The skill

Save as .claude/skills/incident-triage/SKILL.md:

---
name: incident-triage
description: Narrow a website or service outage to a layer, in order,
  before proposing a cause. Use when a site is down, unreachable,
  intermittently failing, slow for some users, showing a certificate
  error, or when someone asks why a domain stopped working.
---

# Incident triage

Your job in this skill is to locate the fault, not to fix it. Resist
proposing a cause until the layer is established.

Go in this order and stop at the first layer that fails. Each layer
depends on the one above, so a failure higher up makes everything
below it meaningless.

## Layer 1: does the name resolve

`dns_lookup` for A and AAAA on the exact hostname the user reported,
not the apex, and not www unless that is what they said.

- No records: DNS is the fault. Check whether the zone answers at all
  and whether the records were recently changed.
- Records present but pointing somewhere unexpected: likely a recent
  change. Run `dns_propagation` to see whether resolvers disagree.
- Resolvers disagree: a change is in flight. Report the TTL and the
  expected settle time rather than treating it as broken.

An AAAA record pointing at an address that does not serve the site is a
real and easily missed fault. It breaks the site for IPv6 clients only,
which presents as "down for some people".

## Layer 2: is the address plausible

`ip_lookup` on the resolved address, and `reverse_dns`.

You cannot ping from here, so do not claim the host is up or down.
What you can establish is whether the address belongs to the hosting
provider you expect. An address in an unexpected network after a
reported change is a strong signal.

`ip_reputation` is worth a look if the symptom is that some users are
blocked rather than all.

## Layer 3: does TLS complete

`ssl_check` on the host and port.

- Handshake fails entirely: the service is not listening, or not
  listening with TLS. This is the clearest "the host is the problem"
  signal available without shell access.
- Certificate expired: the fault, and the fix is a renewal.
- Chain incomplete: will break some clients and not browsers, so it
  explains "works for me" reports. See the TLS skill for the detail.
- Hostname mismatch: often means traffic is reaching the wrong
  service, such as a default virtual host or the wrong load balancer
  target.

## Layer 4: what does HTTP say

`http_headers` on the full URL.

- 5xx: the application is reached and failing. The network is fine.
  Stop blaming DNS.
- 404 on a URL that should exist: routing or a deploy, not
  infrastructure.
- 401 or 403: access control, possibly a WAF rule or a new edge
  policy.
- A redirect loop, or a chain that does not terminate: configuration,
  commonly an apex and www rule fighting a platform level redirect.
- 200: the origin is serving. If users still report failure, the fault
  is client side, regional, or in a cache.

## Rules

- **Never report a symptom as a cause.** "The certificate is expired"
  is a cause. "The site shows an error" is not.
- **Say what you could not check.** You have no shell, no logs and no
  internal metrics. Name the gap instead of filling it with a guess.
- **Distinguish all users from some users.** Some users points at DNS
  propagation, IPv6, a regional edge, or an incomplete chain. All users
  points at the origin, the certificate or the record.
- **A recent change is the most likely cause.** Ask what changed
  before theorising. If a record changed, check propagation before
  anything else.

## Report back

1. The layer that failed, and the layers you cleared.
2. The evidence, with the actual values returned.
3. The most likely cause, with your confidence.
4. The next check, naming who can run it if you cannot.

Why ordering matters more than depth here

The reason to number these layers is that each one makes the ones below it unreadable.

If a name does not resolve, every HTTP symptom is downstream noise. If TLS does not complete, the status code tells you nothing. An assistant investigating out of order will produce a thorough, confident analysis of the wrong layer, and because the writing is good it is persuasive.

The ordering also stops the most expensive mistake in incident response, which is changing DNS during an outage. DNS changes are slow to take effect and slow to undo, so making one while diagnosing turns a ten minute incident into an hour. Establishing that the fault is a 500 from the application, and therefore not DNS, is worth more than any individual check.

The two symptoms worth special handling

"It works for me." Almost always one of four things: DNS propagation mid change, an IPv6 record pointing somewhere wrong, an incomplete certificate chain where the reporter's browser has a cached intermediate, or a regional edge. All four are in the skill, and all four are invisible if you only check from one place.

"It is slow, not down." Nothing in this procedure diagnoses slowness well, and the skill should say so rather than improvise. Redirect chains are the one exception worth checking, because each hop is a round trip and a three hop chain on the apex is a real and fixable cost.

Verifying the findings

For the first few incidents, run the domain through an exact tool and compare:

Everything in one pass: domain X-ray
Resolver by resolver agreement: DNS propagation
Where redirects actually end: redirect tracer
Certificate and chain: SSL chain fixer
Raw response: HTTP headers

Auditing an unfamiliar domain in one query shows the same ground covered in a single pass, which is the faster path when you have no prior context at all.

Extending it

incident-triage/
  SKILL.md
  escalation.md   who owns DNS, TLS, the edge and the application
  known.md        faults you have had before and what they looked like
  runbooks.md     links to the real runbooks

known.md is the file that compounds. Every incident you resolve is a symptom and a cause worth recording, and after a dozen entries the skill starts recognising your own recurring faults rather than reasoning from first principles each time. That is the point at which it beats a generic checklist.

Domain X-ray — One query, one report: registration, nameservers, hosting, CDN, WAF, mail, certificate, DNSSEC, and every DNS record.