How to test an agent skill before you trust it
A skill is instructions, so it fails quietly. There is no error, no stack trace and no obvious moment where something went wrong. The usual outcome of a first attempt is a file that never loads, and you find out weeks later.
There are four distinct failure modes and they need different tests.
Failure 1: it never fires
By far the most common, and the only one with a simple test.
Open a fresh session. Type the request the way someone would actually type it, in their words, without naming the skill or hinting at it. Then check whether the skill loaded.
The cert on staging is throwing an untrusted error
Not this, which proves nothing:
Use the tls-triage skill to check staging
If it does not fire, the problem is the description and nothing else. The body is irrelevant, because it was never read. The description is the only part of a skill that is always in context, and it is what the request gets matched against.
The fix is vocabulary. Compare what you wrote against what you typed, and add the missing words:
# never fires
description: TLS certificate chain analysis and validation.
# fires
description: Diagnose TLS and certificate faults on a live host,
including expiry, incomplete chains and untrusted roots. Use when a
site shows a certificate warning, a client reports an untrusted or
expired certificate, or curl fails where a browser works.
The second contains "untrusted", "expired" and "curl fails where a browser works", which are things people say. The first contains "validation", which nobody types when something is broken.
Test with at least five different phrasings, including a vague one and one that describes only the symptom. A skill that fires on three out of five is a skill that will miss the real request.
Failure 2: it fires when it should not
Less common, more annoying. An over broad description pulls a skill into unrelated work, which wastes context and bends the answer.
The test is a set of nearby requests that should not trigger it. For a TLS skill: a question about DNS, a question about a code change, a question about HTTP status codes. If the TLS skill loads for those, the description is reaching too far.
The usual cause is a description written to cover everything the author might one day want, rather than one job. Two tight skills beat one loose one.
Failure 3: it fires, reads, and ignores the procedure
This is the one worth most of your attention, because the output looks fine.
Take a case where you already know the answer. A host with a deliberately incomplete certificate chain is ideal, because the correct diagnosis is specific and the tempting wrong answer is obvious.
Then check three things, in this order:
Did it follow the order? If the skill says check expiry, then chain, then hostname, did it do that, or did it jump to the interesting hypothesis? Skipping ahead is the normal failure, and it is what numbered steps exist to prevent.
Did it avoid the trap? If you wrote down a trap, the test is whether the trap was avoided. The browser caching note in a TLS skill is only earning its keep if the assistant stops treating "works in my browser" as evidence.
Was the output the shape you asked for? If the skill specifies cause, then evidence, then fix, and you got three paragraphs of prose, the format instruction is not landing. Usually it is buried at the end of a long file. Move it, or shorten what comes before.
Confirm the real answer independently. For a chain fault, the SSL chain fixer will tell you exactly what the host sends, so you can see whether the assistant read it correctly or filled in a plausible story.
Failure 4: it invents the data
The most serious failure, and the one that produces the most convincing wrong answers.
A skill that instructs an assistant to "look up the SPF record" needs something that can look up an SPF record. If that capability is absent, the instruction does not fail. The assistant produces an SPF record. It will be well formed, plausible, and not yours.
The test is deliberate: run the skill with the tool removed.
Disconnect the MCP server, or run the skill against a domain you know has no such record, and read what comes back. You want a refusal. What you often get is fluent invention.
If it invents, the fix goes in the skill as an explicit instruction, because the default behaviour is to be helpful:
Never state a record's contents from memory or infer it from the
domain's other records. If a lookup tool is unavailable, say so and
stop. Do not describe what the record probably contains.
This matters more for network skills than most, because skills running through the Claude API execute in a sandbox with no network access at all. An assistant in that environment cannot resolve anything, so a skill that assumes it can will hallucinate every answer. The capability has to come from an MCP server, and the skill should say which one and what to do when it is missing. When to use a skill and when you need an MCP server covers the split.
Keep a few known cases
Once a skill works, the thing that breaks it is editing it. Keep three or four cases with known answers written down next to the skill:
tls-triage/
SKILL.md
cases.md
# Known cases
| Host | Expected finding |
| --- | --- |
| expired.example.test | Leaf expired 2024-03-01 |
| incomplete.example.test | Missing intermediate; browser succeeds, curl fails |
| wrong-name.example.test | Hostname not in SAN list |
| self-signed.example.test | Complete and unexpired, root not trusted |
Run them after any edit. It takes two minutes and catches the common regression, which is that tightening the description to stop false triggers also stops the real ones.
This is also the file that makes a skill maintainable by someone other than you.
Test the tools separately
A skill is one half of a pair. If the answers are wrong, establish which half before editing either.
Call the tool directly, outside the skill, and check the raw response. If the tool returns the wrong thing, the skill is innocent. If the tool is right and the conclusion is wrong, the skill is at fault.
Testing an MCP server before agents use it covers that side, and the MCP inspector will show you what a server actually exposes and how it responds.
A short checklist
Before you rely on a skill:
- It fires on five real phrasings, with no hints.
- It does not fire on three nearby requests that belong elsewhere.
- On a known case, it follows the order, avoids the trap, and returns the format.
- With the tool removed, it refuses rather than invents.
- Three or four known cases are written down beside it.
Most skills in the wild fail the first and the fourth. Those two are worth the effort even if you skip the rest.