toolhq.io

All posts
8 min readby Jameel Haider

How Let's Encrypt actually issues a certificate

Automated certificate issuance feels like a black box until it fails, at which point the error message assumes you know the protocol. ACME (RFC 8555) is not complicated once you see the shape of it, and most renewal failures map cleanly onto one of its steps.

The account

Your client generates an account key pair on first run and registers the public key with the CA. Every later request is signed with that key, which is how the CA knows one client from another.

This key is not your certificate key and is not used in any certificate. Losing it means losing the rate limit history and any authorizations associated with the account, but not the ability to get certificates, since you can simply register again. It is worth backing up, and it is not a disaster if you do not.

The order

To request a certificate you create an order listing the identifiers you want covered, meaning the hostnames. The CA responds with a set of authorizations, one per identifier, each containing several possible challenges.

You prove control of each identifier by completing one challenge from each authorization. Authorizations are cached for a period, so a name you validated recently can be reused without repeating the challenge. This is why a renewal sometimes completes instantly and sometimes performs validation again.

The three challenge types

HTTP-01 is the default. The CA gives you a token. You serve a file at:

http://example.com/.well-known/acme-challenge/<token>

containing the key authorization, which is the token joined to a thumbprint of your account key. The CA fetches it over plain HTTP on port 80.

The details that trip people up:

  • It starts on port 80, not 443. Blocking port 80 entirely breaks it.
  • Redirects to HTTPS are followed, and the certificate on the target is not validated during this fetch. Redirecting to HTTPS is fine even with an expired certificate.
  • It cannot issue wildcards.
  • The name must resolve publicly. Internal names and split horizon DNS fail here.

DNS-01 proves control by publishing a TXT record at:

_acme-challenge.example.com

containing a hash of the key authorization. This is the only challenge that can issue a wildcard certificate, and the only one that works for a host with no public HTTP service. It requires an API on your DNS provider for automation, and it is sensitive to slow propagation, since the CA queries the authoritative servers shortly after you publish.

For a wildcard on *.example.com, the record still goes at _acme-challenge.example.com. Requesting both the wildcard and the apex creates two authorizations that use the same record name, so the provider must support multiple TXT values at one name.

TLS-ALPN-01 completes the proof inside a TLS handshake on port 443 using a dedicated ALPN protocol. It suits environments where port 80 is unavailable and something already terminates TLS. It cannot issue wildcards, and it requires the terminating server to support the mechanism directly.

The CAA check

Before validating, the CA checks your CAA records to confirm it is permitted to issue for the domain. This happens at issuance time, every time, not once at setup.

A CAA record that lists a different CA blocks issuance even though nothing about your web server changed. It is a common cause of a renewal that suddenly fails on a domain that worked for a year, usually after someone added CAA for a different provider. CAA records explained covers the syntax and the tree walk, and the DNS Lookup tool will show what is published.

Multi-perspective validation

Let's Encrypt validates from several network locations and requires agreement. This defends against an attacker who can influence routing or DNS from one vantage point.

The practical consequence is that a configuration which answers correctly from your location can still fail. Geo-restricted firewalls, region specific DNS answers and anycast setups that behave inconsistently all produce failures that look impossible from where you are sitting. If validation fails and everything appears correct locally, checking how the name resolves from elsewhere with the DNS propagation checker is the fastest way to see it.

Finalize and download

Once every authorization is valid, you send a CSR containing the public key for the certificate itself. This key is generated by your client and is separate from the account key. The CA signs it and gives you a URL to download the certificate and its issuing chain.

The CSR carries the names, so they must match the order. To inspect a CSR before submitting it, the SSL decoder shows the subject and SAN entries, and the CSR generator builds one if you are doing it by hand. What is inside a CSR covers the structure.

Rate limits

The limits are per registered domain and are the usual cause of a client that worked yesterday and fails today:

  • 50 certificates per registered domain per week. Registered domain means example.com, so all subdomains share one budget.
  • 5 duplicate certificates per week. A duplicate is the same exact set of names. Reissuing repeatedly while debugging exhausts this quickly.
  • Failed validation limits apply per account and hostname per hour, so a broken configuration retried in a loop locks itself out.
  • 300 pending orders per account.

Two habits avoid most of this. Use the staging environment while debugging, since it has far higher limits and issues untrusted certificates that are perfect for testing. And do not retry on a tight loop after a failure, because the retry itself becomes the reason issuance keeps failing.

Renewal

Clients typically renew at 30 days remaining on a 90 day certificate. The window exists so that a month of failed attempts still leaves time to notice.

The failure mode worth guarding against is silent: renewal breaks, nobody reads the cron output, and the certificate expires. External expiry monitoring is the check that catches it, since it does not depend on the same machinery that failed. The SSL checker reads the live chain and expiry for a host, the bulk SSL checker does it across a list, and monitoring SSL certificate expiry covers what to alert on.

Mapping errors to steps

SymptomWhere it went wrong
Timeout fetching the challenge fileHTTP-01, port 80 blocked or name resolves elsewhere
404 on the challenge URLHTTP-01, web server not serving .well-known
TXT record not foundDNS-01, propagation or wrong record name
CAA forbids issuanceCAA check, record lists another CA
Works locally, fails at CAMulti-perspective validation
Too many certificates already issuedRate limit on the registered domain

To confirm the validation records themselves are visible before retrying, the DCV checker looks up the TXT and CNAME records used for domain control validation, which turns a guess into a check. Domain control validation explained covers how the same proofs work at other CAs.