toolhq.io

All posts
7 min readby Jameel Haider

Planning a DNS cutover: TTL strategy that avoids a split brain

The migration plan says "change the A record at 02:00, done by 02:05". At 09:00 the next morning a fraction of users are still hitting the old server, orders are landing in two databases, and nobody can say when it will stop.

That is a TTL problem, and it is entirely avoidable with one change made several days earlier.

TTL is a cache lifetime, not a propagation delay

A DNS record carries a time to live in seconds. When a resolver answers a query, it caches the answer for that long and serves it from cache without asking again. There is no mechanism to tell a resolver that a cached record is now wrong. There is no push, no invalidation, no purge.

So the phrase "DNS propagation" is misleading. Nothing propagates. Your authoritative servers change instantly, and the rest of the internet catches up as each resolver's cached copy expires, at whatever moment it happened to fetch it.

The practical consequence: after a change, the old value can be served for up to one full TTL, and the window is different for every resolver. With a 24 hour TTL, the last stragglers arrive a day later. With a 300 second TTL, five minutes.

The timeline

The fix is to lower the TTL before the change, and to do it far enough ahead that the old, long TTL has itself expired everywhere.

WhenAction
T minus 7 daysAudit current TTLs on every record you will touch, and on the parent NS records
T minus 3 daysLower TTL on those records to 300 seconds
T minus 2 daysConfirm resolvers are now returning the low TTL, not the old one
T minus 1 dayBring the new server up and test it directly by IP or hosts file
T zeroChange the record
T plus 15 minVerify from multiple networks; keep the old server running
T plus 48 hoursDecommission the old server
T plus 48 hoursRaise the TTL back to 3600 or higher

The gap between lowering the TTL and the cutover must exceed the previous TTL. If the record was at 86400, a resolver that cached it an hour before your change will hold the 86400 value for almost a full day and will not see the 300 until then. Lowering the TTL three days before a cutover on a record that was at 24 hours is the minimum; a week is comfortable.

Raising the TTL afterwards matters too. A permanent 300 second TTL means every resolver in the world queries your authoritative servers 288 times a day per record, which costs you resilience: if your DNS provider has an outage, a long TTL keeps you resolving from caches while a short one takes you offline in minutes.

Verify the low TTL actually took

Checking your own provider's control panel proves nothing. Query a resolver and look at the TTL it reports:

dig +noall +answer example.com A @1.1.1.1
example.com.  287  IN  A  203.0.113.10

The 287 is the remaining lifetime of that resolver's cached copy, counting down. Query it twice a few seconds apart and watch the number drop. If it starts near 300, the new TTL is live. If it starts near 86400, the old one is still cached and you are not ready.

Query more than one resolver, since each holds its own independent cache. The DNS propagation checker queries a spread of public resolvers at once and shows which are still serving the old answer, which is exactly the "is it safe to decommission" question you need answered at T plus 48 hours. For a single detailed look at a record including its TTL, the DNS lookup tool shows the answer as a resolver returns it.

Caches that ignore your TTL

Lowering the TTL does not bind everyone.

Resolvers clamp. Many resolvers enforce a minimum cache time regardless of what you publish, commonly 30 to 60 seconds, and some enforce a maximum too. A TTL of 0 does not mean "never cache" in practice.

Browsers cache separately. Chrome and Firefox maintain their own DNS cache with their own timers, typically about a minute, on top of the operating system cache. A user with the tab open may hold an old answer past your TTL.

Operating systems cache. Windows has a DNS client cache, macOS has mDNSResponder, and many Linux setups run systemd-resolved or nscd. How to flush DNS cache covers clearing these when you need a clean test.

The JVM is the notorious one. With a security manager installed, networkaddress.cache.ttl historically defaulted to caching successful lookups forever. A long running Java service can hold a resolved IP for its entire uptime. If any consumer of your service is a JVM application, plan on restarting it or confirm the property is set to something sane.

Connection pools do not re-resolve. An open TCP connection is bound to an IP address. Long lived pools, persistent HTTP connections and gRPC channels keep using the old server long after DNS changed, because they never look it up again. This is why the old server must keep serving after the cutover rather than being turned off at T zero.

Negative caching is a separate TTL

If a resolver asks for a name that does not exist, it caches that absence. The lifetime comes not from the record, which does not exist, but from the minimum field of the zone's SOA record.

example.com. 3600 IN SOA ns1.example.com. hostmaster.example.com. (
  2026090501 7200 3600 1209600 3600 )
                                 ^ negative cache TTL

The practical trap: someone tests new.example.com before the record is created, gets NXDOMAIN, and that answer is now cached for an hour across every resolver that was asked. The record then exists, and the name still fails for people who tried early.

Create the record first, then announce it. If a name has already been queried and cached as nonexistent, there is nothing to do but wait out the SOA minimum. Keeping that value modest, 300 to 900 seconds, limits the damage. DNS records explained covers the rest of the SOA fields.

Changing nameservers is a different job

Everything above assumes you are changing records inside a zone you already control. Changing which nameservers are authoritative is slower and less under your control, because the NS records live in the parent zone at the registry, and you cannot set their TTL. TLD delegations commonly use 48 hours.

So the sequence is different:

  1. Build the zone completely at the new provider first, with every record matching the old zone.
  2. Verify by querying the new nameservers directly: dig example.com A @ns1.newprovider.com.
  3. Only then change the delegation at the registrar.
  4. Keep the old nameservers serving the identical zone for at least a week.

During the overlap, resolvers will use whichever set they have cached, so both must give the same answers. If the zones differ, you get genuine split brain, where the answer a user receives depends on which nameserver their resolver happens to have cached. Anything DNSSEC signed needs the DS record handled deliberately as well, since a mismatched DS breaks resolution outright rather than degrading; DNSSEC explained covers that transition.

Things that catch people out

  • A CNAME's TTL and its target's TTL are independent. A 300 second CNAME pointing at a 3600 second A record still gives you an hour of stale addresses.
  • CDN and load balancer hostnames often carry deliberately short TTLs, around 60 seconds, because the provider re-points them frequently. Do not raise those.
  • Email is more forgiving than web traffic: a sending server that connects to the old MX will usually be retried, so an MX record change is lower risk than an A record change, though the same TTL logic applies.
  • Apex records cannot be CNAMEs, which constrains the options for moving a root domain. CNAME at the apex covers ALIAS and ANAME alternatives.

The rule to remember is that a TTL change takes one old TTL to become effective. Plan backwards from the cutover, not forwards from the decision.

Related reading: DNS propagation explained covers why the delay is uneven, DNS records explained is the reference for each type, and how to flush DNS cache covers clearing local caches during testing.