toolhq.io

All posts
7 min readby Jameel Haider

Ping and traceroute: reading what they actually tell you

Ping and traceroute are the first two commands anyone runs when something feels slow, and their output is misread more often than almost any other diagnostic. High latency at hop 7 usually means nothing. Asterisks in the middle of a traceroute usually mean nothing. A host that does not respond to ping is frequently perfectly healthy. Both tools are useful, but only if you know what part of the output carries information.

What ping measures

Ping sends an ICMP echo request and waits for an echo reply, reporting how long the round trip took.

$ ping -c 4 example.com
64 bytes from 93.184.216.34: icmp_seq=0 ttl=56 time=24.1 ms
64 bytes from 93.184.216.34: icmp_seq=1 ttl=56 time=23.8 ms
64 bytes from 93.184.216.34: icmp_seq=2 ttl=56 time=68.2 ms
64 bytes from 93.184.216.34: icmp_seq=3 ttl=56 time=23.9 ms

4 packets transmitted, 4 received, 0% packet loss
round-trip min/avg/max/stddev = 23.8/35.0/68.2/19.2 ms

Three numbers matter, and the average is the least of them.

Packet loss is the important one. Any sustained loss above zero on a wired path is a real problem, and even 1 to 2 percent degrades TCP noticeably, since every lost segment triggers retransmission and congestion control backs off. On HTTP/2 over TCP, a single lost packet stalls every multiplexed stream at once.

Jitter, the variation between samples, matters more than the mean for anything interactive. The run above has a good average and a bad outlier, which is worse for a video call than a uniformly slower path.

The floor tells you the physical distance. Light in fibre covers roughly 200 km per millisecond, and with routing overhead a realistic figure is about 1 ms per 100 km each way. A 24 ms round trip is consistent with a few thousand kilometres; you cannot optimise below that number without moving the server, which is what a CDN does.

Note also the ttl=56 in the reply. That is the remaining hop count, and since common starting values are 64, 128, and 255, a value of 56 implies eight hops. It is a rough way to guess how far away a host is even when traceroute is blocked.

A host that does not answer ping is not necessarily down. ICMP is routinely blocked at firewalls, deprioritised by routers under load, or rate limited. Cloud providers commonly drop it by default. Before concluding a server is unreachable, test the port you actually care about, since a TCP handshake to 443 proves far more than an echo reply.

What traceroute does

Traceroute exploits the TTL field. It sends a packet with TTL 1, which the first router decrements to zero and discards, returning an ICMP "time exceeded" message that reveals its address. Then TTL 2 for the second router, and so on until the destination replies. Each line is one hop further along the path.

$ traceroute example.com
 1  192.168.1.1        1.2 ms   1.1 ms   1.3 ms
 2  10.20.0.1          8.4 ms   8.2 ms   8.9 ms
 3  * * *
 4  ae-1.core.isp.net  71.2 ms  70.8 ms  71.4 ms
 5  93.184.216.34     24.3 ms  24.1 ms  24.2 ms

Two things in that output look alarming and are not.

Hop 3 shows asterisks. The router did not send a time exceeded message. That means it is configured not to generate ICMP, or it rate limited the response, both extremely common. Traffic passed through it fine, which you know because hops 4 and 5 answered. Asterisks are only meaningful if every hop from that point onward is also blank, which indicates the path genuinely stops there.

Hop 4 shows 71 ms, and hop 5 shows 24 ms. Latency appears to drop, which is impossible for a path that only grows. The explanation is that intermediate times measure the round trip to that router's control plane, not to the destination. Generating an ICMP error is a low priority task handled by a slower processor, so a busy core router can report high latency while forwarding traffic at line rate. Only the final hop's latency describes the actual path. Intermediate values matter only when the elevated latency persists through every subsequent hop, including the last one.

Two more properties worth knowing. Traceroute shows the forward path only; the return path can be completely different, and a problem on the return leg is invisible here. And modern networks load balance across multiple links, so consecutive probes may traverse different routers and produce output that looks inconsistent when nothing is wrong.

Reverse DNS names on the hops are informative once you learn to read them: strings like lon, fra, iad, sin are airport codes marking city, and they let you see the geography of the path. You can resolve any hop address yourself with the reverse DNS tool, and look up who owns it with the IP lookup tool. Reverse DNS and PTR records covers why those names exist.

MTR: the tool to reach for instead

MTR runs traceroute continuously and aggregates the results, which turns a single ambiguous snapshot into a statistical view:

$ mtr -rw example.com
HOST                     Loss%  Snt   Last   Avg  Best  Wrst StDev
1. 192.168.1.1            0.0%  100    1.2   1.3   1.0   4.2   0.4
2. 10.20.0.1              0.0%  100    8.4   8.6   8.1  14.0   0.9
3. ae-1.core.isp.net     18.0%  100   71.2  70.9  69.8  88.1   3.1
4. ae-3.edge.isp.net      0.0%  100   23.9  24.1  23.6  31.2   1.2
5. 93.184.216.34          0.0%  100   24.3  24.2  23.8  29.9   1.0

Hop 3 reports 18 percent loss while every hop after it reports zero. That is not a problem: it is a router rate limiting the ICMP responses it generates, while forwarding traffic perfectly. Loss that does not continue to the final hop is an artefact. Loss that appears at one hop and persists through every subsequent hop, including the destination, is a real fault at or after that point.

That single rule resolves most of the arguments that traceroute output produces in support tickets.

Working out whether it is the network or the server

Latency and loss look similar to a user regardless of cause. A short sequence separates them:

  1. Ping and MTR to the destination. Loss persisting to the final hop means a network path problem, and the hop where it begins tells you whose network to raise it with. Clean results mean the path is fine and the problem is above it.
  2. Time the connection phases. curl -w "dns:%{time_namelookup} connect:%{time_connect} tls:%{time_appconnect} ttfb:%{time_starttransfer}\n" -o /dev/null -s https://example.com splits total time into DNS resolution, TCP connect, TLS handshake, and time to first byte. A slow time_namelookup is a DNS problem; a slow time_appconnect is a TLS problem; a slow time_starttransfer with fast connect is the application. The other useful flags are in the curl cheat sheet.
  3. Compare from elsewhere. If a second location is fine, the problem is local or regional rather than with the server.
  4. Check resolution separately. A stale or wrong DNS answer looks like an outage. Confirm what the record says with the DNS lookup tool and what your machine has cached, as described in how to flush your DNS cache.

The pattern to internalise: ping tells you whether packets arrive and how consistently, traceroute tells you the shape of the path, and neither tells you anything about your application. Once the network is proven clean, stop looking at it.