RE:NODE
Обзор хостинга

Сеть13 мин чтения

Reading traceroute and MTR without fooling yourself

Loss in the middle of a trace usually means nothing. How traceroute works, what the MTR columns mean, and how to tell a real fault from a decoy.

Эта статья пока на английском. Мы её переводим.

0 прочтений

Here is the short version, because it is the part that gets people to the wrong conclusion: packet loss shown at a hop in the middle of a trace, which does not continue to the final hop, means nothing at all. It is the single most common misreading of a traceroute, it generates an enormous number of support tickets, and it is caused by routers deliberately deprioritising the replies traceroute depends on. The only loss that matters is loss that reaches the destination. The same applies to a latency spike at one hop that does not carry through to the end.

The second thing to know is that a traceroute shows you one direction, and the reply packets came back by a route you cannot see. That is why a useful report is always two traces, one from each end, taken at the same time. Everything below is how to get those, how to read them, and what to do with the answer - which is sometimes "nothing, and here is why".

How traceroute actually works#

Every IP packet carries a TTL - a hop counter, called the hop limit in IPv6. Each router that forwards the packet decrements it by one, and a router that decrements it to zero throws the packet away and sends an ICMP Time Exceeded message back to the sender.

Traceroute abuses this on purpose. It sends a probe with TTL 1, and the first router replies "time exceeded" - now you know hop 1. It sends another with TTL 2, and the second router replies. It walks outward like that, three probes per hop by default, until something replies from the destination itself rather than with a time-exceeded message.

The probes themselves differ by tool, and this matters more than people expect:

ToolDefault probeNotes
Linux tracerouteUDP to high ports from 33434Destination replies with port unreachable
Windows tracertICMP Echo RequestSame probe type as ping
macOS tracerouteUDP, like the BSD original-I switches it to ICMP
mtrICMP Echo Request-u for UDP, -T for TCP
tcptraceroute, traceroute -TTCP SYN to a chosen portGets through firewalls the others do not

The consequence: a firewall that drops UDP but allows ICMP produces a trace that dies halfway with one tool and completes with another, and neither result says anything about whether your game traffic gets through. If you care about a specific service, probe it with the protocol and port it actually uses.

mtr is traceroute and ping combined. It runs the walk over and over and keeps statistics per hop, which is the only way to see loss and jitter rather than a single sample. For diagnosis, use mtr and forget plain traceroute exists.

The commands worth knowing#

bash
# The one to use. Report mode, 100 cycles, wide output, ASNs and IPs shown.$ mtr -rwzbc 100 203.0.113.10# Probe the port the game actually uses, so ECMP hashes the same flow.$ mtr -rwzbc 100 -T -P 25565 mc.example.com$ mtr -rwzbc 100 -u -P 27015 cs.example.com# Plain traceroute over TCP to a web port, when ICMP is filtered.$ traceroute -T -p 443 example.com# Fast, numeric, no reverse DNS delays.$ mtr -rwnc 100 203.0.113.10

On Windows, install WinMTR - it is a GUI, it does the same job, and it is what most hosts ask for. The built-in tools are weaker but still useful:

code
tracert -d 203.0.113.10pathping -q 100 203.0.113.10

tracert -d skips reverse DNS, which removes the pauses that make it look stuck. pathping sends a large number of probes per hop and prints per-hop loss, so it is closer to mtr; it also takes several minutes on a long path, so start it and go and do something else.

Run at least 100 cycles. Ten probes cannot distinguish 0% loss from 10% loss with any confidence, and a report with -c 10 in it is not evidence of anything.

Reading an MTR report, column by column#

code
$ mtr -rwzbc 100 203.0.113.10Start: 2026-09-21T14:02:11+0200HOST: laptop                    Loss%  Snt  Last   Avg  Best  Wrst StDev 1. AS???     192.168.1.1        0.0%  100   0.6   0.8   0.5   3.1   0.3 2. AS64500   100.64.0.1         0.0%  100   8.9   9.4   8.1  21.0   1.6 3. AS64500   198.51.100.9      12.0%  100   9.8  10.2   8.9  30.4   2.1 4. AS64501   192.0.2.17         0.0%  100  24.1  24.6  23.8  40.2   1.9 5. AS64502   192.0.2.201        0.0%  100  25.0  25.3  24.6  33.1   1.1 6. AS64502   203.0.113.10       0.0%  100  25.2  25.5  24.8  35.0   1.2
ColumnWhat it means
Loss%Probes to this hop with no reply. Only meaningful on the last line
SntHow many probes were sent to this hop
LastThe most recent round trip, in milliseconds
AvgThe mean round trip. The number people quote
Best / WrstThe fastest and slowest samples, which bracket the jitter
StDevHow much the times varied. This is your jitter figure

That report is healthy. Hop 3 shows 12% loss and everything after it shows none, which is a router that rate-limits its own replies rather than a router that is dropping traffic. If hop 3 were really dropping one packet in eight, hops 4, 5 and 6 could not possibly be at 0% - their probes pass through it.

Read it in this order: last line first for loss, then Avg on the last line for latency, then StDev and the Best to Wrst spread on the last line for jitter. Only when the last line shows a problem do the middle hops become interesting, and then only to find where the problem starts.

The AS numbers are the other half of the story. They tell you whose network each hop belongs to, so you can see where your ISP hands off to a transit provider and where that provider hands off to the hosting network. The hop where the AS changes is a peering point, and peering points are where congestion actually lives.

Four things that look like faults and are not#

ICMP rate limiting. Generating a Time Exceeded reply is work for a router's control processor, and a busy core router will happily forward millions of packets per second while answering only a handful of traceroute probes. It is configured to protect itself. Symptom: loss or a wild latency figure at one hop, with clean hops after it.

Control-plane latency. The same cause, different symptom. A hop shows 200 ms while the destination two hops later shows 25 ms. The router was slow to generate a reply about a packet it forwarded instantly. Latency that does not propagate forward is not latency your traffic experienced.

*** * * at a hop.** A router configured not to send Time Exceeded messages at all, or a firewall dropping them. Extremely common inside carrier networks and around cloud providers. If the trace continues past it and completes, the hop is invisible, not broken. If the trace goes * * * from a hop to the end and never completes, that is different - something is blocking your probe type, and a TCP trace to a port the destination actually listens on will usually get through.

Private addresses and phantom hops. 10.x, 172.16-31.x, 192.168.x and the 100.64.0.0/10 range inside a trace are normal - carriers use private space for internal transport, and 100.64 specifically is carrier-grade NAT. Separately, classic traceroute changes the destination port on every probe, and where the network load-balances across several equal-cost paths, each probe can take a different one. That produces hops that appear and disappear and loss figures that make no sense. Fixing the port with -P keeps every probe on the same path and makes the result reproducible.

Two more smaller ones: MPLS tunnels can hide every router inside them, so a trace may jump from one city to another in a single hop with a latency increase that looks alarming and is just distance; and a trace taken while the network reconverges after a route change will show one half of a route and then the other.

What a real fault looks like#

code
HOST: laptop                    Loss%  Snt  Last   Avg  Best  Wrst StDev 1. AS???     192.168.1.1        0.0%  100   0.7   0.9   0.5   4.2   0.4 2. AS64500   100.64.0.1         0.0%  100   9.1   9.6   8.2  19.7   1.4 3. AS64500   198.51.100.9       0.0%  100  10.0  10.4   9.1  24.0   1.8 4. AS64501   192.0.2.17         6.0%  100  24.3  38.9  23.9 210.4  31.7 5. AS64502   192.0.2.201        6.0%  100  25.4  40.1  24.7 214.8  32.9 6. AS64502   203.0.113.10       6.0%  100  25.6  40.6  24.9 219.1  33.4

Three things make this real, and all three have to be present before you say so out loud:

  1. The loss starts at one hop and continues at roughly the same rate to the end. 6% at hop 4, 6% at every hop after. The probes that reach hop 6 had to survive hop 4.
  2. The latency degradation persists. Avg jumped from 10 to 39 at hop 4 and stayed there. Compare Best with Avg: Best is still 24 ms, so the path is capable of 24 ms and something is queueing.
  3. `StDev` and `Wrst` explode together. A Best of 24 and a Wrst of 210 with a standard deviation of 32 is a congested link, not a long one. Pure distance gives you a high Avg with a low StDev.

That pattern - low Best, high Wrst, high StDev, starting at one hop - is congestion, almost always at a peering point at a particular time of day. A high Avg with a tight spread and no loss is distance, and distance is physics. Latency, jitter and packet loss is the shorter version of why those three get confused.

Both directions, because the return path is different#

Internet routing is asymmetric by default. The route from you to the server is chosen by your ISP and its transit providers; the route back is chosen independently by the hosting network and its providers. They are frequently not the same path, and they can have completely different congestion.

Every number in your trace is a round trip, so a clean-looking forward path with bad numbers at the end can be a return-path problem you cannot see any of. The only way to see it is a trace run from the other end back to you.

outbound pathreturn pathinvisible to your traceYour connectionwhere the trace startsYour ISPfirst few hopsTransit outwhat your trace showsThe serverDE-01, GermanyTransit backa different route
The path out is not the path back

To run the return trace you need a shell on the server. On a VDS you have one, so mtr -rwzbc 100 <your-address> is all it takes. On a container-based game or app plan you do not get a shell - the panel console is the application's own output and command line, not a terminal - so that half of the picture has to come from a ticket. Include your own public address, or the trace will be run to nowhere useful.

If you are behind CGNAT at home, there is no address to trace back to, and the return-direction test has to be done from a machine that does have a public address. That is worth knowing before you spend an evening on it.

Matching a trace to what players report#

A trace is only useful if it corresponds to the complaint. Three checks before you draw conclusions:

  • Probe the right thing. Most games are UDP - TCP vs UDP for game servers covers why - and a path that is clean for ICMP can still have a firewall or a rate limit in front of the game port. Trace with -u -P at the game port, or -T -P for TCP games such as Minecraft, and confirm the port is what you think it is with game server ports explained.
  • Test while it is bad. A trace taken at 11:00 tells you nothing about the congestion everybody feels at 21:00. Peering congestion is an evening phenomenon. Run mtr during the bad window, for several minutes.
  • Rule out the server first. Lag that affects every player simultaneously, regardless of where they live, is usually not the network. Check the panel's CPU and memory graphs and the tick rate before touching a trace - reading a server load graph and what tick rate actually means are the faster route to that answer, and for Minecraft specifically, why TPS drops.

One player lagging is that player's connection. Every player lagging is the server or its uplink. Players from one country lagging is a route.

What you can do about each answer#

Hop 1 or 2 is bad. That is your own network: wifi, a cheap router, a saturated upload, or somebody else in the house streaming. Test over a cable before anything else. A wifi link that shows 30 ms of jitter to your own router is the entire problem and no host can fix it.

Your ISP's own network is bad. Their support will usually not accept an MTR, but they will accept "packet loss between your equipment at hop 3 and my connection". Be specific, include timestamps with a timezone, and repeat the test over several days so they cannot call it a one-off.

A transit network in the middle is bad. This is the honest and unsatisfying case: neither you nor your host has a contract with that network, and neither of you can route around it directly. What a host can sometimes do is influence which of its own transit providers the return traffic uses, which is exactly why they need the trace from your end. Send it, and be patient.

The destination network is bad. Loss that starts at the last hop or two, affecting everybody, and persisting across days. That is worth a ticket with evidence. On RE:NODE, a ticket from the panel reaches all staff and takes private attachments, which is the right place for traces and screenshots.

Be aware of what a trace cannot tell you. It cannot distinguish congestion from an attack in progress; large volumetric floods are dropped upstream, and traffic shaped to look like real players is not filtered at all - DDoS attacks on game servers explains where that boundary sits. And the usual advice to "pick a closer location" does not apply here: RE:NODE runs one location, in Germany, on hardware we run. If your players are in Brazil, no configuration changes the speed of light across the Atlantic, and the right answer is to know that before you buy rather than after. Choosing where your server lives is the honest version of that trade.

Sending a report that gets acted on#

Whoever reads your ticket wants to reproduce your conclusion, not your feeling. Include:

  1. Both directions, if you can get them, taken within a few minutes of each other.
  2. At least 100 cycles, as text rather than a screenshot, so it can be searched and quoted.
  3. Timestamps with a timezone. mtr report mode prints a start time. Say when the problem happens and when it does not.
  4. The address and port, and which protocol you probed with.
  5. Who is affected: one player, a region, or everybody, and since when.
  6. A comparison. A trace to a different server in the same city that is clean turns an anecdote into evidence.

What not to include: a single tracert with four probes per hop and a red circle around a hop in the middle. That is the report this entire post exists to prevent.

FAQ#

Why does one hop show 50% loss but the rest show none?

The router at that hop is rate-limiting the ICMP replies traceroute needs, which is normal and deliberate. It is forwarding your actual traffic fine, which is proved by the hops after it showing no loss. Only loss that continues all the way to the final hop is real.

Should I use traceroute or MTR?

MTR, for anything you intend to act on. Traceroute takes three samples per hop, which is not enough to measure loss or jitter. MTR repeats the walk continuously and gives you loss, average, best, worst and standard deviation per hop. WinMTR is the Windows version.

How many packets should I send?

At least 100, with -c 100. Below that the loss percentage is noise. If you are chasing an intermittent problem, run it for several minutes during the bad period rather than once.

Why is the latency higher than a ping to the same address?

It should not be, on the final hop - that line is effectively a ping. If intermediate hops show higher numbers than the destination, those are control-plane delays in generating replies, not delays your traffic experienced. Read the last line.

Does a trace show the path my game traffic takes?

Only if you probe with the same protocol and port, and even then only approximately. Networks load-balance across equal-cost paths using a hash of the addresses and ports, so a different port is potentially a different path. Use -T or -u with -P set to the real port.

The trace stops halfway and never reaches the server. Is the server down?

Usually not. A firewall somewhere is dropping your probe type or refusing to answer, while still forwarding real traffic. Try a TCP trace to a port the server actually listens on. If the game connects, the path is fine whatever the trace says.


Комментарии

Полностью анонимно: без аккаунта, без почты, без cookie. Мы храним имя, которое вы ввели, текст и время - больше ничего. Количество ссылок ограничено, разметка не отображается.

0/2000