· AMBER team
Postmortem: 41 minutes of packet loss on the Frankfurt uplink
A faulty optics module, a bad monitoring threshold, and what changed afterwards.
On 2 November, game traffic on the Frankfurt node saw 3–8% packet loss for 41 minutes. Sessions stayed connected; tick times spiked and players reported “lag” without a disconnect, which is exactly the shape of problem that is easy to misdiagnose from the outside. This is what happened and what we changed.
Timeline (local time)
| Time | Event |
|---|---|
| 20:14 | First packet loss observed on one of two transit links. |
| 20:22 | Monitoring fires on latency, not on loss — the loss threshold was set at 10%. |
| 20:31 | Engineer confirms loss on the link with mtr; second link is clean. |
| 20:44 | Traffic moved off the affected link. Loss stops on game ports. |
| 20:55 | Optics module replaced; link brought back and monitored. |
What went wrong
The module itself was failing — an intermittent fault that only appeared under load. That part is hardware and will happen. The interesting failure was the monitoring: we alerted on latency above a threshold, and 3–8% loss on one of two links raised average latency only slightly because most traffic still took the healthy path. The alert that should have fired did not, and eight minutes passed before anyone looked.
The second mistake was slower to correct than it should have been. We have a documented runbook for a bad link, and moving traffic is a single command. It took thirteen minutes from confirmation to action because the on-call engineer was verifying that the loss was not caused by the game server itself. That caution was reasonable; the missing piece was a dashboard that would have shown the same loss pattern across every instance on the node, which would have made the diagnosis immediate.
What changed
- Packet loss on any transit link now alerts at 1%, not 10%, and it alerts independently of latency.
- The node dashboard shows per-link loss next to per-instance tick time, so “is it the game or the network” is one glance.
- The runbook now starts with “compare across instances” rather than with the affected server's log.
- Spare optics for every node model are held on site instead of ordered when needed.
What did not change
We did not add a status page entry at the time, because the page had no manual override and the automated check reads game ports, which stayed reachable. That was the wrong call — 41 minutes of degraded play is an incident whether or not a health check turns red. The status page now has a manual entry path, and using it during an incident is part of the runbook.
Forty-one minutes of degraded play, no data loss, no disconnects. We would rather not have written this post. We would rather have the monitoring that would have made it fifteen minutes.