A single radio system failed and every train in Germany stopped at once

The whole network parked itself in an evening

On the evening of June 23, Deutsche Bahn suspended train service across all of Germany. The cause was GSM-R — the dedicated mobile network railways use for safety-critical voice and train-control communication. With it down, DB held every train at a station rather than let anything run blind. Its own notice read: "Due to a nationwide outage of the GSMR digital rail radio system, all trains are being held at stations." Service began staggering back to life just after midnight, restarted deliberately because the current draw of thousands of trains accelerating at once could itself cause problems.

A detail several commenters latched onto: GSM-R is built with heavy redundancy and is marketed as fail-safe, yet the fallback failed too. One informed guess in the thread was that handsets falling back to the normal GSM network still have to authenticate with their GSM-R credentials — so when the home subscriber registry breaks, even roaming dies.

Sabotage, or just Deutsche Bahn being Deutsche Bahn

The thread divided immediately between two explanations, and the funniest line captured the prior: "If this weren't Deutsche Bahn, I'd say it's a cyber attack. Given that this is Deutsche Bahn, though, it may just as well be a maintenance issue."

The sabotage case was made most concretely by someone who works the field:

"Since invasion of Ukraine there have been numerous cable cuttings on train tracks, several train derailments, some fires. It has become so bad that police helicopters are regularly patrolling train routes at night… So as a person working in cyber security, I'd put this into the sabotage bucket." — a commenter working in cybersecurity

The counter-case was mundane and, to many, more likely. Word from the German rail community pointed at a software update, and one commenter sketched a specific failure mode:

"GSM-R is 90s GSM, they'll likely have a fried HLR & VLR because in any GSM network these are fundamental, without them you can't even get roaming from public phone networks working." — a commenter who diagnosed a fried HLR/VLR

There's also a structural reading that blames neither hackers nor a bad patch but the last three decades. An account of DB's half-finished privatization laid it out: the government spun the railway into a private company it still wholly owns, then skimped on maintenance to fatten the numbers for a sale that never came.

"the maintenance debt came rearing its ugly head and now everything is royally screwed… the only thing that will solve this is if they re-nationalize the tracks & infrastructure." — a commenter on DB's botched privatization

Passengers in the thread reported the experience in real time: an ICE stuck in Munich with the delay creeping from 30 minutes to two hours, riders in Erfurt watching the fastest Munich–Berlin trains released first to clear the tracks, two founders in Oberhausen giving up and ordering an Uber to make a morning flight. What none of them got, for hours, was an announcement that the problem was nationwide.

The uncomfortable note for engineers: this same outage happened roughly two years ago. Two years was apparently not enough to make the failsafe actually fail safe.