One power failure in one London zone took down HSBC, Revolut and Lloyds
Google Cloud's status page blamed a power event in europe-west2-b. The commenters spent the day asking why banks hadn't survived one zone.
The outage began 19 October at 23:08 PDT; Google called it a power event in a single zone. Per the Neowin headline, HSBC, Revolut, Bank of Scotland, Lloyds and Equifax were hit.
An SRE in the thread says multi-zone setups exist everywhere but zone failure is almost never rehearsed.
What Google said
The Neowin report names HSBC, Revolut, Bank of Scotland, Lloyds and Equifax among the affected. In the thread, nickcw quoted the status page: multiple services in europe-west2 were affected from 23:08 PDT on 19 October, "due to a power event" in europe-west2-b, with partial mitigation expected within 30 minutes. He added that reporting the times in PDT is unhelpful for UK users.
Why multi-zone didn't help
"Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out." — a commenter who has run multi-AZ systems at several employers
Another argued a Google zone isn't an AWS availability zone, and that some regions share facilities. A third said certain hardware exists only in some zones, so placing VMs means trading off zone spread. A commenter pointed to legal limits on where a country's cloud data may sit, which keeps the eggs in nearby baskets.
One reader defended the banks: the plan may simply have been "restore backups into another zone" if the zone is lost, with waiting a few hours accepted as the cheaper risk.
If your failover plan has never been executed, treat it as untested.
Get breakdowns like this every day
The top Hacker News stories, unpacked — by email, Telegram, or RSS.
One researcher found 333 Tor exit relays quietly intercepting SSH sessions
About a tenth of Tor's exits were infected with the Ebury backdoor and redirecting SSH traffic through twelve attacker servers.