Northflank: assume every environment variable is stolen, and customers say AI keys are already being used

From one Wiz alert to "assume all environment variables have been compromised"

Northflank's status page marks a "Security incident" as identified, alongside a major outage, with updates running from October 30 to 31 (UTC). A Wiz threat alert arrived at 11:43 UTC on October 30. Network policy patching began at 12:02, Wiz's incident response team was requested at 13:00, and further patches went out through 13:05.

The scope statement moved the wrong way for customers. Northflank first said environment variables had been decrypted, then told users to "assume all environment variables have been compromised". Multiple users reported that their AI keys had been used without authorization. The page says the investigation is ongoing and does not specify what data was taken.

Customer instructions are blunt: rotate every secret in workloads, secret groups and global secrets; revoke AI provider keys with their issuers and issue new ones; disable public access to addons; check provider usage and billing logs. Northflank turned on Postgres and MongoDB secret rotation for all users, and on October 31 enabled audit logs for every customer, including free ones.

Will this push infrastructure startups back to hyperscalers?

The first reaction in the thread was a prediction. dangoodmanUT expected a "convergence back to hyperscalers" after incidents like this and Railway's. The replies went the other direction. digitaltrees argued a bespoke dokku setup on your own hardware makes a smaller and less rewarding target, and a commenter pointing to Azure answered the hyperscaler claim with a single reply: "Azure would beg to differ."

One reader noted what the attackers were actually after:

"they're AI API keys, things that only existed because people were using hyperscalers instead of running inference on their own hardware"

"I count at least six" multi-tenant platforms since September

A platform/DevOps and infosec practitioner zoomed out from Northflank:

"I count at least six substantial, lightning-speed attacks against multi-tenant as-a-service vendors since September. Not this year--just since last month. PingCAP/TiDB, Fireworks AI, Zeabur, OpenCode Console, Railway, now Northflank."

Whether the cause is AI-accelerated attackers, as that commenter believes, or ordinary tenant-isolation bugs found at speed, the shared pattern is the payload. Environment variables were the one place where a customer's cloud, database and model-provider credentials all sat side by side.