All posts

Your host was breached: rotate environment variables in the right order

Security · 4 min read ·

On 21 October 2026 Railway published a security disclosure: that morning it detected anomalous activity in systems that store customer environment variables, and its investigation so far indicates an unauthorized third party may have accessed encrypted variables and, for some of them, obtained the decrypted contents. Railway recommends that all users roll every environment variable stored there, including API keys and private registry credentials, and review the logs of those services. If you deploy a Laravel or Node app anywhere, the lesson reaches past one company: your secrets live in someone else's database, and the day that database leaks is the day you find out whether you can rotate them.

What Railway has said

According to Railway, the affected variables may contain API keys and database and registry credentials. The company says its initial analysis suggests the impact may be limited to a subset of users, but it still asks everyone to roll. It believes the incident is part of a series of attacks recently aimed at platform companies, with the goal of reaching AI inference providers, and it notified those upstream providers so they could revoke keys proactively. Deployments were paused during the investigation, so users could not ship new versions until builds were restored. Railway says it will email confirmed-affected customers and post updates on status.railway.com.

That last detail matters for planning. If your rotation depends on deploying through the platform that was breached, you may have to wait for it, which is a good reason to have the list ready before you need it.

Provider issues the key Your repo should hold none Your CI secrets never in build logs Host env store the copy in the news every copy is a place it can leak
One key, several copies. Rotating means changing it at the provider, then in every copy.

The inventory you should already have

What follows is general guidance, not anything Railway prescribed. Start with a list of every secret the app uses and where each one came from. For a typical Laravel or Node app that means the database password, queue and cache credentials, mail service keys, payment provider keys, AI provider keys, OAuth client secrets, webhook signing secrets, object storage keys such as S3, and monitoring tokens like Sentry. Next to each one write the provider, who can rotate it, and which services read it. Without this list, rotation turns into reading old code to find out what the app even talks to.

Laravel has one secret with side effects. Changing APP_KEY invalidates what was encrypted or signed with it: encrypted cookies, sessions, values stored with Crypt, signed URLs. Laravel supports a list of previous keys that are still used for decrypting during a rotation; read the docs for your version before you rely on it. The shape is a comma-separated list in the env:

APP_KEY=base64:the-new-key
APP_PREVIOUS_KEYS=base64:the-old-key

Keep in mind that if the old key leaked, data encrypted with it was readable by whoever has it, so re-encrypt what matters and drop the old key once it is no longer needed.

Rotate in this order: create, deploy, revoke

The order that avoids downtime is simple. Create a new credential at the provider while the old one still works. Put the new value in the env and deploy. Check that the app is using it. Only then revoke the old one, at the provider, not just by deleting it from your env file: a key you removed from your app is still a working key for anyone who copied it. Afterwards, open each provider's usage dashboard and look for activity you did not cause, especially on paid APIs.

1. Create new key 2. Deploy new value 3. Verify app uses it 4. Revoke old at the provider the old key keeps working until step 4, so nothing goes down in between
Revoke last, and revoke at the source. Deleting a line from .env revokes nothing.
A secret is only rotated when the old value stops working at the provider, not when it disappears from your env file.

Make the next one smaller

Again as general guidance: use scoped keys with spending limits where the provider offers them, separate keys per environment so a staging leak is not a production leak, and never let a secret reach a git repo or a build log. Each of these shrinks what one leaked store can cost you.

The next step is concrete. Today, write the inventory for one app: every secret, its provider, and the create, deploy, revoke steps for it. If you are on Railway, follow its instructions and roll everything now. If you are not, do the drill anyway, because the question is not whether your host is next but whether you could answer in an hour.

Source: Railway Blog, "Security Disclosure: Wednesday October 21st".

Notes from readers

  1. No notes yet. The first one is the bravest.
Building something like this? I can help. Message me on WhatsApp