On October 21, Railway told customers that an unauthorized party had accessed environment variables, including some decrypted API keys and credentials. It paused deployments and asked everyone to rotate all secrets. In the same news cycle, version 1.107.0 of LiteLLM on PyPI was reported to contain a credential stealer and a backdoor driven from GitHub. The 1.106.2 release and the 1.107.0 release candidates are reported clean.
Two different incidents. Same target: the pile of API keys sitting in your environment.
Where your secrets actually live
Think about how many places a production key sits right now. The platform's dashboard. The CI system. A .env file on three laptops. The container at runtime. Maybe a Slack message from eight months ago.
The Railway incident hit the first one. You can run a perfectly tidy codebase and still lose keys because the place you stored them got breached. The LiteLLM incident hit the last one. A dependency ran inside your process, and anything in that process could read your environment.
AI projects make this worse. A typical AI feature holds keys for a model provider, a vector database, a payments account, and probably a few connectors. These keys are often broad, because someone needed the demo working by Friday. They also cost real money when abused, which makes them worth stealing.
The real question is how long rotation takes
Everyone says "rotate your secrets." Fewer teams can answer how long that takes them. If your honest answer is "a day, if the right person is online," then you have a process problem and the next breach will find it.
A rough test: pick one production key and rotate it today, start to finish, without breaking anything. Time it. If it hurts, you've learned something cheap.
What makes rotation fast:
- One owner per secret, written down, so nobody is guessing who can change it
- Secrets injected at deploy time from one source, not copied into five dashboards
- Separate keys per service and per environment, so one leak doesn't open everything
- Spend caps and usage alerts on every provider key, because a stolen AI key shows up on your bill first
Treat dependencies like code that can read your secrets
A package update is code you didn't write running with your permissions. Most teams auto-bump versions on a schedule and trust the registry. That's how a bad release reaches production within hours.
A few habits help without slowing anyone down. Pin exact versions and commit your lockfile. Add a short delay before adopting brand new releases, a few days is often enough for a bad one to get noticed. Run your build and your tests in an environment that holds no production credentials. And for anything that sits between your app and your model providers, like a gateway or proxy, be extra strict. It sees every key you have.
What we'd do this week
Check whether you were affected: if you host on Railway or ever installed LiteLLM 1.107.0, rotate everything those environments could reach. Then audit your keys. List every production secret, who owns it, and when it was last rotated. You'll probably find a few nobody remembers creating.
Then make rotation routine. If you rotate keys monthly, a leak stops being an emergency. It becomes a Tuesday task, and the stolen key is already dead.
We're here to help founders and teams design and build digital products that are built to scale with you, not slow you down. If you're looking to build something, get in contact with us today!