upvote
I don’t know if this is a real question, but for most credentials a stolen key is worse than downtime on a product.
reply
No, a leaked credential is a nothing event in a mass leak as far as expected loss goes.

You’re better off waiting and monitoring until you’re sure you can get back online if your service supports any kind of revenue.

reply
What are you talking about.
reply
The customers of my company are angry. It is 9.45 AM here in SE Asia. Things are bad.
reply
Yeah. OpenRouter disabled my keys because of this and now i'm in a catch-22
reply
thought I was the only one, openrouter is auto disabling railway affiliated keys
reply
Same situation. Frankly unbelievable - how do you not lead with this? You can't tell customers to rotate their keys and not inform them in the same breath them that actually, environment variable updates currently cannot deploy.
reply
They're actively firefighting an incident, and are likely going to make some mistakes along the way. In this case, they've erred on the side of security over availability, which in some cases is the better outcome. I know not in all, but some.
reply
They did.
reply
No. They did not. From https://blog.railway.com/p/security-disclosure-wednesday-oct... :

> What you should do: As such, we strongly recommend that you immediately roll all environment variables, including any API keys, stored in Railway, as well as any private registry credentials, and review logs for those services for unusual activity.

> All deployments are paused while we investigate. Until builds are restored, you will not be able to roll out new versions. We apologize for this inconvenience.

They lead with rotating keys.

Next, I did not realize that this meant not being able to restart an existing deployment to bring up new env vars. They also speak of builds. I don’t need a new build. I need my existing deployment to work.

It was not at all clear ‘paused’ mean this; maybe I should have realized, but I had interpreted it as doing new deployments or builds. New services. Code changes. New builds as they even say. It’s fully reasonable to pause that in this sort of situation and so I had interpreted their statement that way.

If there is a clear impact from following advice — ‘do what we recommend and your service will not function’ — those who provide that advice need to make that clear.

reply
Their first message where they recommended rotating keys did not make that clear (though obviously if you cannot deploy, rotating keys means you will go down)
reply
Yeah, I had to rotate my data store keys ASAP. Now I'm stuck with a broken application. Best option, horrible option.
reply
deleted
reply