An update on FM Dojo AI interruptions
What happened, why keys were revoked, and what has changed.
Over the past few days, some FM Dojo users have seen “AI service authentication error” instead of an AI response. I’m sorry for the interruption and the confusion, especially if you could see credits in your account and had no obvious reason for a request to fail.
The issues involve a security incident at Railway, the platform that hosts FM Dojo, and API-key revocations by Anthropic and OpenAI. Here is what happened, what I’ve changed, and what remains unconfirmed.
What happened
Railway reported a security incident on October 21. According to Railway, the attacker’s access ended on October 22, and its October 24 update says it has no further evidence of compromise. Railway’s disclosure includes potential exposure of environment variables, which can contain service credentials.
An API key is the credential FM Dojo uses to authenticate with an AI provider. If that provider disables the key, requests using it can no longer succeed, even when a customer has credits available.
On October 23, Anthropic disabled a key, and FM Dojo recorded invalid-key errors shortly afterward. Customers also reported authentication errors. Those reports confirmed that people were having trouble using the service, although they did not establish that every feature or model was affected.
On October 24, OpenAI separately disabled an FM Dojo key. These were two different providers and credentials, so replacing one did not address the other.
Why replacement keys were also revoked
Railway’s October 24 clarification matters here. It said that a batch of key identifiers shared with providers during the incident response inadvertently included newer replacement keys. Some of those keys were then revoked as a precaution.
Railway says these later revocations are part of its response to the October 21 incident, not evidence of a new incident. A replacement key being revoked does not, on its own, show that the replacement was compromised. Railway also says keys that are still working do not need another rotation because of this batch.
That explains why a provider’s revocation notice can sound more alarming than the later incident update. The key really was disabled, but that fact alone does not establish a fresh leak.
What I have changed
I updated the Anthropic credential and replaced the OpenAI key. There was also a deployment gap during the Anthropic replacement: the new key had been saved but had not yet been deployed. That distinction mattered, and I’m sorry it added to the disruption.
The replacement OpenAI key is now deployed. If you still encounter an error, please report it using the details below.
What this means for you
An authentication error is about FM Dojo’s connection to the provider. It does not, by itself, mean you have run out of credits. Adding credits is not a fix for a disabled service key.
The scope of any FM Dojo customer-data exposure is not yet established.
If you still see an error, please let me know which feature and model you were using, roughly when it happened, and the error text. A screenshot can help, but please leave out passwords, API keys, and other sensitive information.
Thank you for reporting these problems. You deserve a straightforward explanation, and I’m sorry for the disruption.
Agi