CrowncodesAdeola

§ Writing11 min read

The outage was in billing

Two Claude incidents refused requests because of account state, on every model at once. Fallback chains need model, provider and account error classes.

On 7 October Anthropic posted this to the Claude status page:

We have identified an issue which resulted in some organizations to be incorrectly paused for having reached their spend limit, resulting in refused requests on the Claude API, Claude.ai, Claude Code, Claude Security, and Claude Cowork. This is now resolved.

Claude status, “Issue with spend limits”

That’s the whole incident note. It was posted as resolved at 21:23 UTC with no earlier updates, no duration and no root cause. Six days before, purchased credits were slow to reach account balances, “resulting in some requests failing for insufficient credit balance”. That one ran from 16:20 UTC (investigating) to 22:36 UTC (resolved).

Neither was a model outage. The models were up and the API was answering. It was answering “no”, to organisations whose standing with the provider had gone wrong, and it would have said the same thing for every model they asked for. The usual fallback chain (try the small Claude model, then the bigger one) walks straight through a failure like that and comes out the other side with nothing.

This post is news analysis followed by design analysis with illustrative code. I haven’t run the code, I have no inside knowledge of either incident, and where the sources don’t say what callers saw I’ve marked my guesses as guesses. My model gateway post now has one paragraph on account failures as a fourth error class. This is the longer version, with a different example.

The running example: a receipt reader

Someone photographs a receipt, and a model fills in the expense claim: merchant, date, total, VAT. The route chain was written a while ago and is the kind that gets written once and left alone:

  1. Claude Haiku 5.5 (claude-haiku-5-5)
  2. Claude Sonnet 5.5 (claude-sonnet-5-5)
  3. gpt-4.1-nano on OpenAI, added as the “different provider” route and never looked at again
  4. A blank form the person fills in by hand

Hold on to route 3. It matters later.

Three classes, sorted by reach

The gateway post sorted errors by whether to retry. That’s one question. The other is how much of your chain a failure has just taken out, and I think that one deserves its own field.

  • A model failure takes out one route. One model is overloaded, rate limited or retired.
  • A provider failure takes out every route at one provider. Their API is down or erroring across models.
  • An account failure also takes out every route at one provider, but only for you. A spend limit, a credit balance, a key, a billing detail.

Provider and account failures reach the same distance, so why separate them? Because of what happens next. A provider outage ends when the provider fixes it, and their status page will tell you. An account failure may never end by itself, because the fix can be a person logging in to a billing console, and no status page is going to mention your expired card.

The incidents, classified

Incident or failure What the caller saw, per the source Class What the receipt reader’s chain does
Claude, 5 and 6 October: elevated errors for Mythos 5.1 and Fable 5.1 (impact 12:40 to 13:10 UTC), then for Opus 5.5 (opened 12:24, resolved 12:43 UTC) “Elevated errors” on requests to the named models. No status codes given. Model Works as designed. This is the failure a model-to-model chain was built for.
OpenAI, 6 October: “Elevated errors across ChatGPT, Codex, and the API including the Agents API”, one of nine incidents in the feed that day Around 30 components listed, Responses and Realtime among them. No status codes or cause given. Provider Route 3 fails. Harmless here, because routes 1 and 2 are elsewhere. A chain made only of OpenAI models has nowhere to go.
3 September: OpenAI, Anthropic and xAI down during overlapping hours From a secondary write-up, which finds no evidence of a shared cause. I haven’t checked it against the status pages. Provider, twice Routes 1 to 3 all fail. Only the blank form is left.
Claude, 1 October: delayed credits “some requests failing for insufficient credit balance”. No status code given. My guess: the 402 billing_error in the errors doc. Account Route 1 fails, route 2 fails the same way. Route 3 saves it, if route 3 works.
Claude, 7 October: organisations wrongly paused at their spend limit “refused requests”. My guess: the documented spend-cap response, a 429 with no retry-after. Anthropic haven’t said. Account Sees a 429 and retries route 1 with backoff. Then does it again on route 2. Route 3 answers in the end, after both backoffs.
Cloudflare AI Gateway, from 6 October: provider rejects the credential on POST /ai/run 401 with error code 2009. With Unified Billing, 503 instead. Account for the 401. I’d file the 503 under provider (my reading, below). If the chain runs through the gateway: stops on the 401 if 401 is fatal, which abandons a working route 3. Retries the 503.
LiteLLM before v1.103.0: a fallback target the caller’s budget doesn’t cover Nothing. As I read the notes, the request went through on the fallback. Account (your own limit) Falls back to a paid model past the budget that was meant to stop it.
OpenAI, 23 October: legacy snapshots shut down, gpt-4.1-nano among them Hasn’t happened yet. The page says the model “will no longer be accessible”. My guess: a not-found error. Model, permanent Nothing, until the day routes 1 and 2 are both out. Then route 3 fails too.

The pattern in the last column is that a chain ordered by model does the right thing for exactly one row.

What each class should trigger

Model failure. Retry if the signal is transient (overload, a rate limit with retry-after), then switch route. Staying on the same provider is fine. If it’s permanent (the model ID doesn’t exist), don’t retry, take the route out of the policy and open a ticket. Nobody needs waking up.

Provider failure. Skip every remaining route on that provider and go to the next provider or the degraded mode. Alert if it lasts. There’s nothing for a person to do except watch the status page, so I wouldn’t page for it unless the degraded mode is unacceptable.

Account failure. Don’t retry. Skip the provider. Page a human now, with the provider, the error body and the request ID in the page. This is the only class where a person can fix it and waiting can’t.

That last one has a dependency that’s easy to miss: the person paged has to be able to do something. If on-call can’t open the provider’s billing page, top up credits or raise a limit, the page wakes someone up to read an error. I’d check that before I wrote any of the code.

For the receipt reader, 7 October should have gone: one refused request on route 1, skip route 2 without trying it, serve from route 3, page. One wasted call per request until a provider-wide breaker opens, and then none.

Status codes don’t hand you the class

It’s tempting to read the class off the HTTP status. You can’t, and the docs are quite open about it.

Anthropic’s rate limits page says that at your tier’s monthly spend cap, requests return a 429 whose error type is rate_limit_error, “the same as for a rate limit, but the response has no retry-after header”. The field that tells them apart is error.details.error_code, set to enforced_spend_limit_reached. A spend limit you set yourself is different again: a 400 invalid_request_error whose message begins You have reached your specified API usage limits. Most retry code treats a 400 as “the request is wrong, give up”, which is the right amount of retrying and the wrong amount of telling anyone.

OpenAI’s error codes guide lists several 429s that are about the account: credit_balance_exhausted, organization_spend_limit_exceeded, project_spend_limit_exceeded and organization_usage_limit_exceeded. They sit beside the ordinary 429 whose advice is to follow Retry-After.

Then there’s the gateway in between. Cloudflare’s 6 October change makes a rejected provider credential on POST /ai/run a 401 with error code 2009. Before that, the changelog’s own table says, Google Vertex returned a 500 “with upstream retries” and other providers a 402 “or another provider-specific status”. The exception is Unified Billing, where a rejected credential is now a 503. I read that as deliberate: under Unified Billing the credential is Cloudflare’s, so it isn’t your account that’s broken. The changelog doesn’t say whether that 503 is safe to retry, and I haven’t tested it.

So the classifier has to read the body. Here’s the shape I’d start from (illustrative code, unrun, covering only the cases above):

type Scope = "model" | "provider" | "account";
type Failure = { scope: Scope; retry: boolean; page: boolean };

type ProviderError = {
  status: number;
  code?: string;      // Anthropic error.details.error_code, OpenAI error.code
  message?: string;
};

const ACCOUNT_CODES = new Set([
  "enforced_spend_limit_reached",        // Anthropic tier spend cap (429)
  "credit_balance_exhausted",            // OpenAI (429)
  "organization_spend_limit_exceeded",   // OpenAI (429)
  "project_spend_limit_exceeded",        // OpenAI (429)
  "organization_usage_limit_exceeded",   // OpenAI (429)
]);

export function classify(e: ProviderError): Failure {
  const account: Failure = { scope: "account", retry: false, page: true };

  if (e.code && ACCOUNT_CODES.has(e.code)) return account;
  if ([401, 402, 403].includes(e.status)) return account;
  if (e.status === 400 && e.message?.startsWith("You have reached your specified")) {
    return account;                      // Anthropic: a spend limit you set
  }
  if (e.status === 404) {                // guess: what a retired model ID returns
    return { scope: "model", retry: false, page: false };
  }
  if (e.status === 429 || e.status >= 500) {
    return { scope: "model", retry: true, page: false };
  }
  throw new FatalRequestError(e);        // other 400s: our request is wrong
}

Two things about it. Availability errors start as model failures, because one response can’t tell you whether the whole provider is down. The calling loop promotes them: if two different routes at the same provider fail with availability errors inside one request, skip the rest of that provider. And an unknown 4xx never becomes an account failure by default, or a bug in a prompt template would page someone every night.

The string match on the 400 is ugly and will break when the wording changes. I’d still rather have it and a test that fails loudly than treat a spend limit as a malformed request.

Your gateway’s budgets are account state too

The account doesn’t have to be the provider’s. If you run a proxy with per-team budgets, that’s a spend limit you operate, and it has the same failure modes in both directions.

LiteLLM v1.103.0, released on 28 September, lists “fix!: re-check budget on router fallback targets”. The release notes put it as “Budgets are re-checked on every router fallback target”, and explain that a request that starts on a free model and falls back to a paid one is now gated at the fallback. I take that to mean it wasn’t before: the fallback chain was a way round the budget. v1.103.3 fixes streamed chat completions through a dotted alias recording $0 spend, because pricing looked up the alias and not the underlying model. A budget can only refuse what the spend log has counted.

In reconciliation work the rule was that a control has to sit on every path to the ledger, including the correction path nobody uses. A fallback route is that path. I haven’t tested LiteLLM for this post, so this is a reading of release notes, but I’d check which version produced any spend numbers I was relying on.

Before 23 October: check every route ID

Back to route 3. OpenAI’s deprecations page has a notice dated 22 April 2026 with a long list of snapshots shutting down on 23 October 2026. gpt-4.1-nano is on it, with gpt-5.6-luna as the recommended replacement. So are gpt-4-turbo, gpt-4o-2024-05-13, o1, o3-mini and o4-mini. The page says OpenAI emails customers who are “actively using the model”. A fallback route that never fires isn’t active use, so I wouldn’t count on that email arriving for route 3.

Four checks, cheapest first:

  1. Search the config. Grep every route policy, env file and eval baseline for the IDs on the 23 October list. Fine-tunes of the listed base models are on the page too.
  2. Ask the API. OpenAI’s GET /models “lists the currently available models”, and the reference shows an optional shutdown_date on each one. A CI job can fail when a route ID is missing from the list or has a date set. I haven’t checked how reliably that field is populated.
  3. Do the same for Anthropic. The model deprecations page has claude-sonnet-4-5-20250929 retiring on 30 November 2026 and says plainly that “requests to retired models will fail”. It also describes a usage export, by API key and model, from the Console.
  4. Run a route canary. Once a day, send one tiny real request down every route, fallbacks included, with each route’s real key. This is the only check that also catches an expired key, an empty credit balance and a paused organisation on a provider you rarely call. It costs a few billed requests.

Then swap route 3 for the replacement and run the receipt eval set against it before the date. A replacement that has never seen your schema is a new route.

What I’d do on Monday

  1. Grep for the 23 October IDs in every route, fallback and eval config, and replace them this week.
  2. Add a scope to your error type: model, provider or account, separate from whether to retry.
  3. Classify from the body. Handle Anthropic’s enforced_spend_limit_reached and self-set limit 400, and OpenAI’s spend and credit 429 codes, as account failures.
  4. Make account failures skip the provider and page, and confirm the person paged can reach the billing console.
  5. Start a daily route canary over every route with its real credentials.
  6. Check gateway behaviour: if you’re on Cloudflare AI Gateway, how your code treats 401 with 2009 and the Unified Billing 503. If you’re on LiteLLM, whether your version checks budgets on fallback targets.

Limitations

This is design analysis. The code is a sketch and hasn’t been run, and the classifier covers only the error shapes I found in the docs on 13 October. Those docs will change.

The bigger gap is that I don’t know what either Claude incident looked like on the wire. Anthropic’s notes say “refused requests” and “insufficient credit balance” and nothing more, so the two rows that started this post rest on guesses about status codes. I also don’t know what a retired OpenAI model ID returns, and I’ll find out with everyone else on the 23rd. The 3 September row comes from a secondary write-up. And a post telling you to page a human about billing has no evidence that the human will be awake.

Sources


  1. 12 min read

    More code, same software: review is the bottleneck

    A Harvard study of 718 firms finds coding agents raise code volume but not shipped output, because review became the constraint. What I'd copy from finance ops.


  2. 13 min read

    Sonnet 5.5 ends forced tool use. Here's how to keep structured output reliable

    Sonnet 5.5 rejects forced tool_choice with a 400; Haiku 5.5 still accepts it. How to keep extraction reliable with auto plus strict tools, a retry and an eval.


  3. 12 min read

    The model gateway: routing, timeouts, retries and what they cost

    Product code should make one call with one contract. A thin gateway underneath owns routing, timeouts, retries, fallbacks and the log of what each call cost.