The model gateway: routing, timeouts, retries and what they cost

My rule for LLM features is that product code calls callModel(workflow, input) and gets back a typed result or a typed failure. It never names a model, a provider or a retry count. Everything else lives in a thin layer underneath, which I call the model gateway.

Update 13 October 2026: I’ve added a fourth error class, account failures, after two Claude incidents this month refused requests because of billing state. I’ve also added a check for retired model IDs in fallback chains (OpenAI retires a batch of snapshots on 23 October) and a note on cost per success from Cloudflare’s router launch. The argument hasn’t changed.

Without that layer, the first feature calls a provider SDK from a route handler and the second copies the first. By the fifth you have five timeout values, three retry loops written from memory and a model name hard-coded in places nobody remembers. Then a provider has a bad afternoon and every feature fails in its own way. LLMs make this edge worse than a normal flaky dependency: a “successful” response can be the wrong shape, a retry can repeat a side effect, and every attempt is billed.

This post is design analysis with illustrative code. I haven’t run the code, and nothing here is a measured result.

The running example is a support-ticket assistant. When a ticket arrives it (1) classifies it, (2) extracts structured fields like order ID and product, (3) drafts a reply for a person to approve, and (4) runs an agent step that may call a tool to create a follow-up task.

Routing by workflow

Routing usually happens by vibes: someone liked a model in a playground and it now serves everything. When I’ve evaluated providers and routed workflows to them in product work, the useful move was to name the one or two properties a workflow can’t compromise on, then pick the cheapest option that meets them on my own evaluation set. For the ticket assistant (the choices are illustrative):

Structured-output reliability gets its own column because it’s the property most likely to break silently when you swap models. The policy lives in one place, keyed by workflow name, where a change can be reviewed. The gateway itself stays thin: it picks a model, calls it within a budget, handles failure, validates the shape and writes down what happened.

Cost per success is the number to route on. On 30 September Cloudflare launched an Auto Router for its AI Gateway, and the launch post reports cost per successful task instead of price per token. On Cloudflare’s own 97-task benchmark (three samples per task), the router scored 86.6% at $0.0084 per success, against 96.6% and $0.0210 for Claude Opus 5.5 and 84.2% and $0.0108 for GPT-6 Sol. Those are a vendor’s numbers on a benchmark I can’t inspect, and I haven’t used the product. What I take from it is the unit, and that the cheaper line in that table fails ten points more often. The post also says the router “applies a switching penalty that grows with the number of tokens already in context”, because another model has to rewrite the cached prompt. A fallback pays the same cost, which the first version of this post ignored.

Timeouts

A streamed generation can be slow to start, or start quickly and run long, so it gets two limits. The time-to-first-token (TTFT) timeout is tight: if nothing has arrived after N seconds the request is probably queued or stuck, so try elsewhere. The total timeout is generous, because a steadily streaming reply is healthy, and it exists to cap cost and stop runaway generations. Non-streaming calls only get the total, which suits a classifier. Anthropic’s docs recommend streaming for long-running requests and note that networks can drop idle connections during long non-streaming calls.

An error can also arrive after the HTTP status was already 200. Anthropic documents error events mid-stream, for example an overloaded_error that would have been a 529 on a non-streaming call. The gateway has to treat a stream that errors or stalls halfway as a failure.

In an agent, per-call timeouts aren’t enough. If the ticket assistant runs four steps, each with a 30-second timeout and two retries, the worst case is several minutes and nobody chose that number. So the run gets a deadline and each call uses the smaller of its own timeout and what’s left. When the budget is gone, later steps don’t start, and the agent returns a partial result or hands off to a person. One AbortSignal passed down from the run stops every in-flight call on cancel.

Retries and account failures

A retry loop is easy to write and hard to write safely. The provider docs are fairly clear on which signals are retryable.

The first version of this post sorted errors into retryable, fatal and timeout. On 7 October Anthropic’s status page gave me a fourth:

We have identified an issue which resulted in some organizations to be incorrectly paused for having reached their spend limit, resulting in refused requests on the Claude API, Claude.ai, Claude Code, Claude Security, and Claude Cowork. This is now resolved.

Claude status, “Issue with spend limits”

Six days earlier, a delay in applying purchased credits had “some requests failing for insufficient credit balance”. Neither note gives a root cause, a duration for the spend-limit pause, or what the refused requests looked like on the wire. I’d guess the paused organisations saw the spend-limit 429 from the table above, but Anthropic haven’t said so.

What I do know is the shape. An account failure is about your organisation’s standing with the provider, so it hits every model you call there at once. Retrying won’t help, and nor will falling back from one Claude model to another. The useful moves are to skip every remaining route on that provider, go to another provider or the degraded mode, and page someone, because the fix may be in a billing console.

Three other rules:

• One retry layer. The Anthropic SDKs already retry transient failures twice by default. Add three gateway retries and one logical call can become nine attempts. I set the SDK’s maxRetries to 0.
• Full jitter. Wait a random amount between zero and base * 2^attempt, capped, with retry-after as the minimum. Without jitter, every client that failed together retries together.
• A retry budget. Cap retries across the process, say as a fraction of recent first attempts, so that a real outage fails fast or goes straight to fallback.

Tool calls

Retrying a classification is harmless: you pay twice and get a label. Step 4 of the ticket assistant is different. The model calls create_follow_up_task, your code creates the task, and then the next model call times out. A naive gateway retries the whole step, the model decides again, and now there are two tasks.

So the gateway retries model calls, never tool executions. Every tool call gets a stable idempotency key (run ID, step number, tool name) and the tool rejects duplicates in software. The agent runner persists the tool result before the next model call and replays it on retry. In reconciliation work I learned that a duplicate posting is worse than a failed one: a failure is visible, and a duplicate looks like data.

Fallbacks and breakers

A fallback model hasn’t been through the same testing as the primary. It may follow your JSON schema less reliably, handle tool definitions differently, or pick different labels at the edges of your set. So every response from every route goes through the same schema check, and for extraction I’d only list fallback models that have passed the same eval set against the same schema.

A fallback route also has to still exist. OpenAI’s deprecations page lists a batch of legacy snapshots shutting down on 23 October 2026, from a notice dated 22 April, including gpt-4-turbo, gpt-4o-2024-05-13, o1, o3-mini and o4-mini. A retired ID in a primary route gets noticed on the day. In a fallback route it sits there until the outage it was meant for. I haven’t checked what error a retired ID returns, so I’d check the IDs against the deprecation pages in CI.

Sometimes the right fallback isn’t a model. Classification falls back to keyword rules and a “needs triage” label, the draft reply to a template, extraction to empty fields and a flag, and the agent step to a suggestion for a person. Write the degraded mode in the same policy entry as the routes, or the one you get is a 500.

A circuit breaker handles the pattern across requests: open past an error threshold so calls skip the route for a cool-down, then let a few through. Scope it to the route, since Anthropic applies rate limits separately per model, and keep a second, provider-wide breaker for account failures. Only count availability signals. Your own 400s and schema failures shouldn’t open anything.

One record per call

The gateway isn’t finished until you can answer “what did the ticket classifier cost yesterday, and how often did it fall back?”

type ModelCallRecord = {
  callId: string;
  runId?: string;              // ties agent steps together
  workflow: WorkflowName;      // "ticket.classify", ...
  routeUsed: string;           // "providerA:small-model"
  fallbackUsed: boolean;
  degraded: boolean;
  attempts: Array<{
    route: string;
    outcome: "ok" | "retryable_error" | "account_error" | "fatal_error" | "timeout" | "invalid_output";
    httpStatus?: number;
    providerRequestId?: string;
    latencyMs: number;
    ttftMs?: number;
  }>;
  retryCount: number;
  latencyMs: number;
  inputTokens?: number;
  outputTokens?: number;
  costUsd?: number;            // summed over attempts
  schemaValid: boolean | null; // null for free text
};

Anthropic returns a request-id header on every response, and you’ll want it for support tickets. Sum cost over failed attempts too, because a call that timed out mid-stream may still have been billed. Keep schemaValid separate from outcome: a 200 with bad JSON is a quality problem. Cost per success for a workflow is then costUsd over all calls, divided by the calls that came back valid.

The gateway sketch

It leaves out streaming, the breakers and the retry budget. The adapters and helpers are stubs.

type Route = { provider: "a" | "b"; model: string };

type Policy<I, O> = {
  routes: Route[];                  // primary first, then fallbacks
  timeoutMs: number;
  maxRetriesPerRoute: number;
  schema?: z.ZodType<O>;
  buildRequest: (input: I) => ModelRequest;
  degrade?: (input: I) => O;
};

class ModelError extends Error {
  constructor(
    public kind: "retryable" | "account" | "fatal" | "timeout",
    public status?: number,
    public retryAfterMs?: number,
  ) { super(kind); }
}

export async function callModel<I, O>(
  workflow: WorkflowName,
  input: I,
  opts: { deadline?: number; signal?: AbortSignal } = {},
): Promise<{ ok: true; value: O } | { ok: false; reason: string }> {
  const policy = getPolicy<I, O>(workflow);
  const record = startRecord(workflow);
  const pausedProviders = new Set<Route["provider"]>();

  for (const route of policy.routes) {
    if (pausedProviders.has(route.provider)) continue;

    for (let attempt = 0; attempt <= policy.maxRetriesPerRoute; attempt++) {
      const remaining = (opts.deadline ?? Infinity) - Date.now();
      if (remaining <= 0) return finish(record, { ok: false, reason: "deadline" });

      const signal = AbortSignal.any([
        AbortSignal.timeout(Math.min(policy.timeoutMs, remaining)),
        ...(opts.signal ? [opts.signal] : []),
      ]);
      const started = Date.now();
      try {
        const res = await adapters[route.provider].send(
          route.model, policy.buildRequest(input), signal,
        ); // adapter maps provider errors to ModelError

        const parsed = policy.schema ? policy.schema.safeParse(res.output) : null;
        record.attempt(route, res, Date.now() - started, parsed?.success ?? null);

        if (parsed && !parsed.success) break;    // invalid shape: try the next route
        record.fallbackUsed = route !== policy.routes[0];
        return finish(record, { ok: true, value: (parsed ? parsed.data : res.output) as O });
      } catch (err) {
        if (opts.signal?.aborted) return finish(record, { ok: false, reason: "cancelled" });
        const e = err instanceof ModelError ? err
          : new ModelError(signal.aborted ? "timeout" : "retryable");
        record.attemptFailed(route, e, Date.now() - started);

        if (e.kind === "fatal") return finish(record, { ok: false, reason: `fatal:${e.status}` });
        if (e.kind === "account") {              // every model on this provider is out
          pausedProviders.add(route.provider);
          alertOnCall(route.provider, e);
          break;
        }
        if (attempt < policy.maxRetriesPerRoute) {
          record.retryCount++;
          await sleep(fullJitterMs(attempt, e.retryAfterMs));
        }
      }
    }
  }

  if (policy.degrade) {
    record.degraded = true;
    return finish(record, { ok: true, value: policy.degrade(input) });
  }
  return finish(record, { ok: false, reason: "all_routes_failed" });
}

Invalid output moves to the next route. A fatal error stops everything, because a 400 means the request is wrong. An account error skips the provider. Retries and fallbacks share one deadline. And the function knows nothing about tools: that lives in the agent runner above it.

What I’d do on Monday

1. List every call site, give each a workflow name, and put a callModel wrapper in front of them, even if it only forwards to the current SDK.
2. Turn off SDK-level retries in the wrapper so there’s exactly one retry layer.
3. Classify errors into retryable, account, fatal and timeout from your providers’ docs, including the spend-cap 429 that looks retryable and isn’t.
4. Write the per-call record before you tune anything, and compute cost per success per workflow from it.
5. Validate the schema on every structured workflow, on every route, and write down a degraded mode for each.
6. Check every model ID in every route against the providers’ deprecation pages, in CI, starting with anything on OpenAI’s 23 October list.
7. Audit side-effecting tools for idempotency keys before any agent loop retries anything. Breakers and the retry budget can wait for the logs.

Limitations

There are no benchmarks here. The routing table is illustrative, and the code is a sketch that hasn’t been run. It leaves out streaming, breakers, retry budgets, cost calculation and the provider adapters, which is where much of the real work is. Prompt caching gets one sentence here and deserves a post. Batching, self-hosted models and data-residency routing are out of scope.

Provider behaviour changes, so recheck the docs I’ve cited. The Cloudflare figures are Cloudflare’s. And I don’t know how the wrongly paused organisations’ requests were refused, which is an awkward gap in a post about classifying errors.

Sources

• Claude API errors
• Claude API rate limits
• Claude streaming messages: error events
• OpenAI API error codes
• Claude status: issue with spend limits (7 October 2026)
• Claude status: delayed credits on the Claude Platform (1 October 2026)
• OpenAI API deprecations
• Cloudflare: Auto Router for AI Gateway (30 September 2026)