LLM Service Outages: Client-Side Retries and Architecture Fallbacks

1. Turning a “three outages” incident into engineering signals

A typical LLM service incident may unfold like this: the model endpoint starts returning overload errors; web chat, public APIs, coding assistant tools, and collaboration workspaces are affected at the same time; only a small number of independently deployed console services remain available. The provider says the root cause has been identified about 20 minutes after the alert, but then enters a cycle of recovery and renewed failure over several hours. After midnight, the authentication path fails for several minutes, and another recurrence follows in the early morning. During the outage, users report that the status page is green while settings, usage, and skill/plugin management pages are still unusable, and requests continue to fail.

For developers, this timeline contains four signals that must be handled separately. Treating them as one event is the starting point for most misdiagnoses.

1. Different error semantics: 429 means “you are sending requests too fast”; it is a client quota and rate problem. Some providers use custom 5xx overload codes to mean “our capacity is insufficient”; this is a server-side admission and capacity problem. The required responses are almost opposite: for 429, slow down; for overload errors, back off, trip the circuit breaker, and prepare fallback.
2. Different failure surfaces: the inference plane (token generation), authentication plane (issuing and validating credentials), and control plane (usage statistics, configuration, and plugin manifests) are independent paths. If they fail together upstream, they do not necessarily recover together. If any one path remains weak, users still experience the service as unavailable.
3. Different definitions of recovery: the provider’s “recovery” is usually based on internal core metrics, while client experience depends on end-to-end reachability. A green status page is a weak signal, not a release condition.
4. Different blast radius: if a single incident takes down model, login, and console together, the upstream likely lacks bulkhead isolation. Among affected customers, the largest losses often come from teams that directly connect a nine-hour unattended pipeline to a single synchronous call: without checkpoints, there is no recovery, only reruns.

The conclusion is direct: upstream availability is not a variable you control. What you can control is how far your system degrades when upstream degrades. That is an architecture design problem.

2. Client side: make retries from causing secondary damage

A common secondary failure during an outage is a retry storm. When the server slows down, clients increase retry frequency at the same time, queues grow, and overload reinforces itself. Retries must therefore be disciplined.

First, route retries by error code. Only idempotent requests that are safe to replay should be retried. Parameter errors and context-length errors in 4xx responses, except 429, are deterministic failures. Retrying them ten thousand times will produce the same result, so they must fail fast and raise an alert.

Second, combine exponential backoff with random jitter (full jitter). Jitter spreads retry phases so clients do not all hit the upstream at the same millisecond.

import random, time
RETRYABLE = {429, 500, 502, 503, 504}
IDEMPOTENT = {"chat", "embeddings", "classify"}  # The application decides whether the operation can be replayed.
def call_with_backoff(fn, *, deadline_budget=30.0, max_attempts=6,
                       base=0.5, cap=8.0, per_call_timeout=10.0):
    end = time.monotonic() + deadline_budget      # Use a global time budget instead of accumulating per attempt.
    last = None
    for attempt in range(max_attempts):
        remaining = end - time.monotonic()
        if remaining <= 0:
            raise TimeoutError("retry budget exhausted")
        try:
            return fn(timeout=min(per_call_timeout, remaining))
        except UpstreamError as e:
            last = e
            if getattr(e, "status", None) not in RETRYABLE:
                raise                               # Deterministic error: re-raise immediately.
            if attempt == max_attempts - 1:
                break
            backoff = min(cap, base * (2 ** attempt))
            time.sleep(random.uniform(0, backoff))  # Full jitter.
    raise last

Third, set a retry budget and a circuit breaker. Beyond a maximum retry count, limit the proportion of downstream requests caused by retries within a unit of time. For example, global retry volume should not exceed 10% to 20% of normal traffic. Once exceeded, only fallback paths should run. When consecutive failures reach a threshold, open the circuit, fail fast, and periodically half-open a probe to avoid exhausting threads and connections with slow requests. The most common avalanche during an outage is not the upstream refusing service; it is the client exhausting its own connection pool.

class Breaker:
    def __init__(self, fail_th=20, cool_down=30.0):
        self.fail_th, self.cool_down = fail_th, cool_down
        self.fails, self.opened_at = 0, 0.0
    def allow(self):
        if self.opened_at == 0:
            return True
        if time.monotonic() - self.opened_at >= self.cool_down:  # Half-open: allow one probe.
            self.opened_at = 0
            return True
        return False
    def on_result(self, ok):
        if ok:
            self.fails, self.opened_at = 0, 0
        else:
            self.fails += 1
            if self.fails >= self.fail_th:
                self.opened_at = time.monotonic()

Fourth, propagate timeout budgets downward. When upstream latency rises, a client-side long timeout, such as 600 seconds, will pile up slow requests. The correct approach is to provide only a total budget at the entry point and reduce timeouts layer by layer based on remaining time, so that “no one waits for late results.”

3. Long-running jobs: from synchronous calls to resumable jobs

A nine-hour pipeline being wasted by one outage is an architecture problem, not merely an upstream problem. Any task that runs for more than a few minutes and contains multiple model calls should be converted into a persistent, sharded job:

# Pseudocode: sharding + checkpoint resume + idempotent writes
for idx, chunk in enumerate(chunks):
    if store.exists(f"job/{job_id}/{idx}.json"):
        continue                                  # Skip completed chunks.
    result = call_with_backoff(lambda: model.invoke(chunk, timeout=60))
    store.put(f"job/{job_id}/{idx}.json", result)  # Write intermediate results to object storage, not memory.
    state.mark_done(job_id, idx)                   # Persist progress externally so the process can resume after restart.

The engineering requirements include: writing intermediate results to persistent storage rather than memory so that an interruption loses only the current chunk; using a message queue for task distribution, because queues naturally provide peak smoothing, redelivery, and dead-letter isolation; designing shard keys to be idempotent so reruns do not create duplicate side effects; and converting long streaming requests into “persist segments and reconnect,” so a network flap or upstream reset only requires replaying the unfinished segment.

4. Platform side: if you own this layer

If you build your own gateway or aggregation layer, common capabilities can be combined into a protection system. Container platforms can provide autoscaling and rolling deployments, but they must leave connection and thread headroom for model calls so that scaling does not lag behind traffic. The gateway layer can apply layered rate limiting and priority queues, prioritizing interactive requests over batch offline jobs. For overloaded requests, apply load shedding: it is better to reject low-priority traffic than to let all requests slow down. Set bulkhead isolation for upstream dependencies: different models, tenants, and business lines should have separate connection pools and concurrency limits, so degradation in one place does not spread.

At the same time, distinguish degradation from failure. A practical degradation sequence is: switch model/endpoint → switch to a smaller model → return a cached or retrieval-based approximate result → accept the task and reply asynchronously → fail explicitly and provide a retry time. Degradation should be driven by switches and policy configuration, not by engineers manually changing code late at night.

5. Observability: do not run operations by following the provider status page

Relying only on the provider’s status page outsources your alerting decision. Build multi-dimensional synthetic probes instead. Cover authentication, list/metadata endpoints, non-streaming short requests, streaming requests, and control-plane pages. Measure success rate and latency independently for each dimension. A “recovery” judgment should require multiple consecutive periods, multiple regions, and all dimensions passing, rather than a single successful probe.

In metrics, distinguish upstream response-code success rate from end-to-end business success rate. The former can hide failures such as “the upstream is normal but our side times out.” Set SLOs and error budgets for each dependency. When more than half of the error budget is consumed, freeze nonessential releases; when it is exhausted, force a capacity and architecture review. Also, treat third-party status pages as an input source in your alerting pipeline, with multi-source cross-validation, rather than as the conclusion.

6. Multi-model abstraction layer: hidden costs beyond routing

Using multiple model capabilities for failover is a common practice, but the differences should be acknowledged in advance. A unified abstraction layer can cover common patterns such as chat, embeddings, and classification, but it cannot smooth away function-calling formats, multimodal input constraints, context-window and output-length limits, tokenization and billing differences, or streaming event structures without cost. A practical compromise is to converge prompt templates and output parsing into an adapter layer that is tightly coupled to the model. Routing strategies should switch only within the same capability tier. Prepare regression test cases for cross-model switching and continuously verify prompt drift across models in CI. Use different routing strategies for cost-sensitive batch tasks and latency-sensitive interactive tasks.

When implementing this on the cloud, you can combine container platforms for stateless gateways and workers, message queues for peak smoothing and redelivery, object storage for intermediate results, logging services for unified call-chain collection, monitoring and alerting for multi-dimensional probes and error-budget rules, and configuration-driven switches for degradation policies. These are generic capability categories and do not depend on any specific provider’s proprietary interfaces.

7. Emergency checklist

• During an outage, have automatic retry storms been stopped (are circuit breakers and retry budgets in effect)?
• Can long-running jobs resume from the most recent checkpoint instead of rerunning from zero?
• Are fallback models prepared by capability tier, and have their output usability been verified?
• Do probes cover authentication and the control plane, rather than only testing one generation request?
• Have the release criteria and verification order for “partial recovery” been defined?
• Afterward, have error budgets, root-cause records, and degradation playbooks been updated?

Occasional upstream unavailability is a fixed reality that cannot be promised away. What truly determines the cost of an incident is whether your system can turn “everything is wasted” into “some tasks are delayed” during the hours when the upstream degrades.