Post-Mortem of a Prolonged AI Service Outage: Troubleshooting, Shared Dependencies, and Resilience

I. Event Overview

According to incident records, the failure began to manifest during the evening. The web interface, developer console, and command-line tools experienced widespread errors almost simultaneously, with error rates spiking in a short period. Subsequently, several APIs became unavailable. The core models experienced another large-scale error during the night, forcing the engineering team to re-enter emergency response mode. System status fluctuated between recovery and relapse several times before finally stabilizing late at night. A lightweight model also experienced anomalies shortly after. The entire incident lasted nearly ten hours, affecting the platform entry points, API services, and multiple models.

Post-incident reports indicated that one direct trigger was a sudden physical accident in an availability zone (AZ) within a regional data center. Equipment was damaged and forced into power isolation, and the affected services required several hours to recover. Other AZs in the same region remained operational, but rescheduling and reconnecting the affected resources took time.

The value of analyzing this incident lies not in a single point of failure, but in how it demonstrates three key aspects: how troubleshooting direction affects recovery speed, how shared dependencies cause multiple services to fail simultaneously, and how retries and traffic rebound extend the outage duration. The following sections expand on these three points.

II. Why Troubleshooting Goes Astray

Initial troubleshooting focused on the user login and logout paths, which is a classic example of symptom-level investigation. The login path is just one segment of the request chain. The actual point of failure could be located in the authentication service, quota system, scheduling layer, or model gateway. When the troubleshooting scope does not match the dependency relationships, teams will spend a significant amount of time on irrelevant components.

Building a dependency graph is an effective method to correct the direction. The specific steps are as follows:

1. List all services that each type of user request traverses, including the entry point, gateway, authentication, scheduling, and backend inference services.
2. Annotate the storage, message queues, configuration centers, and downstream APIs that each service depends on.
3. Calculate how many request paths share each component; prioritize troubleshooting the component with the highest sharing degree.

Highly shared components are often the primary suspects for a failure. In this incident, multiple entry points and multiple models experienced anomalies in quick succession, indicating that they likely shared certain underlying capabilities.

2.1 Causes of State Fluctuations After a Fix

A recovery followed by a relapse is typically caused by three types of issues. First, the fix was only applied to a subset of replicas, while other replicas remained in an abnormal state; the problem resurfaced when traffic was routed back. Second, error states remained in the cache and were reloaded after the service restarted. Third, client retries continuously accumulated during the outage, and the sudden traffic surge upon recovery caused the system to overload again.

Judgments should be based on metric curves rather than the service status at a single point in time. During troubleshooting, observe the error rate, queue depth, retry count, and the health of each replica simultaneously to confirm whether the curves have truly dropped and stabilized.

2.2 Implications of Multiple Models Failing Sequentially

When multiple models experience issues sequentially over time, it usually means they share a common cause. Common shared causes include the same authentication system, the same resource scheduling pool, the same network egress, or the same configuration release. A characteristic of common-cause failures is that the failure points may not overlap perfectly in time. Therefore, recovery requires verifying each dependency link one by one; you cannot assume that fixing one point will resolve all issues.

III. Failure Propagation Mechanisms

3.1 Shared Dependencies Amplifying Impact

Shared dependencies are the primary reason multiple services fail simultaneously. If an authentication service is called by all entry points, its increased latency will directly propagate to every entry point. Even if the backend inference capacity is intact, users will not be able to complete their requests. During design, you should differentiate between capabilities that must be shared and those that can be isolated, and establish independent capacity planning and degradation paths for shared components.

3.2 Retry Storms

Retries are a common method to improve availability, but poor design can amplify failures. Assuming a call chain contains n layers, and each layer retries up to r times upon failure, the bottom layer will receive approximately (r+1)^n times the original number of requests in the worst-case scenario. With a three-layer call chain where each layer retries twice, the pressure on the bottom layer can be amplified to 27 times the original.

A reasonable retry strategy should include the following elements:

• Only retry on retriable errors, such as timeouts and temporary unavailability; do not retry on parameter errors.
• Use exponential backoff, where the wait time doubles each time, with a maximum cap.
• Add random jitter to prevent a large number of clients from retrying simultaneously at the exact same moment.
• Centralize retries at one layer of the chain to avoid every layer retrying.

The following example demonstrates an implementation of exponential backoff with random jitter:

import random
import time
class TransientError(Exception):
    """Represents a retriable transient error."""
def call_with_backoff(func, max_retries=3, base=0.5, cap=8.0):
    """Exponential backoff with random jitter to prevent clients from retrying simultaneously."""
    for attempt in range(max_retries + 1):
        try:
            return func()
        except TransientError:
            if attempt == max_retries:
                raise
            delay = min(cap, base * (2 ** attempt))
            time.sleep(random.uniform(0, delay))  # Full jitter

3.3 Coupling of Control Plane and Data Plane

The control plane is responsible for configuration distribution, routing adjustments, and resource allocation, while the data plane handles the actual processing of requests. When they are coupled, anomalies in the control plane will drag down the data plane. For example, if the configuration center is unavailable and the data plane reads the configuration on every request, the entire system will fail. The solution is to have the data plane cache the last valid configuration and continue operating based on that cache when the control plane fails.

IV. Multi-AZ and Regional Disaster Recovery

4.1 N+1 Capacity Planning

The prerequisite for multi-AZ deployment is that the remaining AZs can absorb the load of the failed AZ. Assume there are n AZs, and the total load is L. Under normal circumstances, each AZ carries L/n. After losing one AZ, the remaining n-1 AZs must each carry L/(n-1). Therefore, the capacity of each AZ should be at least L/(n-1), and the total capacity should be at least n·L/(n-1).

Taking three AZs as an example, the total capacity must reach 1.5 times the peak load to maintain service after losing one AZ. With two AZs, the total capacity needs to be 2 times the peak load.

Capacity reservation is costly, but it is the most reliable guarantee. If you wait to scale out after a failure occurs, the scaling process itself takes time and may fail due to resource constraints.

4.2 Defining Recovery Objectives

Disaster recovery design should first define two metrics:

• RPO (Recovery Point Objective): The maximum acceptable amount of data loss, measured in time. For example, an RPO of 5 minutes means that at most 5 minutes of recent data can be lost during a failure.
• RTO (Recovery Time Objective): The longest acceptable time from the occurrence of the failure to service recovery. For example, an RTO of 30 minutes means the service must be restored within 30 minutes.

Once the metrics are determined, you can decide on the synchronization method for replicas, the failover mechanism, and the frequency of drills. Setting metrics too loosely will mask risks, while setting them too strictly will significantly increase costs.

4.3 Data Replicas and Consistency

Cross-AZ data replication is generally divided into synchronous replication and asynchronous replication. Synchronous replication ensures data consistency but incurs higher write latency and can reduce availability during cross-region network jitter. Asynchronous replication has lower latency but may lose unsynchronized data during a failover. The choice between the two depends on the business requirements for consistency and latency.