Capacity Governance for LLM Services: Lessons from a Traffic Surge Event

1. Incident Background: A “Happy Problem”

Following the release of a new model version, an LLM platform experienced a sudden influx of traffic and new users that far exceeded expectations. Compute resources were rapidly depleted, forcing the platform to urgently close the new user subscription channel and issue a late-night status announcement. The announcement clarified several key points: legacy plans were not removed, existing users could continue using the service and maintain auto-renewals, users who manually disabled auto-renewal could still renew via the interface, and existing users could upgrade their plans. However, new plans were temporarily unavailable for purchase due to compute constraints, with availability to be determined.

From an engineering perspective, this was not a post-mortem of a “crash,” but rather a classic case of capacity planning mismatch: the supply-side scaling speed could not keep up with the instantaneous burst on the demand side. The real takeaway is not about an unexpected spike in popularity, but how technical teams should design a supply system that is bufferable, observable, and reversible when business growth far outpaces resource reserves.

Notably, right before the release, the platform launched an invitation-based subscription campaign. This indicates that prior to the release, the team assessed the compute headroom as “relatively sufficient” and even hoped to consume idle resources through promotional activities. The sudden traffic peak directly pierced through the reserved buffer. This detail serves as a strong warning for capacity planners: operational activities and technical capacity must establish a strong linkage mechanism.

2. Capacity Planning: From Static Headroom to Dynamic Watermarks

2.1 Why Static Reservation Fails

The traditional approach is to reserve resources based on historical peaks multiplied by an empirical coefficient (e.g., 1.5x to 2x). However, the traffic characteristics of LLM services differ significantly from standard web services:

• High per-request resource consumption: A single inference request can occupy a substantial amount of accelerator card VRAM and compute time. As concurrency rises, resource consumption grows linearly or even super-linearly.
• Long-tail request duration distribution: Long-context and long-output scenarios significantly extend the duration a single request holds resources, amplifying queueing effects.
• High traffic burstiness: Version releases, viral social media propagation, and marketing campaigns can generate peak traffic several times higher than daily averages within hours.

Therefore, static reserved headroom is almost guaranteed to fail in “viral upon release” scenarios. A more robust approach is to establish a tiered watermark model:

Total Capacity = Baseline Guarantee Capacity + Elastic Scaling Capacity + Peak Buffer Capacity
Baseline Guarantee Capacity: Covers daily P50 traffic, long-term resident
Elastic Scaling Capacity: Covers P90 traffic, auto-scales per minute-level metrics
Peak Buffer Capacity: Covers release/campaign scenarios, requires advance reservation and locking

2.2 Key Metrics for Watermark Assessment

It is recommended to monitor at least the following dimensions and incorporate them into the pre-release “capacity review” checklist:

Metric Category	Example Metrics	Purpose
Resource Layer	Accelerator utilization, VRAM usage, queue depth	Identify physical bottlenecks
Service Layer	Time to first token (TTFT), end-to-end latency, timeout rate	Identify user experience inflection points
Business Layer	New subscriptions, active sessions, concurrent requests	Correlate business actions with capacity
Cost Layer	Compute cost per request, elastic scaling cost	Evaluate scaling economics

Among these, queue depth is often the most sensitive leading indicator. When the queue begins to continuously back up, it indicates that effective throughput is nearing its limit. Initiating scaling at this point is often dozens of minutes too late.

2.3 Operational Campaigns Must Pass Capacity Review

Returning to the invitation campaign in the incident: if the marketing team plans to launch a user-acquisition campaign that could bring in incremental users, the technical team should be informed in advance to complete a capacity assessment. Establish the following workflow:

1. Marketing submits expected campaign increments (new users, peak concurrency, duration).
2. The technical team translates this into resource requirements, checking the elastic scaling limits and peak buffer margins.
3. Both parties confirm the “capacity redline” and agree on thresholds for triggering degradation or pausing the campaign.
4. During the campaign, compare actual traffic against estimates in real-time; if deviation exceeds the threshold, initiate the contingency plan.

This workflow is not complex, but it transforms the relationship between “campaigns” and “capacity” from disconnected silos into an integrated chain.

3. Subscription and Admission: Designing the “Switch” as an Operational Capability

The core measure taken by the platform in this incident was suspending new user subscriptions—essentially a form of admission rate limiting. While this action seems simple, it requires a complete set of supporting engineering capabilities.

3.1 Tiered Design for Admission Control

It is recommended to split admission control into three independently togglable layers:

• Registration Layer: Whether new account creation is allowed (coarsest granularity).
• Subscription Layer: Whether new users can purchase plans (the measure used in this incident).
• Request Layer: Concurrency and rate limits per user/tenant (finest granularity).

The independence of these three switches means the team can choose the most appropriate braking method based on the source of the pressure. For example, if compute is tight but has some margin, they can close only the subscription layer while keeping the registration layer open, allowing users to experience free quotas first. If even free quotas are hard to guarantee, rate limiting at the request layer should be tightened.

3.2 Continuity Guarantees for Existing Users

The announcement specifically emphasized that “legacy plans were not removed, current users can use the service normally, and auto-renewals are supported.” This is a critical engineering decision. From a system design perspective, it implies:

• Plan configuration must be versioned: Legacy plans are retained as independent configuration items and cannot be overwritten by the launch of new plans.
• The renewal process must be idempotent: If a user manually disables auto-renewal and later renews, the state transition must be predictable, avoiding a deadlock where “once closed, it cannot be reopened.”
• The upgrade path must be reversible: When existing users upgrade to a new plan, the original plan information should be retained for rollback or comparison.

These details directly impact user trust. Compute scarcity is understandable, but “collateral damage to existing users” will rapidly consume brand equity.

3.3 Engineering Essentials for Rate Limiting and Degradation

When compute approaches its limit, in addition to admission control, request-side degradation measures should be prepared:

Degradation Strategy Priority (from light to heavy):
1. Reduce call frequency for non-core features (e.g., historical session summaries, recommendation requests)
2. Set shorter timeouts and smaller batch processing windows for low-priority queues
3. Tighten rate limits for free-tier users to guarantee SLA for paying users
4. Queue + friendly prompts, instead of throwing direct errors
5. Suspend new subscriptions (Admission Layer)

The key principle is: degradation must be orderly, observable, and recoverable. Every degradation action should log its trigger conditions and impact scope. Recovery should proceed in the reverse order of degradation, gradually opening up to avoid secondary shocks caused by a sudden “all-or-nothing” release.

4. Release Cadence: Gradual Rollout is More Than “Batch Release”

The trigger for this incident was the traffic peak brought by the version release. For model services, the release cadence is itself a part of capacity governance.

4.1 Pre-release Capacity Drill

It is recommended to complete a “capacity drill” before the official release, including:

• Simulating the post-release request distribution with load testing traffic to verify if elastic scaling can deliver resources within the target timeframe.
• Verifying the cold-start latency of the scaling chain (image pulling, model loading, service pre-warming), which is often the true bottleneck of scaling speed.
• Ensuring monitoring alert thresholds and on-call response mechanisms are in place.

4.2 Gradual Rollout and Traffic Allocation

Gradual rollout should not only focus on “whether the new version is stable” but also on “the traffic increment brought by the new version.” It is recommended to gradually release traffic according to ratios and observe queue depth and latency inflection points at each stage:

Gradual rollout stages example: 1% -> 5% -> 20% -> 50% -> 100%
Observation window per stage: >= 30 minutes
Promotion criteria: Queue depth stable, P99 latency below threshold, error rate shows no abnormal increase
Rollback criteria: Any core metric worsens for two consecutive observation windows

4.3 The “Staggering” Principle for Releases and Campaigns

One of the most important lessons from the incident is that the marketing campaign and the version release occurred almost back-to-back. It is recommended to explicitly state in the release management guidelines: do not stack marketing campaigns that could bring significant incremental traffic within a certain window before or after a major version release. If stacking is absolutely necessary, a capacity review must be conducted and peak buffer reserved.

5. Observability and Incident Response: Engineering Capability Behind Late-Night Announcements

Issuing a late-night announcement indicates that the team completed the “discover problem - assess impact - formulate strategy - communicate externally” chain in the first moments. The efficient operation of this chain relies on two things: observability and incident response plans.

For observability, it is recommended to at least achieve:

• Unified collection of resource, service, and business layer metrics, displayed correlatively on a single dashboard.
• A dedicated observation view for the “24 hours post-release,” highlighting the dynamic changes between new user count, concurrency, queue depth, and latency.
• Tiered alerting: Resource watermark alerts trigger scaling, experience metric alerts trigger degradation assessments, and business metric anomalies trigger marketing synchronization.

For incident response, it is recommended to pre-write a playbook for “compute scarcity” scenarios, clarifying:

1. Who has the authority to trigger admission layer switches.
2. The order of degradation and recovery.
3. External announcement templates and information sync channels.
4. Continuity guarantees for existing users (which commitments must be upheld).

Documenting and drilling these contents ensures calm decision-making at 2:00 AM.

6. Actionable Checklist for Technical Teams

Synthesizing the above analysis, the response experience for a compute scarcity incident can be crystallized into the following checklist:

• Capacity Side: Establish a tiered watermark model, complete pre-release capacity drills, and verify scaling cold-start latency.
• Operations Side: All acquisition/promotional campaigns must go through capacity review, with clear incremental estimates and redline thresholds.
• Admission Side: Registration, subscription, and request layer switches must be independently controllable; existing user continuity must be guaranteed.
• Degradation Side: Degradation strategies must be graded and orderly; both triggering and recovery must be logged and observable.
• Release Side: Standardize gradual rollout ratios and observation windows; stagger releases and marketing campaigns.
• Incident Response Side: Document and drill the playbook; clarify announcement templates and decision-making authority in advance.

Compute scarcity itself is not terrifying; it is a concomitant phenomenon of rapid business growth. What truly determines user experience and brand reputation is whether the team can connect “capacity - admission - degradation - release - incident response” into a closed loop, turning every traffic peak into an opportunity to harden the system rather than a cycle of passive firefighting.