Navigating LLM Compute Shortages: Capacity Planning, Degradation, and Business Strategy

1. The Phenomenon: A Typical “Compute Shortage” Incident

Recent industry discussions have highlighted a representative scenario: shortly after releasing a new generation model, an AI assistant product experienced a traffic peak. During peak hours, the interface alerted users to a “compute shortage, automatically switching to fast mode.” Instead of issuing a templated apology, the operations team candidly replied, “We are adding more compute capacity; in the meantime, feel free to use other similar services.”

Beyond the specifics of this case, the phenomenon encapsulates the core contradiction in operating LLM services: the speed of compute supply for inference cannot keep pace with model capability upgrades and user demand growth. This article unfolds along three dimensions: why compute shortages occur (technical root causes), how to scientifically plan and defend capacity (engineering methods), and the business logic behind public statements made during compute crunches (decision-making considerations).

2. Technical Root Causes: Why LLM Inference is Compute-Hungry

2.1 Compute Characteristics of Autoregressive Generation

Unlike traditional request-response services, LLM inference is autoregressive. Generating each token requires a forward pass, and the Key-Value (KV) Cache for historical context must reside in GPU memory. This means:

• The compute overhead per request is proportional to the output length. Long-text and code-generation tasks consume significantly more resources than short Q&As.
• Concurrent requests compete for memory bandwidth. As the KV Cache expands linearly with context length, the actual supportable concurrency drops significantly as typical conversations become longer.
• User-perceived “lag” is often not a network issue, but rather an increase in inference queue wait times.

2.2 Capability Upgrades Multiply Compute Demands

The evolution of new-generation models almost universally points toward higher compute consumption. Mixture of Experts (MoE) architectures introduce scheduling overhead while scaling parameters; multi-agent parallelism and tool-calling chains mean a single user request might trigger dozens of internal sub-task inferences; long-context and multimodal inputs further amplify per-request costs. For every increment of “smartness” a model gains, inference costs may multiply several times—a structural contradiction all LLM service providers must face.

2.3 Rigid Constraints on the Supply Side

High-end accelerator cards have long procurement cycles and high unit costs. Furthermore, compute resources are contested between training new models and online inference. When a new product launch brings a pulse of traffic, the “slow-moving variable” of supply cannot catch up with the “fast-moving variable” of demand, causing capacity gaps to erupt centrally.

3. Capacity Planning: Turning “Gut Feeling” into “Calculable Metrics”

3.1 Define Key Metrics First

The first step in capacity planning is not buying GPUs, but defining a metrics system:

• Throughput: Peak QPS, average input/output tokens per request.
• Latency: Time To First Token (TTFT), end-to-end P95/P99 latency.
• Resources: GPU utilization per instance, memory usage, queue depth.

3.2 A Practical Estimation Framework

In the absence of granular data, the following relationship can be used for a first-order estimation:

Concurrent Requests ≈ Peak QPS × Average Residence Time per Request (Queuing + Inference)
Required Instances ≈ ⌈(Peak QPS × Output Tokens per Request) ÷ Effective Throughput per Instance (Tokens/s)⌉ × Redundancy Factor

Illustrative estimation script:

def estimate_instances(peak_qps, avg_out_tokens, tokens_per_sec, headroom=1.5):
    """First-order capacity estimation: only for magnitude planning, must be calibrated with stress test data"""
    demand = peak_qps * avg_out_tokens   # Total tokens needed per second at peak
    raw = demand / tokens_per_sec        # Theoretical instance count
    return int(raw * headroom + 1)       # Redundancy factor covers traffic jitter and failure domains
# Example: Peak 50 QPS, average output 800 tokens, single instance throughput 800 Tokens/s
print(estimate_instances(50, 800, 800))

It must be emphasized: tokens_per_sec must come from full-chain stress testing under real business loads, not nominal peak compute. Batch processing throughput, online concurrent throughput, and memory-bandwidth-bound throughput can differ by several times.

3.3 Stress Testing and Peak Coefficients

• Apply gradient pressure to find the “inflection point load” of a single instance (where latency starts to degrade non-linearly), and set a safe operating level at around 70% of the inflection point.
• For predictable pulse scenarios like product launches or major events, peak coefficients should be significantly adjusted upwards, with capacity expanded in advance.
• Capacity planning must cover the “degraded state”: when the flagship model is unavailable, the fast model must be able to independently handle full traffic.

4. Engineering Responses During Peak Traffic: Degradation, Rate Limiting, and Elastic Scaling

4.1 Model Tiering and Automatic Degradation

The aforementioned “switched to fast mode” is a standard industry multi-tier model service strategy: services are divided into flagship, fast, and lightweight tiers. Traffic is normally split by configured ratios, but during peak hours, new traffic is automatically routed to lower-consuming tiers to prioritize the availability of core links. Illustrative configuration:

inference:
  tiers:
    - name: flagship        # Flagship tier: strongest capability, highest compute overhead
      share: 0.7
    - name: fast            # Fast tier: lightweight architecture, low latency and consumption
      share: 0.3
degradation:
  trigger:
    - gpu_util > 0.9 for 5m
    - queue_depth > 500
  action:
    - new_traffic_to: fast      # Route new traffic to the fast tier
    - long_task_defer: true     # Defer long tasks to an asynchronous queue

4.2 Rate Limiting and Priority Scheduling

• The gateway layer uses token bucket algorithms to rate limit by user/tenant dimension, preventing individual heavy calls from dragging down the global system.
• Introduce priority queues: subscription users and critical business requests are scheduled first; free traffic queues or is degraded.
• Reject “early and clearly”: rather than letting a request time out in the queue, it is better to quickly return an explicit rate-limiting response with retry suggestions.

4.3 Peak Shaving and Asynchronous Processing

Non-real-time tasks such as batch document processing, report generation, and offline analysis should enter message queues and be consumed by background compute during off-peak hours. Online services yield the peak window for real-time requests. Combined with object storage for intermediate results and logging services for full-chain tracing, limited online compute can be used where it matters most.

4.4 Elastic Scaling and Resource Pooling

• Orchestrate inference instances uniformly through a container platform, and automatically scale up or down based on monitoring and alerting metrics (GPU utilization, queue depth, P99 latency).
• Mix online inference with offline training and evaluation tasks, yielding compute to batch processing jobs during off-peak periods.
• Self-built data centers are suitable for stable baseline loads, while cloud elastic resources handle pulse peaks—a robust hybrid strategy. Taking Baidu AI Cloud as an example, its elastic compute resources, container services, message queues, and monitoring capabilities can be directly combined to build an inference foundation of “self-built baseline + elastic peak.”

4.5 Caching and Reuse

Prefix caching (reusing KV Cache for identical system prompts or long contexts) and semantic caching (directly hitting historical results for high-frequency similar questions) can significantly reduce redundant computation. They are among the most cost-effective optimization techniques available.

5. Business Considerations: Why Telling Users to “Try Competitors” Can Be a Rational Decision

When facing compute shortages, operators publicly suggesting users temporarily try other services is an “unconventional” move. However, there are robust business logics behind this approach.

5.1 Honesty is Better Crisis PR

During service anomalies, what users truly care about is certainty and respect. Templated “we are expediting processing” responses only exhaust patience. Conversely, stating “we are indeed short on compute, you can use alternatives first and come back when we are ready” sets clear expectations and offers an out. In an environment fatigued by “absolute superiority” marketing, honesty itself is a scarce brand narrative, capable of turning an incident into a positive brand impression.

5.2 Product Confidence Driving Retention

Daring to push users away implies a belief: users come for the product capability and will not leave due to temporary congestion. If operators lack confidence in their capabilities, the rational choice is to desperately hold onto every session; conversely, letting go signals to the capital market that “the product speaks for itself.” Compared to growth stories in marketing materials, this signal is far more persuasive.

5.3 Strategic Shift from User Acquisition to Product

A recurring industry lesson is that massive user acquisition spending without product strength to back it up wastes acquisition costs and generates negative word-of-mouth due to poor experiences. Shifting budgets from traffic buying to model R&D and compute infrastructure slows growth in the short term but is healthier long-term. An outage objectively serves as proof of “strong demand”—capacity maxing out shortly after a new release shows heavy usage is real—but the prerequisite is that capacity must be replenished quickly, otherwise goodwill will rapidly backlash into a trust crisis.

5.4 Outlook on Industry Landscape

This posture also implies a larger bet: the endgame of AI services is not a single super-app winner-takes-all, but multiple players coexisting and dividing tasks by scenario. Rather than consuming users with a terrible experience during congestion, it is better to candidly acknowledge the bottleneck and maintain the overall reputation of the AI category, competing on capability once capacity is restored. For users, what matters is never picking a side once, but which provider can consistently get the job done.

6. Implementation Checklist for Technical Teams

1. Build Metrics Before Expanding: TTFT, P99, queue depth, and GPU memory levels must have real-time monitoring and alerting.
2. Enable Tiered Services by Default: Route traffic via ratios across flagship/fast/lightweight tiers. Write degradation strategies into configurations rather than scrambling on the spot.
3. Normalize Stress Testing: Conduct full-chain stress tests before every model upgrade and major promotion to calibrate capacity models with measured throughput.
4. Hybrid Resource Strategy: Use self-built or long-term committed resources for baseline loads, and cloud elastic compute for pulse peaks to balance cost and certainty.
5. Cache First: Prefix caching and semantic caching offer low investment and high returns; they should be implemented first.
6. Transparent Communication Mechanism: Productize status pages, rate-limit notices, and recovery progress syncs. The speed and candor of incident communication directly impact brand equity.

7. Conclusion

Service congestion caused by compute shortages appears to be an operational incident on the surface, but at its core lies the structural contradiction between the leap in model capabilities and the pace of compute supply. Technically, it tests the systematic capabilities of capacity planning, degradation design, and elastic scaling. Commercially, it tests operators’ judgment regarding product strength, user relationships, and industry landscape. Treating capacity as a science to calculate and honesty as a strategy to deploy is the correct posture for navigating the compute cycle.