Understanding Token Quotas, Caching, and Pricing Behind LLM Service Capacity Crunches

Background: What “Pausing New Sales” Really Means for LLM Supply Constraints

When a major AI assistant platform recently announced it would temporarily stop accepting new memberships due to compute capacity constraints, the public discussion focused on “can I still buy a plan.” Behind that question lies a set of technical and commercial issues worth understanding for any developer:

• Why does LLM inference capacity become tight in a short period?
• What is the underlying logic of using “time windows + token quotas” for rate limiting?
• How do subscription plans and API pay-as-you-go pricing compare?
• What dimensions should users evaluate when choosing a model service?

This article does not target any specific product. Instead, it abstracts the scenario into a general engineering case, examining five areas: inference cost structure, quota system design, context caching mechanics, migration strategies for existing users, and a user-side evaluation framework.

Why Compute Capacity Becomes Scarce: Starting with Inference Cost Structure

1. Training Is One-Time; Inference Is Continuous

Training costs for large models are high but constitute a one-time investment that can be amortized over time. What grows relentlessly is inference cost—it rises linearly with every user request and scales in proportion to user growth. When user growth outpaces the procurement, delivery, data center deployment, and network validation cycle of accelerators (typically measured in months), a supply-demand gap emerges. That gap manifests externally as “compute shortage.”

2. Long Contexts Significantly Increase Per-Request Cost

Standard attention mechanisms scale quadratically with context length. Prefilling a 100K-token document is far more expensive than processing a 10K-token input. Meanwhile, KV Cache memory usage grows linearly with token count: for a medium-to-large model, the KV state per token is on the order of hundreds of kilobytes. A single request with a 100K-token context can occupy tens of gigabytes of GPU memory. The more long-context requests a system handles, the lower the concurrency per GPU and the lower the effective throughput.

3. RAG and Agentic Workflows Amplify Call Volume

What appears to the user as “one message” may translate into a dozen model calls on the platform side:

• Web search requires rewriting the query, then summarizing and filtering retrieved results.
• Long-document Q&A involves parsing, chunking, and vectorization.
• Agent workflows run multi-step loops.

The richer the value-added features, the greater the actual compute consumption behind a single message. This is also why many platforms tighten usage quotas when such features are heavily used.

Quota Systems: Understanding “Time Window + Token Quota” Design Logic

1. Why Token-Based Quotas Instead of Request Counts

Request costs vary drastically: a short Q&A with 10 tokens versus a long-document analysis with 100K tokens can differ by thousands of times in computational expense. A per-request quota would be exhausted by a small number of long-context requests. Token-based metering aligns more closely with real costs, which is why almost all major platforms use token quotas.

2. Why Time Windows Like “X Tokens per 5 Hours”

Compared to a monthly aggregate cap, a short time window serves a peak-shaving function: it prevents users from concentrating all their quota into a single high-demand hour and limits the space for scripted abuse. The shorter the window, the smoother the platform’s instantaneous load, and the more users the same compute can serve.

3. The Conversion Math Between Subscriptions and API Pricing

A subscription plan is essentially “fixed price + a maximum quota.” For light users, it is a discount; for heavy users, it creates arbitrage opportunities that the platform must contain with quotas and rate limits. Below is an illustrative calculation (numbers are fictional and used only to demonstrate the method):

• Assume the effective blended API price (including cache hits) is approximately ¥10 per million tokens.
• Plan A: ¥1,400 per month for a 500M token monthly quota. That works out to about ¥2.8 per million tokens—roughly a 2.8x discount. High value per token, but the absolute price threshold is high.
• Plan B: ¥100 per month for a 20M token monthly quota, approximately ¥5 per million tokens—about a 5x discount. Lower barrier to entry. But if its time-window quota is only 1M tokens per 5 hours, a heavy user will hit the ceiling by mid-morning, degrading the actual experience.

The lesson: evaluating a plan requires more than the discount rate. You must check whether the time-window quota matches your usage cadence. When the proportion of heavy users exceeds expectations and marginal service costs approach the price per customer, pausing new sales and prioritizing quota commitments for existing users becomes the rational choice for the platform.

Context Caching: The Key Variable That Determines Your Real Bill

1. How Cache Hits Work

In multi-turn conversations or when using repeated prefixes (e.g., a fixed system prompt or continuous questions on the same document), the KV Cache for the prefix portion can be reused. The billed rate for cached tokens is typically significantly lower than for uncached tokens. The cache hit rate is the single largest factor determining whether the same workload produces a small or large bill.

2. Engineering Practices to Improve Cache Hit Rate

The core principle is: stable content first, dynamic content last.

# Bad example: prefix contains a timestamp and random ID, so cache rarely hits
messages = [
    {"role": "system", "content": f"Current time: {now()}. You are a technical assistant."},
    {"role": "user", "content": f"Session ID: {uuid4()}. Please interpret: {doc}"},
]
# Good example: stable rules and the document first, volatile information last
messages = [
    {"role": "system", "content": "You are a technical assistant. (Long-term stable instructions)"},
    {"role": "user", "content": f"{doc}\n\nSession ID: {uuid4()}\nPlease interpret the above document."},
]

Additional practices worth noting:

• Keep all turns of the same task within a single conversation to avoid accidental prefix changes.
• When estimating costs, compute the effective unit price using “hit rate × cached price + miss rate × list price” rather than using the list price directly for budgeting.

Context caching, pay-as-you-go pricing, and quota management are now common capabilities on mainstream model service platforms such as Baidu AI Cloud. Specific rules vary by platform, so refer to the official documentation.

Migrating Existing Users: Smooth Transitions for Products and Billing Systems

When handling “what happens to old plans,” mature engineering approaches generally cover four points:

• Protecting existing entitlements: Do not force-cancel old plans. Support continuous subscription with a stable price anchor to avoid the trust damage of “existing users pay more first.”
• Grace periods: For users who manually disabled auto-renewal, keep a workable renewal window (e.g., 24 hours). This requires decoupling “subscription status” from “entitlement status” in the billing system.
• Smooth upgrades: Allow old plans to upgrade to new tiers according to defined rules, completing the entitlement switch within the window to prevent double charging.
• Limiting incremental sales: When supply is constrained, pause new tier purchases and prioritize honoring quota commitments to existing users.

Together, these four measures express a single principle: when capacity is limited, honor existing commitments before expanding.

User-Side Practice: How to Evaluate and Choose a Model Service

Abstracting away from any specific product, you can evaluate an AI service across five dimensions:

1. Accuracy and traceability: Does it provide cited sources based on retrieval augmentation? Does it retrieve only when necessary? Retrieving indiscriminately for every question mixes low-quality results into the context, reducing efficiency and potentially introducing errors.
2. Hallucination control and correction behavior: When pointed out an error, does the model merely issue a formulaic apology, or does it actually revise its reasoning chain? The latter matters far more for technical work.
3. Memory and personalization: Cross-session memory eliminates the need to restate long-term preferences and constraints, significantly reducing interaction cost.
4. Stability and quota policy: Consider availability during peak hours, daily or per-window limits, and the degraded experience after hitting a cap.
5. Cost transparency: Are billing rules, cache pricing, and quota conversion methods clearly documented?

On the cost side, a simple script can estimate monthly spend:

def monthly_cost(req_per_day, in_tok, out_tok,
                 p_in, p_out, hit=0.0, p_hit=None):
    """Estimate monthly API cost (RMB); prices are RMB per million tokens."""
    p_hit = p_in if p_hit is None else p_hit
    in_cost = req_per_day * in_tok / 1e6 * (hit * p_hit + (1 - hit) * p_in)
    out_cost = req_per_day * out_tok / 1e6 * p_out
    return (in_cost + out_cost) * 30
# Example: 200 requests/day, avg 8000 input tokens, 1000 output tokens,
# input list price ¥10/M, cache price ¥2/M, hit rate 0.8, output ¥20/M
print(monthly_cost(200, 8000, 1000, 10, 20, hit=0.8, p_hit=2))
# Output ~292.8, i.e., roughly ¥293/month for this workload

Plug in your own daily request volume and average input/output token counts, then compare against the effective per-token rate of candidate subscription plans to decide whether a subscription or pay-as-you-go pricing is more economical. For batch processing tasks, consider off-peak execution and leveraged asynchronous or batch API categories offered by the platform to further reduce costs.

Summary

“Compute shortage” and “pausing new sales” are not marketing rhetoric. They reflect a real capacity scheduling problem under supply constraints: inference costs grow linearly with usage, and long contexts plus retrieval-augmented workflows continuously amplify per-request overhead. Platforms respond by using time-window quotas, tier designs, and limiting incremental sales to maintain service levels. For developers, understanding how to convert token quotas, optimizing real bills through cache hit rates, and building your own cost model are the practical steps to ensure every dollar of budget is spent where it matters in a supply-constrained environment.