Engineering Compute Quotas and Subscription Mechanisms for LLM Services

{2333}# Engineering Compute Quotas and Subscription Mechanisms for LLM Services

I. Background: When Inference Services Hit Capacity Limits

With the continuous surge in LLM API calls, compute supply on the inference side has become a critical bottleneck restricting service quality. Recently, due to compute shortages, an LLM platform suspended the purchase of new subscription plans and adjusted the quota rules for existing ones. This incident sparked widespread discussion in the developer community: Can legacy plans continue to be used? How is the unit compute cost for new plans calculated? How much does the cache hit rate impact actual expenditure? These questions fundamentally point to a single technical proposition: how to design a quota system under limited GPU inference resources that balances fairness, predictability, and commercial sustainability.

This article does not evaluate specific vendors’ products. Instead, it abstracts a general methodology for capacity governance of LLM inference services from an engineering perspective, serving as a reference for architects and platform operators.

II. Basic Models for Compute Quota Metering

2.1 The Conversion Chain from Tokens to Credits

LLM service billing typically uses tokens as the smallest metering unit, but user-facing plans often present usage as “credits” or “points.” A stable conversion coefficient is required between the two. When designing this coefficient, three types of costs must be considered:

• Compute cost: The GPU time overhead differs between the prefill and decode phases; in long-context scenarios, the prefill proportion increases significantly.
• Cache cost: Whether the KV Cache is hit or not directly impacts actual compute consumption.
• Scheduling cost: Batching efficiency, queuing latency, and overhead from preemption and retries.

A reasonable conversion model can be abstracted as:

Actual deducted credits = (input_tokens × unit_price_in + output_tokens × unit_price_out) × discount_factor
discount_factor = f(cache_hit_rate, context_length, batching_efficiency)

The discount factor is not constant but changes dynamically with request characteristics. Descriptions like “calculated by average cache hit, roughly a 28% discount” or “roughly 50% off for low-tier plans” essentially describe the discount factor intervals corresponding to different subscription tiers. When designing plans, the platform should explicitly publish the calculation criteria for the discount factor; otherwise, users cannot estimate their actual available usage, leading to expectation gaps.

2.2 Time Windows and Quota Refresh

There are two common quota models:

1. Rolling window: Uses a sliding window, such as the “past 5 hours,” to calculate usage. The advantage is peak shaving and suppressing burst traffic, but the downside is high implementation complexity and difficulty for users to intuitively understand.
2. Fixed period: Resets by natural day or natural month. It is simple to implement and reduces cognitive load for users, but lacks constraints for short-term high concurrency.

In practice, many platforms adopt a hybrid model of “short-period rate limiting + long-period total quota”: the short period prevents individual users from maxing out the cluster instantaneously, while the long period controls overall costs. This dual-layer design needs to be implemented separately at the gateway layer and the billing layer—the former handles rate limiting, and the latter handles cumulative deduction (quota accounting).

III. Cache Hit Rate: The Underestimated Cost Variable

3.1 Why Caching Significantly Reduces Costs

In autoregressive inference, the KV Cache reuses the attention key-value pairs of historical tokens. When multiple requests share the same prefix (e.g., identical system prompts or identical long document contexts), the platform can reuse already computed KV blocks, skipping redundant prefill computations. The higher the hit rate, the lower the actual GPU time per token.

For a typical “long system prompt + short user input” scenario, if the system prompt accounts for over 80% of the length, cache hits can reduce prefill costs by an order of magnitude. This is the technical logic behind descriptions like “average cache hit costs about 10 currency units/m”—the gap between the nominal unit price and the effective unit price is primarily determined by cache reuse.

3.2 Engineering Techniques to Improve Hit Rates

• Prefix stabilization: Place system prompts, few-shot examples, and tool definitions at the very front of the request and maintain byte-level consistency. Avoid breaking prefix matching with dynamic content like timestamps or random IDs.
• Context chunking reuse: Use a fixed chunking strategy for long documents to align block boundaries across different requests.
• Session affinity routing: Route requests from the same session to the same inference instance to increase the local cache hit probability.
• Cache-aware batching: The scheduler prioritizes merging requests with identical prefixes when forming batches.

Below is an illustrative example of prefix stabilization:

# Not recommended: dynamic content breaks prefix consistency
system_prompt = f"Current time is {now()}. You are an assistant."
# Recommended: static prefix + dynamic content placed later
system_prompt = "You are an assistant."
user_content = f"[Time:{now()}]\n{question}"

This detail may seem minor, but in high-concurrency scenarios, it can impact overall compute expenditure by several multiples.

IV. Tiered Plans and Legacy User Compatibility Strategies

4.1 Core Trade-offs in Tiered Design

The essence of plan tiering is a combination of price discrimination and resource isolation. Common dimensions include call rate per unit time, maximum context length, concurrency, model tier, and priority queues. When designing, two extremes should be avoided:

• Tiers too dense: Users face choice paralysis, and operational complexity increases.
• Tiers too sparse: Low-tier user experience drops off a cliff, and the premium for high tiers lacks persuasion.

The mention of a low-tier plan having “only 1m available usage per 5 hours” leading to a poor experience reminds us: the short-period quota of a low-tier plan must cover the token consumption of a typical deep usage session. Otherwise, users will be rate-limited after a single long conversation, generating strong frustration. A reasonable approach is to first create user behavior profiles, calculate the P90 token consumption per session, and then reverse-engineer the lower limit for the short-period quota.

4.2 Legacy Plan Compatibility and Migration

When a platform adjusts its plan system, existing users are the most sensitive group. In engineering, the following strategies are typically adopted:

1. Grandfathering existing plans: Legacy plans remain usable and renewable, but are simply hidden from the new purchase entry points.
2. Smooth upgrade: Provide a one-time upgrade path. Once upgraded, it cannot be reverted, and a clear quota comparison must be shown in the UI.
3. Auto-renewal management: Allow users to manually disable auto-renewal, while retaining a time window (e.g., 24 hours) after disabling to restore renewal, preventing service interruptions caused by accidental operations.
4. UI transparency: Display the credit consumption curve, cache hit rate, and estimated remaining usage time on the billing page.

The common goal of these strategies is to reduce users’ decision costs and uncertainty, rather than using rule complexity for short-term gains. In the long run, the transparency of the quota system directly determines user retention.

V. Engineering Best Practices for Capacity Governance

Combining the above analysis, here are recommendations for platform providers and consumers respectively.

For Platform Providers:

• Establish capacity watermark monitoring, aggregating QPS and GPU utilization across multiple dimensions such as model tier, tenant level, and time period.
• During capacity constraints, prioritize the strategy of “restricting new purchases, preserving existing quotas” to prevent service degradation for already paid users.
• Publish quota conversion criteria and discount factor intervals to reduce customer support tickets.
• Handle burst traffic via queue priorities rather than hard rejections to improve overall throughput.

For Consumers (Developers and Enterprises):

• Implement request deduplication and result caching at the application layer to reduce redundant upstream calls.
• Stabilize system prompt prefixes to actively increase cache hit rates.
• Use asynchronous batching for non-real-time tasks to shift calls to off-peak hours.
• When selecting plans, reverse-engineer required quotas based on your business’s P90 token consumption rather than just looking at nominal unit prices.

VI. Conclusion

Compute scarcity is not an isolated problem for a single platform, but a structural challenge in the scaling process of LLM services. The solution lies not in simple “rate limiting” or “price hikes,” but in building a quota and billing system that is measurable, explainable, and predictable. Engineering techniques such as cache reuse, prefix stabilization, dual-layer rate limiting, and legacy compatibility collectively form the technical foundation of capacity governance. For developers, understanding this mechanism not only helps in making more economical API call decisions but also allows for reserving architectural flexibility against upstream capacity fluctuations during the design phase.