Compute Allocation Strategies and Membership Tier Restructuring for LLM Services

I. Background: The Compute Supply Crisis Driven by Parameter Scaling

In July 2026, a leading LLM platform released a new generation model with 2.8 trillion parameters, making it one of the largest open-source models globally. The leap in model capabilities was directly reflected in user behavior—within 48 hours of release, request volumes surged to multiples of the original estimates, rapidly approaching the capacity limits of existing compute clusters.

This event is not an isolated case. As LLM technology evolves from text-based conversations to multimodal understanding, complex reasoning, and code generation, the growth curve of model parameters has far outpaced the hardware iteration speed predicted by Moore’s Law. Taking a 2.8 trillion parameter model as an example, the VRAM required for a single forward inference exceeds the capacity of mainstream accelerator cards. It heavily relies on large-scale tensor parallelism and pipeline parallelism strategies, splitting the model across dozens or even hundreds of compute cards. This leads to dual cost pressures:

• Memory Bandwidth Bottleneck: Larger parameter scales mean more weight data needs to be read per unit time, causing an exponential rise in access pressure on High Bandwidth Memory (HBM).
• Amplified Communication Overhead: In distributed inference, the computation of each transformer layer is accompanied by AllReduce communication across multiple cards. Larger parameter scales increase the communication ratio, meaning actual effective compute falls far below theoretical peaks.

Therefore, when user requests spike rapidly in a short period, queue latency, timeout rates, and error rates in the compute cluster will deteriorate drastically. Without intervention, all users (including existing paid users) will face an unavailable service experience, leading to user churn and brand reputation damage.

II. Pausing Subscriptions and Prioritizing Existing Users: Emergency Governance Under Compute Scarcity

Faced with this crisis, the platform announced late on July 19 that it would immediately pause new user subscription entries on the C-end, while dedicating all existing compute to serving already subscribed users, ensuring the highest priority for existing user rights.

This decision is rooted in a classic multi-objective optimization under constrained resources problem. From an operations research perspective, the platform needed to balance the following objectives:

1. New User Acquisition: Keeping subscriptions open brings incremental revenue but dilutes the experience for existing users.
2. Existing User Retention: Guaranteeing service quality for loyal users is the foundation for maintaining long-term cash flow and reputation.
3. System Stability: Overloading triggers a cascading effect, where a large number of request timeouts and retries further amplify system pressure.

In this scenario, pausing new user subscriptions is essentially a rational choice of sacrificing short-term growth for system stability. The underlying technical logic includes:

• Peak Shaving: Controlling the total volume of concurrent requests to prevent instantaneous traffic from exceeding the queue depth limit of the scheduler.
• Priority Queue: Tagging subscribed users with high-priority labels and employing Preemptive Priority Scheduling in the queuing system to ensure high-priority requests receive compute resources first.
• Degradation and Rate Limiting: Degrading non-core features (such as web search, long document parsing) to concentrate compute power on the main conversational chain.

It is worth noting that pausing subscriptions is not canceling them. For existing users, their rights remain unaffected, and since new user volume drops to zero, the average compute available per user actually increases. This move gained broad understanding within the user community and bought buffer time for subsequent rights structural adjustments.

III. Rights Decoupling: Separating Main and Specialized Entitlements

During the subscription pause, the platform simultaneously pushed forward a more fundamental architectural adjustment—splitting the originally bundled single membership into two independent subscription options: the “Kimi Main Entitlement” and the “Kimi Code Entitlement.”

3.1 Why the Split?

Under the traditional single membership system, all users share the same compute pool, regardless of whether their use case is daily Q&A, office writing, or code generation. This was barely feasible when model sizes were small, but once parameter scales reached the trillion level, the resource consumption differences across business scenarios were drastically amplified:

• Code Generation Tasks: Require longer generation sequences, higher sampling temperatures, and frequent tool calls (like executing test cases). The compute consumption per request can be 5-10 times that of a normal conversation.
• Office Document Processing: Focuses on stable processing of long contexts, demanding high VRAM capacity but relatively less sensitive to inference throughput.
• Daily Conversations: High request frequency, low consumption per request, and extremely sensitive to latency.

If these three types of demands are mixed in the same resource pool, the platform can only reserve resources based on the “worst-case scenario,” leading to massive compute idling. After the split, each entitlement corresponds to independent resource quotas and scheduling policies, enabling more granular cost accounting and traffic governance.

3.2 Defining the Boundaries of the Main Entitlement

The “Kimi Main Entitlement” covers three core entry points: Web, App, and collaborative work platforms (Work). This division follows the principle of terminal consistency—regardless of the entry point, the underlying inference engine, context management mechanism, and security policies are identical, ensuring a cross-platform experience with no differences.

Core strategies for resource scheduling under the Main Entitlement include:

• Elastic Concurrency Limit: Dynamically adjusting the number of concurrent requests per user based on current cluster load. When compute is tight, priority is given to ensuring low latency per request rather than expanding the concurrency window.
• Context Cache Reuse: Establishing a KV Cache for frequently occurring conversational prefixes (like system prompts, common document fragments) to directly reuse repeated attention computations, saving about 30% of compute overhead.
• Intelligent Routing: Routing requests to the optimal compute node based on the user’s region, network latency, and current node load, reducing queueing probabilities.

3.3 The Isolation Value of Specialized Entitlements

Splitting the code generation scenario into an independent entitlement provides core value through stability guarantees via resource isolation. In a hybrid deployment model, bursty requests from code generation (e.g., code completions triggered intensively during daytime working hours) cannibalize resources from daily conversational users. Post-split, the two entitlements have their own independent capacity planning and elastic scaling strategies, achieving:

• Fault Domain Isolation: If a batch of inference nodes for the code service fails, it does not affect users of the Main Entitlement.
• Differentiated SLAs: The Main Entitlement provides a 99.9% availability commitment, while the Code Entitlement can offer 99.5% availability for batch processing scenarios but allows queue waiting, thereby improving overall resource utilization without increasing total costs.

From an operational perspective, the split subscription model also provides more flexible adjustment space for future pricing strategies. For example, high-frequency code developers could be offered a “pay-as-you-go + monthly deduction” combo, while lightweight users could be offered a lower-barrier monthly option for the Main Entitlement.

IV. Model Iteration and Compensation: Upgrading Resource Governance in the K3.1 Era

In October 2026, the platform launched the new K3.1 series, starting with the K3.1 Pro, featuring 3 trillion parameters and supporting an ultra-long context of 1 million tokens. The model’s improved capabilities brought compute challenges again, but this time, the platform’s response was noticeably more mature.

4.1 Resource Challenges of the New Model

A 3 trillion parameter model requires over 2TB of VRAM for inference. Even using the current highest-end single cards (approx. 80GB HBM), it requires at least 25 cards to fully load the model weights. Factoring in the VRAM needed for the KV Cache (which can exceed 20GB per request with a 1 million context), the VRAM resources occupied by a single request are massive.

This means:

• The number of concurrent requests a single cluster can serve drops further, potentially from “thousand-level concurrency” to “hundred-level concurrency.”
• Processing latency for long-context requests increases significantly, requiring inference optimization methods like speculative decoding and early-exit mechanisms to reduce time-to-first-token.
• Request scheduling needs to introduce a compute consumption estimation mechanism, estimating resource requirements based on context length before a request enters the queue, and using load balancing strategies to decide whether to accept it.

4.2 Rate Limiting and User Compensation Mechanisms

During the initial rollout of K3.1 Pro, some users experienced rate limiting (requests being queued or rejected). Due to increased context lengths, quotas originally billed by token were consumed rapidly. On October 26, the platform issued an apology and introduced compensation measures:

• Usage Compensation: Compensated all active subscription members with an extra 25% of their total monthly quota.
• Baseline Reset: If a user’s consumed quota for the month was less than 25% of the total, their remaining usage was reset to zero before adding the compensation, ensuring the compensation was perceptible.

The design of this compensation mechanism is worth learning from; it is a classic practice of SLA (Service Level Agreement) breach management. When a service provider fails to deliver promised quality, it should not stop at verbal apologies but repair user trust through quantifiable resource compensation. At the implementation level:

• Accurate Accounting of Compensation: Requires establishing request-level metering logs recording token consumption, queue wait time, and whether rate limiting was triggered for each request, aggregating this by user dimension to generate compensation reports.
• Automatic Issuance over Application-based: Compensation quotas are automatically credited, reducing user operational costs and avoiding user drop-off due to cumbersome application processes.
• Validity Period for Compensation: To prevent users from hoarding compensation quotas, a reasonable usage window (e.g., valid for 30 days) is set to encourage continuous user activity.

4.3 From Emergency Governance to Normalized Resource Operations

Comparing the two events in July and October clearly shows the platform’s evolutionary path in compute governance capabilities:

Dimension	July (First 2.8T Model)	October (First 3T Model)
Response Strategy	Paused new subscriptions, served only existing users	Allowed new subscriptions, but controlled load via rate limiting
User Communication	One-way announcement	Public apology + quantified compensation
Resource Scheduling	Unified pooling, no differentiated priority	Main/Code entitlements scheduled independently, granular SLAs
Cost Accounting	Rough estimation	Fine-grained metering based on request-level logs

This reflects an industry trend: as model parameter scales move from hundreds of billions to the trillion level, compute is no longer an infinitely suppliable resource, but a scarce asset requiring fine-grained governance.

V. Future-Oriented Compute Expansion and Architectural Evolution

5.1 Practical Route for Compute Expansion

The platform explicitly stated that the ultimate goal of pausing subscriptions is to “gradually open more subscription slots as new compute comes online.” Compute expansion is not just about purchasing more GPUs; it involves a series of systems engineering tasks:

• Cluster Network Upgrade: Larger parameter scales mean higher communication ratios. Introducing high-bandwidth, low-latency RDMA networks (like InfiniBand or RoCE) is necessary to reduce collective communication latency by an order of magnitude.
• Storage Tiering: Migrating cold data (rarely used conversation logs) to low-cost storage, while keeping hot data (active session contexts) on NVMe SSDs or in memory, reducing disk I/O bottlenecks when inference nodes read historical contexts.
• Scheduler Upgrades: Moving from single-cluster scheduling to multi-cluster federated scheduling, supporting the scheduling of a single request’s multiple shards across clusters in different locations. This means cross-datacenter bandwidth guarantees and disaster recovery designs must be synchronized.

5.2 More Granular Billing and Entitlement Models

Building on the split of Main and Code entitlements, the future subscription system can further evolve into:

• Scenario-Specific Subscription Packages: E.g., a “Document Enhancement Pack” for long-document analysis, or an “API Acceleration Pack” for high-concurrency calls.
• Reserved Capacity: For enterprise users, locking in compute quotas for specific time periods via prepayment to guarantee deterministic performance for critical business.
• Dynamic Priority and Bidding: Offering discounted compute during off-peak hours, allowing users to obtain non-real-time inference services at lower prices (e.g., overnight batch processing tasks), thus flattening the全天resource load curve.

5.3 Transparent Operations and User Education

A commendable point in this incident is that the platform clearly explained the root causes to users via announcements and apology letters, rather than being evasive. For LLM services, compute scarcity should not be viewed as an “unspeakable internal flaw” but as a normalized state of technical operations that should be transparent to users. Transparent communication brings three benefits:

1. Reduced Expectation Gap: Understanding the relationship between model scale and compute consumption gives users stronger psychological expectations for occasional rate limiting.
2. Off-Peak Usage Guidance: If the platform publishes daily load curves, users can spontaneously migrate non-urgent tasks to off-peak hours, achieving a win-win.
3. Building Trust Assets: When users recognize the platform’s candid attitude, even if problems occur, they will factor “historical response quality” into their renewal decisions, rather than judging solely based on a single current failure.

VI. Conclusion: A Paradigm Shift from Compute Crisis to Service Governance

Reviewing the entire incident, from the request flood triggered by the release of a 2.8 trillion parameter model, to the emergency control of pausing new subscriptions, and finally to the implementation of entitlement splits and quantified compensation, this is fundamentally a necessary path for an AI platform transitioning from “savage growth” to “fine-grained operations.”

For any team providing LLM services, the following points are directly applicable universal experiences:

• Establish a “Traffic Light” Mechanism for Compute Capacity Planning: Trigger warnings when cluster load exceeds 70% of capacity, initiate rate limiting at 85%, and must trigger circuit breaker protection at 95%.
• Build a Full-Link Metering System from Day One: Only by recording resource consumption and queue latency at the request level can fair, verifiable compensation decisions be made during a crisis.
• Deeply Couple User Entitlement Design with Resource Scheduling Strategies: The entitlement structure should not merely be a commercial pricing issue; it must reflect the physical constraints and cost structure of the underlying compute.

The LLM competition is far from over, but the true watershed may not be who releases the largest parameter model first, but who can operate the compute resources behind these models more sustainably. The problems exposed and the methodologies distilled from this incident hold significant reference value for the entire industry.