LLM API Rate Limits Explained: The GPU Memory Economics of Long-Context Inference and KV Cache

    When a viral open-source LLM platform saw token usage jump roughly eightfold in a week and then had to throttle access, the instinct was to blame tightening compute controls. This article shows why the real bottleneck was GPU memory, not chip supply. Starting from the KV cache mechanism that powers the inference engine, it breaks down the cost structure of long-context inference, explains why the cost per generated token can rise between 3.6x and 7x when context grows from 16K to 256K, and separates external compute constraints from internal efficiency bottlenecks. It then lays out practical optimization paths—paged KV cache, prefix caching, KV quantization, length-aware scheduling, speculative decoding, and token-budget capacity planning—alongside product strategy and observability guidance for teams that want to raise effective token throughput per GPU without hurting the user experience.

    Introduction: From viral growth to hard limits

    Since the second half of 2025, the large-model industry has seen a striking tension play out.

    On one side, usage climbed exponentially. After a leading open-source LLM platform in China launched a popular model version, token consumption reached 33.6 trillion in a single week—roughly an 8x jump week over week—topping global usage rankings. Demand this strong is, for any engineering team, a “sweet validation.”

    On the other side, service stability came under strain. In October alone, the platform suffered 8 service interruptions. By November, it had tightened several usage limits:

    • Disabling long conversations inactive for more than 7 days;
    • Capping single-request input at 40,000 characters;
    • Setting a daily conversation quota per user.

    High growth accompanied by high load is nothing unusual. What deserves a developer’s attention is a sharper question: why, despite steadily increasing compute investment, would a free product reach the point of having to actively put the brakes on demand?

    Starting from the KV cache—a core mechanism of the inference engine—this article systematically dissects the cost structure of long-context inference, clarifies the difference between external compute constraints and internal efficiency bottlenecks, and offers practical optimization ideas.

    1. The obvious explanation: tightening external compute controls

    Coincidentally, around the time these limits took effect, the U.S. Bureau of Industry and Security released a 132-page “interim final rule” on restricting remote access to advanced semiconductors.

    The rule’s key shift is in regulatory focus: the target moves from the physical cross-border movement of chips to the supply of remote compute. Under the rule, any overseas data center that provides “substantial” advanced AI-chip compute to a company headquartered in China must obtain a U.S. license. Countries such as Singapore, Malaysia, Thailand, Indonesia, and Brazil—previously viewed as “workaround channels”—were placed directly on a high-risk transshipment list.

    This effectively closes, at the institutional level, the path by which Chinese companies indirectly obtained Nvidia’s high-end chips by renting Southeast Asian data centers. After direct procurement had already been restricted, that path was an important channel for domestic AI firms to supplement their compute supply.

    The two events overlapped closely in time, so “a compute shortage forced the throttling” became the most intuitive explanation—and the easiest narrative to accept. But coincidence in timing is not causation, and the two need to be examined separately:

    Dimension External controls Internal efficiency bottleneck
    What it affects Access to incremental compute Efficiency of existing compute
    Core conflict Can’t buy or rent more advanced compute Low token output per unit of compute
    Main mitigations Diversify supply, adapt domestic chips, inference optimization KV cache governance, scheduling, resource tiering
    Time to see results Quarters to years Weeks to months

    Third-party measurements cited here offer corroborating evidence. A study titled “You don’t need long context,” submitted to a preprint platform on November 3, 2025, found that as context grows from 16K to 256K, the marginal value of the extra context is quite limited while the cost rises severalfold.

    In other words, the main driver of the limits was not that the total chip supply had run dry, but that a large volume of idle, low-efficiency long conversations under the free tier was structurally crowding out GPU memory. The external blockade tightens the “incoming water supply”; the limits address the “leakage in what’s already there.” Fixating only on the former makes it easy to miss where cost governance can actually make a difference.

    2. Understanding the cost: the KV cache mechanism

    2.1 Why attention needs to “remember” the past

    Large language models are built on the Transformer architecture, whose core attention mechanism lets the model “look back” at all previous tokens every time it generates a new token. The problem is that generating each token would otherwise require recomputing the keys (Key) and values (Value) of all historical tokens—an enormous amount of redundant computation.

    The standard engineering solution is the KV cache: after processing each token, its Key and Value vectors are stored. Subsequent generation only needs to compute the current token’s Query, then run attention against the cached Keys and Values—no need to recompute the historical portion.

    It trades space for time, and the price is that GPU memory stays occupied.

    2.2 Quantifying GPU memory usage

    The KV cache size of a single request can be approximated as:

    1. KV cache bytes ≈ 2 × L × H_kv × d_head × S × bytes

    Each parameter means the following:

    • 2: K and V are both cached for each token;
    • L: number of model layers;
    • H_kv: number of KV attention heads (much smaller than the query heads in structures such as grouped-query attention, GQA);
    • d_head: dimension of each head;
    • S: sequence length, i.e., the context length;
    • bytes: bytes per value (2 for half precision).

    The KV cache is thus directly proportional to the context length S: double the context, double the GPU memory. With multi-head attention (MHA), each request must store the Keys and Values for all attention heads on its own. GQA/MQA let multiple query heads share one group of KV heads, sharply reducing GPU memory overhead—a key reason mainstream models have widely adopted these structures in recent years.

    The following is an illustrative estimation function:

    1. def estimate_kv_bytes(layers: int,
    2. kv_heads: int,
    3. head_dim: int,
    4. ctx_len: int,
    5. bytes_per_elem: int = 2) -> int:
    6. """Estimate the KV cache GPU memory footprint of a single request at a given context length
    7. Args:
    8. layers: Number of model layers
    9. kv_heads: Number of KV attention heads (usually far fewer than query heads in GQA/MQA structures)
    10. head_dim: Dimension of each attention head
    11. ctx_len: Context length (in tokens)
    12. bytes_per_elem: Bytes per value (2 for half precision)
    13. """
    14. return 2 * layers * kv_heads * head_dim * ctx_len * bytes_per_elem
    15. # Example: 60 layers, 8 KV heads, head_dim 128, half precision
    16. per_token = estimate_kv_bytes(60, 8, 128, ctx_len=1) # ~240 KB
    17. print(f"Per token per request: {per_token / 1024:.0f} KB")
    18. print(f"32K context: {estimate_kv_bytes(60, 8, 128, 32768) / 1024**3:.1f} GiB")
    19. print(f"128K context: {estimate_kv_bytes(60, 8, 128, 131072) / 1024**3:.1f} GiB")
    20. # Assume a 40 GiB GPU memory pool for the KV cache, average context 32K
    21. pool_bytes = 40 * 1024**3
    22. one_req_32k = estimate_kv_bytes(60, 8, 128, 32768)
    23. one_req_128k = estimate_kv_bytes(60, 8, 128, 131072)
    24. print(f"32K context concurrency: ~{pool_bytes // one_req_32k} requests")
    25. print(f"128K context concurrency: ~{pool_bytes // one_req_128k} requests")

    These parameters are illustrative; actual values vary by model and should follow the architecture of the model in use. But the pattern is clear: within the same GPU memory pool, when the average context is 128K, the number of requests that can be processed concurrently is only about a quarter of that at a 32K context.

    2.3 The overlooked half: GPU memory retained by idle sessions

    Many developers focus only on “how long the input is” but overlook another key point: when a session is idle, is its KV cache still retained?

    To speed up multi-turn responses, mainstream inference services try to keep a session’s KV cache resident in GPU memory; a cache hit saves the cost of re-running prefill. If a service retains sessions by “inactivity timeout,” then:

    • A 128K conversation can hold ~30 GB of KV cache; even if the user steps away to a half-hour meeting, that memory stays occupied;
    • a single such session can equal the demand of dozens of active short sessions;
    • the more active users there are, the larger the share of GPU memory “frozen” by idle long sessions.

    This is a classic form of structural waste: resources are occupied but produce no token output.

    Returning to the limits themselves, all three measures have a clear technical purpose:

    1. Disabling long conversations inactive for more than 7 days is essentially a batch reclaim of KV caches retained for the long term;
    2. Capping single-request input at 40,000 characters (roughly tens of thousands of tokens) directly controls the GPU-memory peak of a single request;
    3. a daily conversation quota caps the cumulative consumption of heavy users.

    3. From 3.6x to 7x: why long-context cost multiplies

    The same third-party study compared the cost of generating a single token at 256K versus 16K context across three open-source models, finding roughly 3.6x, 5.6x, and 7x. Note that the comparison is the cost of generating one token. Why would generating the same single token cost several times more? That requires looking at the two phases of inference.

    3.1 Prefill: compute-intensive but manageable

    The prefill phase processes all input tokens at once. Attention computation has a complexity of roughly O(n²), and every input token participates in the Key and Value projections. The longer the context, the higher the time to first token (TTFT).

    Fortunately, this phase is highly parallel. Large matrix operations let the GPU’s compute be fully utilized, so the growth in unit cost is largely under control.

    3.2 Decode: bandwidth-bound, cost amplified

    The decode phase is what really drives up cost. When tokens are generated one at a time, every step must move the entire historical KV cache from GPU memory into the compute units. This is fundamentally a memory-bandwidth-bound operation: much of the GPU’s compute sits idle, and the bottleneck is moving data.

    With GPU memory bandwidth fixed, the key to higher throughput is batching. The more requests in a batch, the more requests served per weight read, and the lower the per-token cost.

    But the size of the KV cache is exactly what limits batch size:

    • The longer the context, the larger each request’s KV cache, and the fewer requests that fit at once;
    • when contexts can reach 256K, long requests occupy batch slots for extended periods;
    • new short requests not only have to queue but may be squeezed into long batches that could otherwise have served many more requests.

    This is the cost dilemma of long-context inference: the difficulty of generating a single token barely changes, yet the cost multiplies several times over. The reason is not that more is computed, but that each GPU can serve fewer requests at once. In queueing-theory terms, the service rate of each server is unchanged, but each customer now takes up more seats, so fewer customers can be served simultaneously.

    The differences in these multiples across models in the study are closely tied to architecture parameters such as the number of KV heads and layers. So these multiples shouldn’t be applied directly to other models; they should be measured and calibrated against the model’s own architecture.

    3.3 Active long sessions vs. idle long sessions

    Overall, what really drives up service cost is not “many conversation turns” but “too-long context.” Roughly classified by user behavior, there are four types:

    User behavior Typical scenario Cost impact Governance approach
    Short context, highly active Everyday Q&A, copy polishing Low cost per request, but high request volume Rate limiting and quotas to protect the baseline experience
    Long context, highly active Codebase analysis, long-document processing High cost per request, but clear output Guarantee capacity; can gently nudge toward paid tiers
    Short context, idle long-term Light users who come and go Limited resource use Session-expiry reclamation is enough
    Long context, idle long-term Long sessions left running unattended High occupation, low output—the main “leakage point” Reclaim on idle; optimize session-retention policy

    For a free product, consumer users on the free tier are numerous over the long run—this platform has roughly 130 million monthly active users. These users don’t directly generate revenue, yet they share the same inference resource pool as paid API users. When a large number of idle long sessions keep crowding out GPU memory, paid users’ latency and service levels suffer too. This is effectively a cost externality: one group’s resource consumption is passed on as a degradation in another group’s experience.

    4. “The good problem”: the gap in supply-demand curve slopes

    Once the cost structure is clear, the deeper logic behind the limits becomes understandable. This is not simply a “compute drought” but a contest of supply-demand slopes.

    • The trouble is on the supply curve: supply expansion is constrained by hardware procurement and delivery cycles, data-center construction cycles, and energy supply. Once the external environment tightens, acquiring incremental advanced compute becomes even harder. This is an almost-linear curve whose elasticity keeps falling.
    • The fortune is on the demand curve: it shoots up almost vertically. Token consumption grew severalfold within weeks, and the users are mostly developers and technology enthusiasts. They “vote” with real usage—the most direct and convincing validation a market can give a product.

    The difference in the two curves’ slopes means that a free product will sooner or later face resource constraints. Every successful platform in history has had to cross this threshold without exception; the only difference is how—by simply cutting features and capacity, or by systematic efficiency gains.

    So the limits themselves are not the answer to the problem but the beginning of cost governance. The real question to answer is: without sacrificing the user experience, how many effective tokens can one GPU produce? Optimization can proceed on several levels around this question.

    5. Optimization paths: raising “tokens per GPU”

    5.1 GPU memory management: from static reservation to paged on-demand

    Paged KV cache management. Traditional implementations often reserve a contiguous block of GPU memory per request, sized for the maximum possible length, while only a small part is actually used; the leftover memory can’t be repurposed and tends to cause fragmentation. The mature industry approach borrows the paging idea from operating-system virtual memory: split the KV cache into fixed-size blocks, allocate on demand, and use a block table to map logical sequences to physical GPU-memory blocks. The benefits are twofold:

    • GPU-memory fragmentation is largely eliminated and actual utilization rises substantially;
    • different requests can share the physical blocks for an identical prefix.

    Prefix cache reuse. Many requests share the same system prompt, tool definitions, knowledge-base background, or code-repository context. Persisting and reusing this KV cache across requests can:

    • compute the prefill for repeated content only once;
    • significantly lower TTFT;
    • in long-context scenarios, often eliminate a substantial share of the redundant cost.

    The key is session routing: requests with the same prefix should be routed to the same replica.

    KV cache quantization and sparsification. Storing the KV cache in int8 or even lower precision can sharply compress GPU-memory usage with almost no loss in quality, raising the concurrency. Combined with attention sparsification and importance-based eviction of historical tokens, it can further bound the cache size of long sessions.

    Tiered storage and cache offloading. Reserve GPU memory for the most active requests. Idle requests’ KV caches can be offloaded to host memory or local SSD and swapped back on demand when the user returns, balancing “responsiveness” against “resource usage.”

    5.2 Context engineering: not every token deserves to be paid for

    Retrieval on demand. For long documents, rather than stuffing in the entire document at once, retrieve only the relevant fragments on demand using retrieval-augmented generation (RAG). The retrieved content becomes a stable prefix for subsequent turns—it hits the prefix cache and sharply compresses the context length.

    Rolling summary and compression. For very long multi-turn sessions, compress earlier turns into a structured summary, keep only recent verbatim content, and treat the summary as a new stable prefix. This keeps the context length within a bounded range.

    Tiered context routing. Estimate token length and expected value from request characteristics and route to different processing paths:

    • short requests go to a lightweight model;
    • long requests go to a high-throughput instance;
    • suspected low-value requests go to a queue.

    5.3 Scheduling and batching: separate long and short so they don’t hold each other back

    Length-aware scheduling. Group requests of similar length into the same batch, avoiding a scenario where a single 256K request is mixed with many short ones and forces the latter to “wait for the slowest companion.”

    Separate pools for long and short requests. Route long requests to a dedicated large-memory instance group and let short requests run with high concurrency in their own group, so their performance doesn’t interfere.

    Continuous batching and preemption. Use continuous batching, which dynamically re-forms batches per iteration—requests leave as they finish and new ones join at any time. Combined with priority and preemption, low-priority long requests’ KV caches can be swapped out when needed so paid requests get resources first at peak times.

    Speculative decoding. A small draft model predicts several tokens at once, and the large model verifies them in parallel. This markedly reduces the number of decode steps and thus the repeated reads of the KV cache, with especially clear gains in long-context scenarios.

    5.4 Scheduling policy example

    The pseudocode below illustrates the basic idea of “length-aware admission + priority preemption”:

    1. from dataclasses import dataclass, field
    2. from enum import IntEnum
    3. from typing import Deque
    4. class Tier(IntEnum):
    5. """Request tiers: a smaller value means higher priority"""
    6. PAID_API = 0 # Paid API request
    7. FREE_ACTIVE = 1 # Active request from a free user
    8. FREE_IDLE = 2 # Swap-back of an idle free-user session
    9. @dataclass
    10. class Request:
    11. tier: Tier
    12. est_tokens: int # Estimated context length
    13. kv_bytes_needed: int # GPU memory needed for the KV cache
    14. blocks: list = field(default_factory=list)
    15. class Scheduler:
    16. """Length-aware, priority-preempting inference scheduler (pseudocode)"""
    17. def __init__(self, kv_pool_bytes: int, block_bytes: int):
    18. self.kv_pool_bytes = kv_pool_bytes
    19. self.block_bytes = block_bytes
    20. self.running: list[Request] = [] # Requests currently executing
    21. self.waiting: dict[Tier, Deque[Request]] = {t: Deque() for t in Tier}
    22. def admit(self, req: Request) -> bool:
    23. self.waiting[req.tier].append(req)
    24. self._try_schedule()
    25. return bool(req.blocks)
    26. def _free_bytes(self) -> int:
    27. used = sum(len(r.blocks) for r in self.running) * self.block_bytes
    28. return self.kv_pool_bytes - used
    29. def _try_schedule(self) -> None:
    30. for tier in Tier: # from highest to lowest priority
    31. while self.waiting[tier]:
    32. req = self.waiting[tier][0]
    33. if req.kv_bytes_needed <= self._free_bytes():
    34. self.waiting[tier].popleft()
    35. req.blocks = [object()] * (req.kv_bytes_needed // self.block_bytes)
    36. self.running.append(req)
    37. continue
    38. # When GPU memory is short, swap out same-tier long requests;
    39. # if still short, swap out lower-tier requests
    40. for victim in sorted(self.running,
    41. key=lambda r: r.est_tokens,
    42. reverse=True):
    43. if victim.tier > req.tier or (
    44. victim.tier == req.tier and victim.est_tokens > req.est_tokens
    45. ):
    46. self.running.remove(victim)
    47. self.waiting[victim.tier].appendleft(victim) # keep cache, swap back later
    48. if req.kv_bytes_needed <= self._free_bytes():
    49. break
    50. if req.kv_bytes_needed > self._free_bytes():
    51. break # still not enough resources, wait for the next round
    52. def on_token(self, req: Request, extra_bytes: int) -> bool:
    53. """Append cache blocks on demand for each generated token"""
    54. need = extra_bytes - self._free_bytes()
    55. if need <= 0:
    56. req.blocks.append(object())
    57. return True
    58. # Prioritize evicting idle caches from other sessions, then preempt low-tier long requests
    59. # ... (engineering implementation omitted)
    60. return False

    5.5 Capacity planning: from “watching QPS” to “watching a token budget”

    When long contexts become the norm, using QPS to measure capacity becomes seriously distorted. The GPU memory used by 10 requests at 128K may equal that of hundreds of requests at 1K. A more sensible approach is to treat GPU memory as a token budget:

    1. Max concurrent context tokens per replica ≈ KV cache GPU memory available / KV bytes per token

    Here is an example:

    1. def max_context_budget(kv_pool_gib: float,
    2. layers: int,
    3. kv_heads: int,
    4. head_dim: int,
    5. bytes_per_elem: int = 2) -> int:
    6. """Estimate the total context tokens a single replica can hold concurrently"""
    7. per_token = 2 * layers * kv_heads * head_dim * bytes_per_elem
    8. return int(kv_pool_gib * 1024**3 / per_token)
    9. # Assume 40 GiB of GPU memory available for the KV cache
    10. budget = max_context_budget(40, 60, 8, 128)
    11. avg_ctx = 8192
    12. print(f"Token budget per replica: {budget / 1e6:.1f}M")
    13. print(f"At an average context of 8K, roughly {budget // avg_ctx} concurrent requests")
    14. def required_replicas(qps: float,
    15. avg_ctx: int,
    16. tokens_per_sec_per_replica: int,
    17. max_budget: int,
    18. headroom: float = 1.3) -> int:
    19. """Estimate the number of replicas needed from peak QPS and average context (with a headroom factor)"""
    20. concurrency = qps * avg_ctx / tokens_per_sec_per_replica
    21. return int(-(-concurrency * headroom // max_budget)) # round up

    In engineering practice, fragmentation and safety margins must be reserved on top of this, and parameters calibrated through continuous load testing.

    6. Product strategy: free doesn’t mean boundless

    Efficiency optimization addresses “using less”; product strategy addresses “using it for the right things.” The two need to advance together.

    Replace count limits with a token budget. Rather than simply capping the number of conversations, set a daily quota by token consumption. Normal users barely notice, while heavy users need to plan sensibly.

    Automatic reclamation of idle sessions. A sensible idle timeout, clearly communicated to users, promptly frees idle GPU memory. Such measures barely affect normal users yet are the most direct and effective way to recover “leakage.”

    Tiered service guarantees. Set differentiated guarantees per user tier—for example, giving paid requests higher priority and stricter latency targets. Cache reservation and quota isolation keep free traffic from crowding out paid capacity.

    Guide rather than block. When a request is about to exceed the limit, proactively suggest options such as “trim the context” or “switch to retrieval mode,” turning the limit into a moment of product guidance.

    Canary rollout and transparent communication. Limit policies should be thoroughly evaluated, rolled out gradually, and monitored for changes in key metrics; at the same time, explain the rules and reasons clearly to users to reduce misunderstanding.

    The pseudocode below shows a request-handling flow that combines idle reclamation with a token budget:

    1. import time
    2. IDLE_TTL_SECONDS = 1800 # Reclaim long-session cache after 30 minutes idle
    3. def reclaim_idle_sessions(sessions: dict, now: float | None = None) -> int:
    4. """Reclaim KV caches of sessions idle beyond the timeout; return the number reclaimed"""
    5. now = time.time() if now is None else now
    6. reclaimed = 0
    7. for sid, sess in sessions.items():
    8. if sess.ctx_len > 0 and now - sess.last_active > IDLE_TTL_SECONDS:
    9. sess.kv_blocks.clear() # free the KV cache, keep the session summary
    10. sess.ctx_len = 0
    11. reclaimed += 1
    12. return reclaimed
    13. def handle_request(req, user, sessions: dict) -> dict:
    14. sess = sessions.setdefault(req.session_id, type("S", (), {
    15. "last_active": 0.0, "ctx_len": 0, "kv_blocks": [], "summary": ""
    16. })())
    17. now = time.time()
    18. # 1. Idle reclamation
    19. reclaim_idle_sessions(sessions, now)
    20. # 2. Quota check: measured by token budget, not by count
    21. if not sess.kv_blocks: # cache invalidated, needs re-prefill
    22. sess.ctx_len = 0
    23. if user.tokens_today + req.est_tokens > user.daily_token_quota:
    24. return {"status": "quota_exceeded",
    25. "hint": "Trim the context or switch to retrieval mode, then retry"}
    26. # 3. Context budget: compress history if necessary
    27. prompt = build_prompt(sess, req)
    28. if prompt.tokens > req.max_input_tokens:
    29. sess.summary = summarize(sess) # rolling summary
    30. prompt = build_prompt(sess, req, summary=sess.summary)
    31. # 4. Admit to the tiered scheduler
    32. scheduler.admit(req)
    33. return {"status": "queued"}

    7. Observability: making cost visible and calculable

    Optimization requires measurement. It’s advisable to establish at least the following set of metrics:

    Metric Meaning Optimization direction
    Prefix cache hit rate Share of requests that hit an existing KV cache Improve routing affinity; optimize prompt structure
    KV cache utilization Share of cache blocks in use out of the pool Tune block size and capacity watermark
    Effective batch factor Average requests per batch vs. max batch capacity Improve batching strategy; reduce fragmentation
    TTFT Time to first token Prefix reuse, context compression, pool separation
    TPOT Generation latency per output token Speculative decoding, batching optimization
    Cost per token GPU cost per million tokens The combined result of all the above

    Among these, the prefix cache hit rate and KV cache utilization most directly reflect whether GPU memory is being used effectively.

    On the metering side, the server already records per-request data such as input length, output length, cached tokens hit, and queue time. Combining this with resource-monitoring data enables cost allocation, so the cost contribution of different scenarios and user groups can be precisely accounted for—providing a data basis for limit thresholds and pricing strategy.

    The end goal is to turn compute from a “black box” into a “white box”: to know exactly where every unit of GPU cost goes, and which requests are “high-consumption, low-output.”

    8. Under external constraints, efficiency is the most reliable “increment”

    Returning to the external controls at the outset: however the difficulty of acquiring incremental compute changes, one rule always holds—the tighter the external constraints, the more important the output efficiency of each unit of compute.

    For teams in China, long-context optimization carries an even more practical significance: it relies less on hardware-generation upgrades and more on engineering and algorithm improvements, making it the most certain and fastest-acting cost-reduction path in the short term. The techniques above—paged caching, prefix reuse, speculative decoding, length-aware scheduling—already have relatively mature engineering practice in the industry and good portability across hardware and models. For putting them into practice, the recommended order is:

    1. Measure: establish a baseline, quantifying KV cache usage and cost per token;
    2. Reuse: raise GPU-memory utilization through paged management and prefix caching;
    3. Compress: lower the average context length through context compression and retrieval-on-demand injection;
    4. Schedule: raise the effective batch size through length-aware scheduling and long-short pool separation;
    5. Tier: balance the experience and cost of different user groups through quotas and priorities.

    Conclusion

    Severalfold growth in usage within a week is the best endorsement the market can give; the limits that followed are the “growth tax” that must be paid when growth is too fast.

    The “good problem” is “good” because the steep demand curve proves the product’s value; it is a “problem” because the supply curve can’t keep up with the demand curve’s slope. What closes that slope gap is not more and more expensive chips, but a deeper understanding of the inference system and more refined engineering.

    When the KV cache turns from a “hidden cost” into a “manageable, controllable resource,” and when the output of every token is fully accounted for, throttling stops being a footnote of inadequate capability and becomes the starting point of sustainable growth.