Revenue Priced per Token, Costs Driven by GB: Cutting Agent Inference Costs

    As open-source models gain share in token call volume, their cost structure tells a different story: revenue is accounted per token, but costs follow GB. This article explains why long-context Agent workloads shift the binding constraint from compute throughput to resident session capacity, breaks down the KV Cache as a growing memory and storage footprint, and lays out actionable optimization paths—prefix caching, tiered offloading, session lifecycle governance, quantization and sparsification, and admission control—along with the metrics needed to manage this "state ledger" and the data-security boundaries to protect.

    Introduction: Strong Share, Heavy Ledger

    In recent industry discussions and public commentary, one observation keeps recurring: the share of open-source models in token call volume is rising rapidly, but their share of total enterprise model spending remains limited. In other words, this is a high-concentration but low-volume share. The share itself isn’t the biggest issue—the real shift is happening in its cost structure.

    Selling tokens looks like selling compute, but what is actually delivered is state. And state doesn’t attach to compute—it lands in memory and storage, precisely the components under the most pressure in this round of price increases. The contradiction emerges: revenue is accounted per token, but costs follow GB. This is why “why does longer context mean higher inference cost” has become a practical concern for more and more teams.

    (Note: The share and cost relationships in this article are qualitative judgments based on industry observation. Specific ratios vary with statistical methodology and samples, and readers should verify against their own business.)

    1. Traffic Is Running, Assets Sit Idle

    While the market fixates on the token call leaderboards from major aggregation platforms, most analysis still anchors the narrative on model capability and pricing advantages, assuming this is ultimately a “compute-for-token” business. This understanding is coherent enough in traditional conversational scenarios, but in the Agent tasks now driving overseas traffic growth, the underlying logic of the workload is diverging.

    1.1 Traditional Conversational Inference: An Idealized Resource Model

    The resource model for traditional conversational inference is intuitive enough, and it is the basis for most current API pricing systems:

    • The user asks a question, the inference cluster computes and returns content;
    • The conversation ends, the session closes, the occupied cache is released, and the hardware returns to idle;
    • The provider can estimate costs from total token volume, and when there is spare cluster utilization, higher throughput dilutes unit cost further.

    So the logic of “run volume at low prices” and “trade scale for profit” is self-consistent, with token counts and resource consumption assumed to grow at similar rates. Resources are fluid; when traffic stops, costs come down too.

    1.2 Agent Workloads: The Resource Assumption Breaks Down

    The main traffic driving incremental growth through overseas aggregation channels is shifting from scattered human-machine conversations to Agent tasks with automated pipelines. The two differ significantly in resource rhythm:

    Dimension Traditional Conversational Inference Agent Long-Session Workload
    Prefill Short prefix, computed once Often ingests tens of thousands to hundreds of thousands of tokens of documents/business data
    Decode A few rounds of short Q&A Dozens of rounds of chained tool calls, continuous retrieval
    Cache residency Released when the session closes Session can be suspended for hours, context reused repeatedly
    Release timing Reclaimed immediately after the request ends Released only via an active close or TTL eviction
    Cost driver Roughly linear growth with tokens Determined by the “time × space” of the resident state

    More critically, without actively closing the session, this context cache keeps occupying resources. As the model continuously generates new tokens, it only appends to the cache and does not automatically return this large block of occupancy to the resource pool. So “assets sit idle”: compute may be temporarily free, but the memory and storage prepared for the context remain firmly held.

    2. Why the Cache Takes Center Stage: From Compute Ledger to State Ledger

    To understand why costs follow GB, we need to go back to the internal mechanics of Transformer inference.

    In autoregressive generation, the model must look back at all previous tokens’ Key/Value vectors for every new token it generates. To avoid recomputation, the inference engine caches the K/V tensors of each layer and each attention head—this is the KV Cache. It brings two direct consequences:

    • Separation of Prefill and Decode: Prefill, which handles long prefixes, is compute-intensive and determines time-to-first-token (TTFT); Decode is memory-bandwidth-intensive and requires the entire KV Cache to stay resident.
    • Memory usage grows roughly linearly with context length: The longer the context and the more concurrent sessions, the more memory is locked.

    An estimation example to illustrate the order of magnitude (parameters are illustrative, not metrics for any specific product):

    1. # Estimate the memory footprint of KV Cache for a single session
    2. def kv_cache_bytes(num_layers, num_kv_heads, head_dim, seq_len, bytes_per_elem=2):
    3. per_token = 2 * num_layers * num_kv_heads * head_dim # K and V each take one copy
    4. return per_token * seq_len * bytes_per_elem
    5. size_gb = kv_cache_bytes(
    6. num_layers=32, num_kv_heads=8, head_dim=128,
    7. seq_len=200_000, bytes_per_elem=2, # FP16 precision
    8. ) / (1024 ** 3)
    9. print(f"KV Cache for a single session is about {size_gb:.2f} GB") # about 24.41 GB

    Working through the formula: a single token needs 2×32×8×128 = 65,536 elements cached, times 200,000 tokens, times 2 bytes per element, gives about 24.4 GB. In other words, this single example session already occupies about 24 GB; with longer context and larger model parameters, a single session’s cache can reach tens of GB. When thousands of such sessions reside simultaneously, memory quickly hits its ceiling, and the system must offload part of the state to host memory and even external storage—descending from GPU high-bandwidth memory all the way to relatively cheap but slower tiers. Storage and memory thus shift from supporting roles to the leads that determine cost and capacity.

    One boundary needs to be added: not all scenarios are like this. In conversations with short context, low concurrency, and immediate session termination, KV Cache usage is small and residency time is short, so costs remain closer to the token-linear model. The “state ledger” effect above mainly shows up significantly under long-context, high-concurrency Agent workloads.

    3. Constraint Point Migration: From Throughput to Resident Session Ceiling

    As long-session Agent traffic grows, the system’s binding constraint is also switching:

    • What was once the focus was inference throughput per unit time (how many tokens come out per second);
    • What increasingly matters now is the ceiling of resident sessions the system can support at once (how many sessions holding state without release it can keep alive simultaneously).

    This explains a counterintuitive phenomenon: even when token generation isn’t prominent during some period, as long as a large number of sessions are suspended waiting for tools to return, memory and storage remain occupied. Resources aren’t “computed away”—they are “stored.”

    To make this new constraint operational, it is advisable to collect at least the following measurable metrics (with clear definitions of methodology):

    • KV Cache size per session: estimated from model structure and seq_len or measured, with methodology as “average/peak GB occupancy per active session”;
    • Memory/VRAM level: real-time utilization of GPU VRAM and host memory, with methodology as “peak and average at 5-minute granularity”;
    • Active vs. suspended session count: distinguishing sessions being generated from those merely resident, with methodology as “count classified by session state”;
    • TTL hit rate/timeout eviction volume: reflecting cache lifecycle governance effectiveness;
    • Prefix cache hit rate: with methodology as “hit tokens/total input tokens”;
    • TTFT (time to first token): with methodology as “P50/P95/P99”;
    • Cost per token: total cost/billed tokens, with methodology that also notes VRAM and storage amortization.

    4. Rebalancing the Cost Structure: Actionable Optimization Paths

    Since the contradiction lies in “state residency,” optimization should also revolve around reducing the time × space cost of state occupancy.

    1. Prefix caching reuse: Agent tasks often share the same system prompt and preloaded documents. Persisting and reusing the KV Cache of the shared prefix across requests avoids re-running Prefill every time, reducing both compute and latency. Prerequisites include a stable cache key (e.g., based on content hashing) and a clear invalidation strategy. Note: Cross-request reuse must handle multi-tenant isolation properly—partition the cache namespace by tenant/session dimension, layer on access control and authentication, desensitize sensitive data before it enters the cache, and provide an auditable cache invalidation and deletion mechanism to prevent cross-user session state leakage.

    2. Tiered offloading: Place the KV Cache across VRAM, host memory, and external storage in tiers by access frequency, sinking cold data and keeping hot data resident, with paging management (e.g., PagedAttention-style approaches) to reduce VRAM fragmentation. Risk reminder: External storage offloading increases access latency, so the impact on TTFT must be assessed; consistency between tiers must also be ensured to avoid reading stale state.

    3. Session governance and lifecycle management: Set clear idle timeouts (TTL) and active eviction policies for suspended sessions to avoid “zombie sessions” occupying resources long-term; when the business allows, release rebuildable state promptly and keep only the minimum necessary context.

    4. Quantization and sparsification: Store the cache at lower precision (e.g., INT8/FP8), or sparsify less important attention, directly compressing per-token storage overhead. Risk reminder: Low precision and sparsification may cause accuracy loss, which must be evaluated within the business-acceptable quality threshold; consistency between quantized and full-precision computation should also be watched.

    5. Admission and scheduling policies: In multi-tenant scenarios, apply admission control and queuing to long-context requests based on real-time resource levels, preventing individual ultra-long sessions from squeezing overall capacity and making the new “resident session ceiling” constraint predictable and manageable.

    Implementing these strategies typically requires supporting infrastructure to work together: tiered storage handles state persistence and sinking, container platforms carry elastically scaling inference instances, monitoring and alerting systems observe VRAM and memory levels in real time, and logging services trace the resource-occupancy trajectory of long sessions. Taking a cloud provider’s object storage, container platform, monitoring and alerting, and other general capabilities as an example (general capability examples only, not a product recommendation or performance commitment), these components are exactly the base modules that can be combined when building a “state ledger” cost-optimization solution.

    Conclusion

    The rising share of open-source models marks a new phase in the competition over performance and pricing; but the real watershed lies in whether one can see clearly the structural mismatch of “revenue per token, cost per GB.” The “compute-for-token” model of traditional conversation is being rewritten by long-session Agent workloads, the cache is moving from backstage to center stage, and the constraint point is migrating from throughput to resident session capacity. Whoever manages the storage and time cost of state more finely—while holding the line on the data-security boundary of the cache—will win the next round of cost and profit space beyond the token price war.