Exploring the Rising Cost of LLM Inference: Cache Architecture and the Economic Dynamics of Agent Workloads

9 hr ago0 views

This article examines the recent price surge in large language model API services, which now charge up to ¥100 per million output tokens, ¥20 per million uncached input tokens, and only ¥1–2 per million cached input tokens. By analyzing the underlying cost structure, we explain how the shift to disaggregated inference architectures and KV cache reuse has created a pricing model that heavily favors high-cache-hit, long-context enterprise workloads while penalizing low-reuse, ad-hoc queries. We also provide practical engineering guidance—including prompt prefix stabilization and dynamic content placement—to help developers optimize costs in agent-based applications.

Understanding the Recent Price Restructuring in LLM APIs

Recent adjustments to LLM API pricing have drawn significant attention. Under the latest billing rules, output tokens are priced at ¥100 per million, uncached input tokens cost ¥20 per million, and cached input tokens are priced at just ¥1–2 per million. Compared to the previous generation, output prices have risen nearly fourfold, while uncached input prices have increased roughly threefold.

On the surface, this appears to be a broad price hike. However, when viewed in the context of the agent era and the evolution of disaggregated inference architectures, it becomes clear that this is a deliberate, granular pricing strategy designed to align costs with actual computational resource consumption.

Key Pricing Ratio Shifts Reflect Architectural Changes

A closer look at the pricing ratios between the old and new model generations reveals important signals:

In the previous generation, the ratio of uncached input price to cached input price was about 6:1, and the ratio of output price to cached input price was about 25:1.
In the current generation, these ratios have widened to approximately 10:1 and 50:1, respectively.

The cached input tier saw the most moderate increase, while output and uncached input tiers rose sharply. This asymmetric adjustment is not arbitrary—it reflects fundamental changes in how inference workloads are executed.

LLM inference consumes significant memory and bandwidth, particularly for key-value (KV) cache storage and decoding operations. As context length grows, these requirements increase linearly or even super-linearly. In traditional fused architectures, prefill and decode operations are coupled, leading to memory fragmentation and poor efficiency for long contexts.

With disaggregated inference architectures, prefill and decode are decoupled across separate compute nodes. This allows high-value prefix tokens to be cached independently and reused across multiple requests. In one reported programming scenario, a cloud provider achieved cache hit rates above 90% using this approach.

The aggressive pricing differentials are therefore a mechanism to manage underlying compute costs: requests that efficiently reuse cached prefixes receive substantial discounts, while requests that bypass the cache and force full recomputation incur premium charges.

Real-World Cost Analysis in Agent Scenarios

To understand the real impact of this pricing model, we must look beyond single-turn conversations. In the chatbot era, a single interaction consumed only a few thousand tokens, making price sensitivity low. In the agent era, however, a single task can involve thousands of tool calls and hundreds of thousands of tokens, with autonomous execution spanning multiple days in some parallel workloads.

Scenario 1: Enterprise-Grade Coding Agent with Long Context

Consider an enterprise agent with an input-to-output token ratio of approximately 20:1—common in scenarios that require carrying extensive code history and documentation. Thanks to fixed system prompts and long-prefix reuse, the cache hit rate reaches 90%.

Effective input price per million tokens ≈ ¥2 × 0.9 + ¥20 × 0.1 = ¥3.8
Output price per million tokens remains ¥100
Blended cost per million tokens ≈ (20 × ¥3.8 + 1 × ¥100) / 21 ≈ ¥8.4

Scenario 2: Occasional Developer with Low-Cache Reuse

Now consider a developer who calls the API sporadically, with each prompt different from the last and no reusable prefix. The cache hit rate drops to 20%, and the input-to-output ratio is 5:1.

Effective input price per million tokens ≈ ¥2 × 0.2 + ¥20 × 0.8 = ¥16.4
Output price per million tokens remains ¥100
Blended cost per million tokens ≈ (5 × ¥16.4 + 1 × ¥100) / 6 ≈ ¥30.3

The contrast is stark: the enterprise scenario with high-frequency reuse costs about 3.6 times less per million tokens than the low-reuse developer scenario. This highlights the core logic of the new pricing structure. It is not a flat price increase, but a strategic discount for enterprise clients who maintain consistent system prompts and high prefix reuse, while occasional users with unpredictable prompts pay the full—or even premium—rate.

Under the previous pricing model, running modern agent workloads that consume hundreds of thousands of tokens would result in guaranteed losses on compute and bandwidth for the cloud provider.

Engineering Countermeasures: Maximizing Cache Efficiency

Given this pricing environment, development teams must adjust their architectural approach to reduce costs through higher cache utilization.

1. Fix and Standardize Prompt Prefixes

When building agents, place system-level instructions, background knowledge, and tool definitions at the very beginning of the prompt. Ensure these parts are character-for-character identical across all requests. Even minor changes—including extra spaces or line breaks—can invalidate the cache and trigger expensive uncached billing.

2. Place Dynamic Content at the End

Keep variable elements such as real-time user input and dynamically retrieved RAG content in the latter part of the prompt. This preserves the stability of the prefix, maximizing cache hits.

# Example: Recommended prompt construction order
prompt = [
    {"role": "system", "content": FIXED_SYSTEM_INSTRUCTION},  # must be identical every time
    {"role": "user", "content": DYNAMIC_RETRIEVED_CONTEXT},   # dynamic, changes per request
    {"role": "user", "content": CURRENT_USER_QUERY}           # current query
]

3. Use Cloud-Native Infrastructure for Session State Management

In complex distributed agent systems, maintaining long-context KV caches requires substantial storage. Development teams can integrate with general-purpose object storage or container platforms to build persistent session state management modules. By storing and verifying hashes of frequently used agent prefixes externally, teams can check cache viability before sending requests to the LLM service. For tasks with low hit probability, preprocessing with a local smaller model or truncation can reduce the cost of expensive uncached calls.

Conclusion

The recent surge in LLM inference prices represents a shift from simple per-token billing to a more refined model based on actual compute consumption. Disaggregated inference architecture has turned the KV cache into a quantifiable, reusable asset. For developers and enterprises, understanding this shift is not just a matter of cost savings—it is essential for designing efficient agent-based applications.