LLM API Pricing: The Cost Economics Behind Prompt Caching
Higher LLM API prices do not necessarily translate into proportionally higher application costs. This article examines how KV cache reuse changes inference economics, compares effective costs for cache-heavy coding agents and low-reuse workloads, and explains how prompt structure and workflow design can help developers control spending.
A recent pricing announcement for a new generation of a large language model sparked debate among developers. The published rates were RMB 100 per million output tokens, RMB 20 per million uncached input tokens, and RMB 1 or RMB 2 per million cached input tokens. Compared with the previous generation, output pricing had risen to nearly four times its earlier level, while uncached input pricing had increased to roughly three times its previous level.
For many developers, the immediate reaction was that the model had become too expensive. Yet evaluating an API solely by its headline rates can obscure the inference architecture and economics behind its pricing.
To understand the actual impact, developers need to examine both the cost of processing different types of tokens and the proportion of their workload that can benefit from caching.
1. How Inference Architecture Shapes API Pricing
LLM inference is computationally intensive. Without cross-request cache reuse, processing repeated input prefixes requires recalculating their key-value cache, or KV cache. This duplicates computation when requests share substantial context.
One approach to reducing this waste is a disaggregated inference architecture that separates compute resources from cache storage. When processing a request, the system checks a shared KV cache pool for a matching prefix. If it finds a match, it can reuse the previously computed context rather than recomputing that portion of the input. A new, uncached prompt requires the corresponding input computation.
Cache reuse reduces computation, although serving a cache hit still involves retrieving cached state and performing the remaining inference work. Its economic advantage comes from avoiding repeated processing of the same input prefix.
A model team’s technical blog reported cache hit rates above 90% in coding workloads using a disaggregated inference architecture. Such a high level of reuse can materially change the cost structure: if 90% of input tokens are served from cache, substantially less input processing is required.
The distinction between cached and uncached pricing therefore reflects differences in resource consumption, rather than treating every input token as equally expensive. Headline price increases alone do not reveal the resulting cost for a particular application.
2. A Cost Model: How Cache Hit Rates Change the Bill
A simple weighted-average model illustrates the impact of caching:
Effective input price =(Cached input price × Cache hit rate)+ (Uncached input price × (1 − Cache hit rate))
For this calculation, the cache hit rate represents the share of input tokens billed at the cached rate. The examples below use RMB 2 per million cached input tokens, RMB 20 per million uncached input tokens, and RMB 100 per million output tokens.
Scenario 1: An Enterprise Coding Agent with High Context Reuse
A coding agent working with long contexts may repeatedly send substantial repository context and stable system instructions. Assume an input-to-output token ratio of 20:1 and a cache hit rate of 90%.
- Effective input price:
2 × 0.9 + 20 × 0.1 = RMB 3.8 per million input tokens - Output price:
RMB 100 per million output tokens - Blended cost across input and output tokens:
(20 × 3.8 + 1 × 100) / 21 ≈ RMB 8.4 per million total tokens
Despite the high output rate, the workload’s large volume of cached input brings down its blended cost.
Scenario 2: An Individual Developer with Low Context Reuse
Consider an occasional API user whose prompt structure changes substantially between requests. Assume a cache hit rate of 20% and an input-to-output token ratio of 5:1.
- Effective input price:
2 × 0.2 + 20 × 0.8 = RMB 16.4 per million input tokens - Output price:
RMB 100 per million output tokens - Blended cost across input and output tokens:
(5 × 16.4 + 1 × 100) / 6 ≈ RMB 30.3 per million total tokens
| Metric | High-Reuse Coding Agent | Low-Reuse API Workload |
|---|---|---|
| Cache hit rate | 90% | 20% |
| Input-to-output token ratio | 20:1 | 5:1 |
| Effective input price, RMB per million tokens | 3.8 | 16.4 |
| Output price, RMB per million tokens | 100 | 100 |
| Blended cost, RMB per million total tokens | 8.4 | 30.3 |
Under these assumptions, the coding agent’s blended unit cost is less than one-third that of the low-reuse workload. The difference reflects both cache reuse and the input-to-output ratio, not caching alone.
The pricing structure effectively rewards applications that repeatedly use stable system prompts and shared context. Workloads with frequently changing prefixes receive much less benefit from the cached-input discount.
These calculations compare two workload profiles under the same pricing schedule. They do not establish whether either workload is cheaper than it was under the previous generation’s pricing.
3. The Business Logic Behind Cache-Based Pricing
Differentiated cached-input pricing suggests two broader commercial priorities.
From Broad User Acquisition to Sustained Enterprise Usage
Early pricing strategies often emphasize attracting developers through very low rates or free access. As model capabilities mature, providers have an incentive to focus on enterprise agent workloads that generate recurring demand.
Architectural improvements can make those workloads more economical to serve. Lower cached-input rates pass some of that advantage to customers whose applications consistently reuse context.
Better Utilization of Shared Infrastructure
High cache hit rates can reduce repeated GPU computation as well as customer bills. Pricing creates an incentive for developers to structure requests in ways that reuse existing context.
This alignment can improve overall cluster utilization and throughput: developers avoid paying for unnecessary input processing, while providers avoid performing it repeatedly.
4. Practical Ways to Reduce API Costs
To benefit from cache-aware pricing, developers should treat prompt organization and workflow design as part of application architecture.
Keep Prompt Prefixes Stable
Place relatively static content—such as system instructions and few-shot examples—at the beginning of the input sequence. Keep that content consistent across requests.
Unnecessary changes near the start of a prompt can prevent reuse of the matching prefix that follows. A useful organizing principle is:
Stable prefix + Dynamic suffix
This structure keeps reusable context ahead of request-specific content.
Group Similar Tasks
Where practical, organize similar tasks into batches or coordinated workflows that reuse shared context.
The objective is not simply to submit more requests together, but to preserve common input prefixes so that repeated work can benefit from caching.
Use Cloud Infrastructure to Manage Context Consistently
Complex enterprise applications can use general-purpose cloud services—such as container platforms, object storage, and message queues—to build context-management middleware. Services of this kind, including those offered by Baidu AI Cloud, can support the surrounding application architecture, subject to the relevant service availability.
For example, an application can retain conversation history in low-latency cloud storage and assemble prompts consistently using the stable-prefix, dynamic-suffix pattern.
The storage layer does not itself guarantee a model-side KV cache hit. Its role is to help the application manage context and construct requests consistently, improving the opportunity for reuse under the API’s caching rules.
5. Conclusion
LLM API pricing changes are not simply a matter of higher numbers on a rate card. They can also reflect evolving inference architectures and a commercial shift toward workloads that reuse substantial amounts of context.
KV cache reuse changes the economics of input processing, making effective costs highly dependent on cache hit rates and the balance between input and output tokens. A cache-heavy coding agent and an occasional, low-reuse API workload can therefore face very different blended costs on the same model.
For developers, the practical response is to evaluate actual workload characteristics rather than headline prices alone. Stable prompt prefixes, reusable context, and carefully structured workflows are increasingly important tools for controlling LLM application costs.
