LLM API Pricing Explained: Token Costs, Prompt Caching, and Cost Reduction for Agents
As large language models (LLMs) move into production, API call cost is shifting from a post-hoc invoice line to a constraint that shapes architecture and model selection. This article breaks down how mainstream LLM services bill for input and output tokens, how falling mid- and low-tier prices change the economics of high-concurrency workloads, how prompt caching lowers effective input cost, why Agent workflows cost far more per completed task than single-turn chat, and how differentiated pricing (model tiers, speed modes, separate input/output/cache rates) shapes what providers offer. It closes with a practical playbook: routing requests by complexity, maximizing cache hit rates, containing context growth, and building token-level cost observability.
Why cost has become part of the architecture decision
Large language models (LLMs) are being adopted rapidly across enterprise applications, and API call cost is no longer just an invoice line item — it has become an upfront constraint on technical decisions. Two trends are running in parallel: prices for mid- and low-tier models keep falling, while capabilities aimed at Agents (智能体) keep improving. Together, they are reshaping both the cost structure and the engineering patterns that developers work with.
For developers, this means better price-performance, but it also means re-learning the billing rules before those savings can turn into real business value. This article takes a neutral view and systematically unpacks the billing logic of LLM APIs, caching mechanisms, differentiated pricing, and cost-reduction methods for Agent scenarios, along with practical recommendations you can act on.
1. The basics: how token-based API billing works
Most mainstream LLM services bill on a token basis. A token is the basic unit the model uses to process text — roughly half a Chinese character or one token’s worth of text, with the exact split determined by the tokenizer. To understand the bill, you need three dimensions:
- Input tokens: tokens consumed by the prompt, the context, and the system instructions you pass in.
- Output tokens: tokens consumed by the content the model generates and returns.
- Billing granularity: pricing is usually quoted per million tokens, with separate rates for input and output.
A consistent pattern is that the output rate is higher than the input rate, because generation involves step-by-step autoregressive decoding, which costs more compute. Schematically, the unit price structure abstracts to:
cost per call ≈ (input tokens / 1,000,000) × input price+ (output tokens / 1,000,000) × output price
It’s also worth noting that many services introduce tiers — flagship, mid-tier, and lightweight, based on model capability. The lower the tier, the cheaper the per-token price, which suits simple tasks at high concurrency; the higher the tier, the stronger the reasoning and complex-task capability, and the higher the price. This tiering is itself the foundation for the “differentiated pricing” discussed later.
2. Mid-tier price cuts and scaling high-concurrency workloads
The falling prices of mid- and low-tier models are not simply a price war; they are a structural optimization of the “call volume — cost” relationship. In real workloads, a large share of requests are structured, predictable tasks: text classification, intent recognition, field extraction, short question answering, and content moderation. These tasks don’t demand deep reasoning, yet they often account for the majority of total call volume.
Once the per-token price of mid- and low-tier models drops, the marginal cost of high-concurrency scenarios narrows noticeably. Take customer-service intent recognition at tens of millions of calls per month: the saving per request looks trivial, but multiplied by the call volume, the reduction in total cost is substantial. That’s why price cuts have a far greater impact on high-volume businesses than on low-frequency, high-value scenarios.
For engineering teams, the real question isn’t “should we use cheaper models” but “how do we build a clear tiered routing strategy”: route requests to different model tiers based on their complexity, so simple requests go down a lightweight path and only complex requests are escalated to a flagship model. Logically, this amounts to a cost-aware routing layer.
3. Prompt caching: the overlooked cost lever
In long-context, fixed-system-instruction, or multi-turn conversation scenarios, the prefix of a request — system prompt, tool definitions, long document background — is often highly repetitive. To address this, the industry broadly supports prompt caching: when the input prefix matches previously cached content, that portion can be billed at a steep discount, typically well below the standard input rate charged for full computation.
The economics of caching abstract to:
cache hit:effective input cost = cacheable prefix × cache read price+ new content × standard input pricecache miss:effective input cost = full input × standard input price
The cache read price is generally lower than the standard input price, so wherever a fixed prefix makes up a large share of the input, caching yields meaningful savings. A few points matter in implementation:
- Prefix stability: caching depends on a deterministic input prefix. Put volatile content (timestamps, random IDs) at the end of the prompt and keep stable content at the front.
- Invalidation management: caches usually have a time-to-live (TTL), so weigh how often the prefix changes against how long the cache stays valid.
- Break-even estimation: caching pays off significantly only when the repeated prefix is long enough and reused often enough.
A sketch of how to structure this:
messages = [{ "role": "system", "content": STABLE_INSTRUCTIONS }, # stable prefix, favors cache hits{ "role": "system", "content": LONG_BACKGROUND_DOC }, # long background, cacheable{ "role": "user", "content": request_specific_query } # volatile part, placed last]
4. Agent upgrades and total cost of ownership
Agents are the center of the current cost discussion. Unlike conventional single-turn question answering, an Agent typically needs multiple rounds of reasoning, tool calls (Function Calling / Tool Use), reflection on intermediate results, and result aggregation to complete a task. This means one “task” may sit behind many model calls and several accumulated context segments, so the total cost is far higher than a single exchange.
Consequently, the economics of an Agent can’t be judged by unit price alone; they should be measured as the total cost of completing one task. The main factors:
- Number of rounds: the finer the task decomposition, the more rounds, and the larger the accumulated token consumption.
- Context growth: every round carries the conversation history forward, so input tokens rise quickly with round count unless the history is trimmed.
- Output length: verbose intermediate reasoning and explanations inflate output cost.
- Framework fit: a good Agent orchestration framework shortens the execution path through context compression, tool-result summarization, and fast fallback on failures.
One direction of recent model upgrades is precisely better fit with Agent frameworks: more structured output, more reliable tool calling, and more compact intermediate representations reduce pointless round trips and redundant output, compressing total cost without sacrificing capability. For developers, this means combining “pick the right model” with “orchestrate it well” — only the two together genuinely reduce cost.
5. Differentiated pricing: balancing scale against margin
Purely low prices tend to lead to commoditization, so vendors increasingly lean on differentiated pricing, covering different needs with different product combinations. Common mechanisms include:
- Tier price gaps: keeping a several-fold price difference between flagship and mid/low tiers, so callers choose what they need.
- Speed tiers: offering a “fast mode” and a “standard mode”, where fast mode buys lower latency at a higher unit price — suited to interaction-sensitive scenarios.
- Input/output separation: pricing input, output, and cache independently, so billing tracks actual resource consumption more closely.
The value of differentiated pricing is that a platform can serve large-scale, low-margin general-purpose calls while capturing premium margins through high-value tiers, balancing scale against profitability. For developers, the rational approach is to first pin down the workload’s latency sensitivity, task complexity, and call volume, then match tiers and modes accordingly — rather than defaulting to either the highest tier or the lowest price.
6. Selection and cost-reduction practices for developers
Bringing the principles above together, here is an actionable cost-reduction path:
- Establish request tiering and routing. Tag requests by complexity: lightweight requests go to a low-price tier, complex ones are escalated to a flagship model. Example logic:
if task_complexity == "low":route_to(lightweight_model)elif task_complexity == "medium":route_to(mid_tier_model)else:route_to(flagship_model)
- Prioritize reusing cacheable prefixes. Front-load stable instructions and long background so the cache hit rate is maximized.
- Optimize Agent orchestration. Control context length, compress tool return values, and avoid unnecessary multi-round round trips.
- Build cost observability. Use logging services and monitoring/alerting to break down token consumption by business line and by tier, and locate cost hotspots.
- Lean on general-purpose cloud platform capabilities. Object storage can manage long documents, message queues can smooth traffic spikes, container platforms can scale elastically, and monitoring/alerting can track call volume — making cost-reduction strategies sustainable in engineering terms. Model services from mainstream cloud platforms such as Baidu AI Cloud typically offer general-purpose capabilities in tiering, cache support, and usage monitoring, which can serve as reference dimensions when evaluating options.
Conclusion
Falling prices for mid- and low-tier models and the upgrade of Agent capabilities are, in essence, pushing the boundaries outward on two axes at once: price-performance and capability ceiling. Lower prices let more high-concurrency scenarios reach production scale, while differentiated pricing and caching turn cost reduction from “passively saving money” into “actively optimizing”.
For developers and enterprises, the real lever isn’t chasing any single number — it’s understanding the billing logic and building a system of tiered routing plus cost observability. When engineering capability meets cost awareness, LLM applications can scale while keeping unit economics sustainable.
