LLM Cost Optimization: Prompt Caching and Five-Tier Reasoning Selection

Once large language model applications enter production, cost optimization is no longer simply a matter of choosing a smaller model. Pricing structures are becoming more granular: standard input, cache reads, output, and reasoning levels can all affect the final bill. Looking only at input and output token prices can therefore lead to inaccurate cost estimates.

As cache-read prices fall and reasoning options are divided into multiple tiers, developers need a more complete cost model. In addition to per-request pricing, that model should account for cache hit rates, retries, the distribution of reasoning levels, and the cost of each successfully completed task.

This article addresses three questions:

• Why are cache reads becoming a key cost optimization variable?
• How should teams choose among five reasoning levels?
• How can caching, routing, and monitoring reduce production costs?

1. Understand the Cost Structure: Cache Reads Are a Separate Variable

The cost of a general-purpose LLM service can usually be divided into several components:

Input cost = Non-cached input tokens × Input price
           + Cache-read tokens × Cache-read price
Total cost = Input cost
           + Output tokens × Output price
           + Incremental cost from retries or reasoning-level upgrades

Standard input and output prices are often relatively stable, while cache reads are billed separately. A cache read occurs when the model reuses previously processed context, such as system prompts, tool definitions, long document sections, codebase instructions, or interaction history.

If this repeated content is billed as standard input on every request, costs can accumulate quickly. When it is served through a cache, the per-token price is typically much lower than the standard input price.

Suppose the cache-read price falls to one-quarter of the standard input price. If r is the proportion of cacheable input tokens and c is the cache-read price relative to the standard input price, the approximate input-side cost factor becomes:

Input-side cost factor ≈ (1 - r) + r × c

For example, if r is 80% and c is 0.25, the input-side cost factor is approximately 0.4. In other words, input costs fall by about 60%.

The reduction in total cost will also depend on output-token usage, retry rates, and the additional expense associated with reasoning levels. Under a typical workload, the overall reduction might be closer to one-quarter. Agentic applications, long-context workloads, and multi-step tool use often benefit more because they repeatedly send the same context.

The key point is that caching is not simply “free acceleration.” It is a cost-control capability that must be designed. Its value increases when cache hit rates are high, stable prefixes are long, and invalidations are infrequent.

2. Why Caching Matters for Agentic Workloads

Agentic workloads typically require multiple rounds of planning, tool execution, observation, and decision-making. Each request may include a system prompt, tool schemas, project rules, codebase excerpts, retrieval results, and prior interaction history. Much of this content remains unchanged over short periods and is therefore well suited to caching.

A code-repair task, for example, may follow this loop:

1. Read the project structure and coding rules.
2. Use tools to locate relevant files.
3. Read selected code sections.
4. Generate a proposed fix.
5. Run tests or static analysis.
6. Continue revising the code based on the results.

Throughout this process, the system prompt, tool definitions, project rules, and previously loaded code sections may be included repeatedly. With an appropriate caching design, later requests can read the stable prefix at the cache-read rate instead of paying the standard input rate again.

The same principle applies to long-document question answering, multi-turn customer support, compliance review, and developer assistants.

However, cache savings do not happen automatically. Several design choices can significantly reduce hit rates:

• Adding timestamps, random IDs, or dynamic user information to the system prompt
• Frequently changing tool definitions and making the prefix unstable
• Reordering the context on every turn
• Placing dynamic content before otherwise stable content
• Failing to define an invalidation policy when document or code versions change
• Failing to isolate data by tenant or permission scope, creating security risks or preventing safe reuse

The first rule of cache optimization is therefore: place stable prefixes first and dynamic content last.

Stable content—including system prompts, tool instructions, output-format constraints, and project rules—should appear near the beginning of the context. Dynamic content, such as the current user request, recent tool results, and temporary state, should appear later.

Cache keys should also be versioned. They may need to distinguish between projects, document versions, tool-schema versions, and permission domains. This prevents incorrect or unsafe reuse.

3. How to Choose Among Five Reasoning Levels

Reasoning options can be organized into five levels, from low to high. Higher levels generally allocate more computation to internal reasoning, planning, and verification. Quality may improve, but latency and cost also increase.

Some services do not provide an option to disable reasoning completely. In that case, even the lowest level has a baseline reasoning cost. It should not be treated as a free mode.

The five levels can be understood as follows:

Level	Typical tasks	Primary benefit	Primary risk
Level 1	Format conversion, simple classification, field extraction, fixed-template generation, and request routing	Low latency, low cost, and high throughput	More errors on complex or ambiguous tasks
Level 2	Routine summarization, RAG question answering, simple code completion, and basic information synthesis	Balanced cost and quality	Limited multi-step reasoning
Level 3	Multi-step tool use, complex code changes, option comparison, and general analysis	Handles moderately complex tasks	Higher cost; usage should be controlled
Level 4	Long-horizon planning, cross-file refactoring, rigorous technical analysis, and complex troubleshooting	Stronger planning and verification	Significantly higher latency and cost
Level 5	High-value tasks with low error tolerance, deep root-cause analysis, complex scientific reasoning, and critical decision support	Prioritizes quality for a small number of critical requests	Highest cost; unsuitable as the default for all traffic

Four principles can guide reasoning-level selection.

Start Low and Upgrade Based on Evaluation

Do not route every request to a high reasoning level by default. At the same time, do not rely on intuition alone.

Create a fixed evaluation set and compare levels using metrics such as:

• Accuracy
• Output-format compliance
• Human intervention rate
• P95 latency

A request should move to a higher tier only when evaluation shows that the quality improvement justifies the additional cost and latency.

Segment Tasks by the Cost of Errors

Tasks with a low cost of failure—such as classification, extraction, and summarization—can usually begin at a lower level.

Tasks with a high cost of failure—such as production change recommendations, compliance judgments, and complex code refactoring—may require escalation at critical stages.

Route Based on Task Decomposability

A complex workflow does not need to use the highest reasoning level from beginning to end. Lower levels can handle information extraction and candidate generation, while higher levels review only the critical parts.

This approach controls cost while preserving quality safeguards.

Define Dynamic Escalation Conditions

Rules or a lightweight classifier can perform the initial routing. The system can then escalate when it encounters conditions such as:

• Low confidence
• Repeated failure during multi-step tool use
• Invalid output formatting
• High-risk operations

The following code illustrates the routing concept:

def pick_reasoning_level(task):
    if task.type in {"format", "extract", "route"}:
        return 1
    if task.type == "rag_qa" and task.context_sufficient:
        return 2
    if task.requires_multi_step_tool_use:
        return 3 if task.error_cost < task.high_risk_threshold else 4
    if task.is_high_stakes_analysis:
        return 4
    if task.is_critical_decision and task.need_deep_verification:
        return 5
    return 2

This is only an example of routing logic, not an actual API call. A production system should also include budget limits, timeout-based fallback behavior, and human review mechanisms.

4. Production Implementation: Caching, Routing, and Monitoring

A practical cost optimization system should be built across four areas.

Stable-Prefix Engineering

Place system prompts, tool definitions, retrieval templates, coding standards, and output-format constraints at the beginning of the context. Manage these components with explicit versions.

Place dynamic user input, recent tool results, and temporary state later in the prompt. Avoid adding timestamps, random values, or frequently changing user information to the stable prefix.

Cache Keys and Invalidation

At a minimum, a cache key should reflect:

• Project identifier
• Document version
• Tool-schema version
• Permission domain

When a document or code version changes, the old cache should be invalidated according to a defined policy. Data from different tenants must remain isolated. A higher cache hit rate must never come at the expense of security.

Reasoning-Level Routing and Budget Guardrails

Define a default reasoning level and escalation conditions for each task type. Also configure:

• Maximum output length
• Retry limits
• Per-task budgets
• Timeout and fallback behavior

For batch workloads, lower levels can process the full batch before a higher level reviews a sample. For latency-sensitive tasks, define a downgrade path so that higher reasoning levels do not make end-to-end latency unpredictable.

Monitoring and A/B Testing

Monitor the following metrics continuously:

• Percentage of input tokens served from cache
• Share of cost attributed to cache reads
• Total tokens per task
• Distribution of reasoning levels
• P95 latency
• Task success rate
• Human intervention rate
• Cost per successful task

When building general-purpose LLM services on platforms such as Baidu AI Cloud, these metrics should be included in unified monitoring rather than tracking only total request volume.

A/B tests should use a fixed evaluation set. Compare quality, cost, and latency across reasoning levels, record failure cases, and regularly remove routing policies that no longer provide sufficient value.

5. Common Mistakes and Implementation Checklist

Common mistakes include:

• Looking only at the price per million tokens while ignoring cache hit rates
• Frequently changing the system prompt and invalidating the cache prefix
• Sending every request to the highest reasoning level
• Ignoring output-token and retry costs
• Using one reasoning level for classification, question answering, and complex planning
• Operating without an evaluation set, making it impossible to determine whether escalation improves results

Before deployment, use the following checklist:

• Have you identified and stabilized the cacheable prefix?
• Is dynamic content placed later to reduce cache pollution?
• Have you defined default reasoning levels and escalation conditions?
• Are you monitoring cost per successful task rather than only request volume?
• Do you have a fixed evaluation set for comparing reasoning levels?
• Are cache versions, invalidation, and permission isolation managed correctly?
• Are output length, retry count, and batch-task budgets controlled?
• Do you regularly review reasoning-level distribution and cost anomalies?

Cache reads and reasoning levels are not isolated pricing parameters. They are interconnected cost levers.

A high cache hit rate reduces the cost of repeated context. Appropriate reasoning-level selection prevents simple tasks from consuming unnecessarily expensive computation. By combining task segmentation, stable-prefix caching, dynamic escalation, and continuous evaluation, teams can keep the total cost of LLM services predictable without sacrificing required quality.