Agent Cost Optimization: Reasoning Tiers and Cache-Aware Model Selection

Long-horizon coding, knowledge organization, and tool-use tasks no longer have the same cost structure as a one-shot question-and-answer interaction. The model repeatedly sees system instructions, repository snippets, historical tool results, and unfinished plans, so the context is read multiple times. At the same time, difficult problems require longer intermediate reasoning. Looking only at the listed input and output prices can easily lead to misjudgments: even if base unit prices stay unchanged, total spend can still shift significantly because of cache hits or reasoning tiers. The following sections explain how to choose models and settings by billing components, cache mechanics, reasoning tiers, and implementation steps.

Break the Bill into Four Computable Components

When billing is token-based, at least four usage categories should be distinguished rather than tracking only “how many calls were made.”

1. First input: tokens newly entering the context in the current turn that have not yet formed a reusable cache, billed at the input unit price.
2. Cache write: the additional cost of writing a reusable prefix into cache. Some billing models fold writes into input, while others price them separately. For model selection, use the actual bill line items; do not replace line items with a verbal “it is roughly cheaper.”
3. Cache read: the cost of reading a cached prefix when it is hit. In agent loops, this item is often the largest component.
4. Output: the final answer and visible intermediate results. If reasoning is billed as output or as separate “thinking tokens,” it must enter the model separately; otherwise, higher reasoning tiers will be systematically underestimated.

A reviewable per-call cost can be written as:

Cost ≈ new input tokens × input price + cache write tokens × write price + cache read tokens × read price + output tokens × output price + reasoning-related tokens × corresponding price.

Conclusions such as “it is 20% cheaper overall” hold only when the workload structure is fixed. Agents with a high cache-read share and very short outputs are sensitive to the read price; one-off long-form generation tasks that barely reuse prefixes are more sensitive to the output price and reasoning tier. When comparing two options, replay the same batch of requests rather than dividing mixed traffic from different weeks directly.

Why Cache Reads Determine the Agent Bill

Agents do not replace the entire context every turn. A typical loop places fixed system constraints and tool instructions first, followed by accumulated file snippets, retrieval results, and tool traces, and finally the incremental content for the current turn. A stable prefix with a growing suffix is exactly the shape that cache hits can exploit.

Cache hits usually require: a byte-stable prefix, the cache still being within its validity period, the request being routed to the same service scope that can reuse the cache, and the prefix length meeting the cache activation threshold. The following practices can silently destroy hit rates:

• Inserting timestamps, random numbers, or changing tracing fields at the very beginning of the system prompt each turn;
• Frequently changing the order, whitespace, or field names of tool definitions;
• Placing “the current user question” before a long document, preventing the long document from becoming a stable prefix;
• Estimating hit rate from total request counts when caches are not shared across replicas or regions;
• Truncating from the middle after a conversation becomes very long, so the prefix boundary changes every time.

In engineering terms, split the prompt into a “stable prefix” and an “incremental suffix.” Order the stable part from highest to lowest importance: safety and output constraints, tool contracts, project-level norms, and long-lived reference material. Put the current task, the latest error, and freshly returned tool results in the incremental part. This way, the most expensive and longest segment is the one being hit, rather than a few lines of polite boilerplate.

Caching is not free memory. Writes have a cost, expiry forces rebuilding, and a bad cache can repeatedly feed stale norms to the model. What suits caching is high-reuse, low-change, versionable content. What suits the incremental part is one-off logs, material containing secrets or personal information, and plan state that changes every turn. For sensitive data, confirm cache retention duration, isolation scope, and deletion policy; do not write content that should not be reused into a shared prefix just to improve hit rate.

When the read unit price drops, the benefit is approximately: hit token volume × (old read price − new read price). If the hit volume itself is small, absolute savings are limited. Therefore, measure hit rate before arguing whether it is “worth changing the prompt structure for caching.” Observability metrics should include at least: cache write tokens, cache read tokens, uncached new input tokens, hit rate (read / (read + cacheable prefix)), and the number of writes caused by prefix churn. Without these metrics, tier experiments cannot be separated from cache experiments.

Why Reasoning Effort Cannot Simply Be Turned Off

Many APIs divide reasoning into multiple tiers from low to high, for example five tiers, and do not allow reasoning to be completely disabled. This usually means the model always performs some internal problem solving; the caller can only adjust “how deeply to think,” not revert to pure reflexive short answers. The cost implication is that the lowest tier still has reasoning overhead, while the highest tier may increase thinking tokens, latency, and failure retries together.

The five tiers can be understood in terms of task risk rather than “the more expensive, the better”:

• Lowest tier: classification, extraction, format conversion, and rewriting answers that already exist. Failure cost is low and the output pattern is fixed.
• Second tier: routine code completion, short explanations, and single-step tool selection. Some judgment is needed, but the context already states the constraints clearly.
• Third tier: small multi-file changes, option comparison, and knowledge work that requires checking whether citations support the conclusion. This is the default tier for most daily work.
• Fourth tier: cross-module defect localization, decision recommendations after summarizing long documents, and multi-step tool planning. Errors propagate downstream.
• Highest tier: open-ended hard problems, highly ambiguous requirements, and scenarios where the cost of getting it right once is much higher than thinking a little longer. Limit the proportion of usage and set timeouts and budget caps.

As the tier rises, the common change is not that “the answer suddenly becomes correct,” but that intermediate search becomes more thorough, the model is more willing to self-check step by step, tool calls increase, and outputs become longer. If evaluation looks only at final correctness and not at tokens and latency, higher tiers will appear universally better; the bill and queueing time after launch will expose the problem.

Having no off switch does not mean every step must use a high tier. A long task can be split: use mid-to-high tiers for planning, low tiers for mechanical execution and format cleanup, and escalate only when validation fails or confidence is insufficient. Write escalation conditions into the policy, for example when tests fail, citations cannot be located in the context, or tool results contradict each other for two consecutive rounds. Unconditionally maxing out every round is equivalent to using the highest tier to “fuel” cache reads.

A Practical Selection Process

Step 1: Tag tasks by type. Single generation, multi-turn Q&A, agent loop, and batch processing. Only agents and long sessions should prioritize cache optimization; if every prompt in a batch is completely different, prioritize reducing output length and reasoning tier.

Step 2: Fix the evaluation set. Include at least successful samples, failure-prone samples, and very long-context samples. Record quality metrics (pass rate, number of defects in manual sampling) and cost metrics (the four token categories, end-to-end latency, retry count). Compare tiers on the same evaluation set to avoid online traffic drift.

Step 3: Stabilize the prefix before tuning tiers. The order cannot be reversed. While the prefix is still fluctuating, high-tier reasoning will spend more tokens on incorrect or redundant context, and the savings from caching will be eaten by retries. After the prefix is stable, start from the third tier: raise one tier if quality is insufficient, and try lowering one tier if quality meets the bar, until the quality curve flattens. Continuing to raise tiers after it flattens usually only increases cost.

Step 4: Set budget guardrails. Set a maximum reasoning tier, maximum tool rounds, and maximum output tokens per session or per task. When cache hit rate is high, people easily overlook that output and reasoning are still billed at high rates; long thinking combined with long answers can still cost more than “full-price input with a short answer.”

Step 5: Treat cache strategy and model retirement cycles separately. Model versions change, and cache keys are often bound to model version. Hit rate will temporarily drop on upgrade day; this is an expected cost, not an incident. Before switching versions, run a small-traffic replay, confirm that the prompt is still stable on the new version, and reserve time for cache warm-up.

The illustrative policy pseudocode expresses only control logic and does not correspond to any specific API:

prefix = stable_spec + tool_contract + repo_norms
suffix = task + fresh_tool_results
tier = default_mid
if task.risk == "format_only": tier = lowest
if task.risk == "cross_module": tier = high
for step in agent_loop:
    call(prefix, suffix, tier)
    if checks_failed and tier < max_tier: tier = tier + 1
    if budget_exceeded: stop_and_summarize
    move_unchanged_blocks_into_prefix()

Quality, Safety, and Cost Must Be Considered Together

Lowering reasoning can save money, but in vulnerability analysis, permission changes, data deletion, and similar actions, lower tiers are more likely to miss constraints. The right approach is to write “do not produce directly exploitable attack steps and do not perform unauthorized operations” into the stable prefix, and use deterministic checks (tests, policy engines, human confirmation) to block high-risk actions rather than expecting a higher reasoning tier to make the system safe automatically. A higher tier only increases the thinking budget; it does not replace access control.

The same applies to scientific or professional questions: benchmark scores show the model’s relative capability on a certain class of end tasks and cannot replace your own task set. When public benchmarks are published late and cover a narrow distribution, their scores are even less transferable to all business scenarios. The selection conclusion should be written as “on our replay set, the pass rate and per-task cost of a given tier,” with sample size and failure types attached.

A very long context is not automatically better. A large window means that incorrect material will also be repeatedly read and counted in the cache. Retrieve or summarize first, then put high-value snippets into the prefix; move stale logs out of the cache area. Set output limits by channel to prevent the model from writing far more explanation than needed at a high tier.

Knowledge cutoff dates, deployment regions, and available channels affect whether caches can be reused, whether data leaves the region, and when old versions become unavailable. These are all part of capacity and compliance planning: confirm version support windows in advance, complete replay and prompt regression before retirement, and avoid being forced into a full switch without cache warm-up after launch.

Conclusion

Base input and output unit prices are only part of the cost. Agent bills are amplified by the cache hit structure: when the prefix is stable, the read share is high, and changes in the read unit price directly change total cost; when the prefix is messy, writes and uncached input occur repeatedly, and tuning reasoning tiers cannot compensate. When reasoning has multiple tiers and cannot be turned off, the default tier should sit where most tasks just meet the quality bar. Escalate only for highly ambiguous, high-loss steps, and cap usage with budgets, round limits, and independent validation. Use a fixed replay set to examine pass rate, the four token categories, and latency together; only then can you prove both “cheaper” and “still correct.”