Cost Optimization for LLM Agents: Tiered Routing and Caching

    This article explains how to reduce LLM agent costs under token-based pricing by combining tiered model routing with context caching. It covers cache break-even calculations, routing thresholds by task difficulty, stable cache key design, timeout-based fallback, and operational metrics for validating cost and quality.

    The cost of an LLM agent mainly comes from three parts: input tokens per request, output tokens generated by the model, and repeated billing when the same context is sent repeatedly. Recently, several vendors have adjusted their price lists. Unit prices for lightweight models and cache reads have decreased, while flagship models still carry a higher premium. Developers need to solve two problems: assign models by task difficulty and make repeated content hit the cache as often as possible. This article provides formulas, thresholds, and fallback flows that can be implemented directly. It mainly targets cloud model APIs billed by token. Self-deployed models, multi-tenant strong-privacy scenarios, streaming output, and long tool-call chains require separate evaluation.

    1. How to Allocate Models and Caching After Pricing Changes

    1.1 Unit Prices for Lightweight Models and Cache Reads

    Some vendors have lowered input and output unit prices for lightweight models in public price lists, and have also reduced billing for context cache reads. The reduction varies by model, region, usage tier, and billing basis; some cuts are large, while others are small. Cost calculations should therefore use the official price list for the day and should not reuse percentages from other articles.

    Cache billing usually has two parts: write and read. The write price is generally slightly higher than ordinary input, and the read price is lower. If a context is used only once, caching may not save money. Only when it is reused multiple times can the savings from reads offset the extra write cost.

    Let Pi be the ordinary input price, Pw the write price, Pr the read price, and n the number of calls to the same context. Without caching, the cost is n × Pi. With caching, the cost is Pw + (n − 1) × Pr. Setting the two equal gives the break-even point:

    n = (Pw − Pr) ÷ (Pi − Pr)

    Example (hypothetical price ratios, not an actual price list): if the write price is 1.25× ordinary input and the read price is 0.1× ordinary input, then n = (1.25 − 0.1) ÷ (1 − 0.1) ≈ 1.28. The same context must be used at least twice before caching is cheaper than not caching.

    1.2 Premium for Flagship Models and Fast Mode

    Flagship model unit prices are usually significantly higher than lightweight models. Some flagship models also offer a fast mode that trades a higher unit price for lower response latency. The premium ratio depends on the specific vendor, model, and service tier; consult the corresponding documentation before integration.

    Whether fast mode is worth enabling depends on two measured numbers: the reduction in P95 latency and the increase in cost per request. Pay for it only when the latency benefit translates into a better user experience. Fast mode is suitable for real-time user-facing interaction. Offline batch jobs usually gain no benefit and should not enable it by default.

    1.3 Framework Fit and Cost Measurement

    A model’s fit with common agent frameworks such as ReAct and Plan-and-Solve affects tool-call count, chain-of-thought length, and output format. These factors all change token consumption. Whether a new model truly reduces cost must be validated on your own task set, not only from vendor claims.

    Use the following formula for cost measurement. Normalize all unit prices to per million tokens, so the result is divided by 1,000,000:

    Cost per request = (input tokens × input price + cache write tokens × write price + cache read tokens × read price + output tokens × output price) ÷ 1,000,000

    For the experiment, select 100 to 300 real requests and run them on the new and old models or configurations. Record the token counts by type, latency, and task success for each request. In addition to average tokens, calculate the following metrics:

    • Cache hit rate: the number of requests that hit the cache divided by all requests.
    • Routing error rate: the share of requests handled by a lightweight model that later need escalation.
    • Degradation rate: the share of requests that trigger timeout fallback.
    • Cost per successful task: total cost ÷ number of successful tasks.

    2. Thresholds for Tiered Model Routing

    Split requests by task difficulty into three tiers:

    • L1 lightweight: classification, information extraction, format conversion, and keyword tagging. Outputs can be validated with rules.
    • L2 standard: multi-step summarization, routine Q&A, and limited tool calls.
    • L3 flagship: complex reasoning, code modification, and high-reliability outputs involving money or compliance.

    The following thresholds are examples and need calibration on your own evaluation set:

    1. If the task type is L3, route directly to the flagship model.
    2. If the input is no more than 4,000 tokens and tool calls are no more than 2, route to L1.
    3. Route other requests to L2.
    4. If the historical failure rate for a task class exceeds 5%, raise that task class by one tier.

    The evaluation set should be stratified by task type and cover long-tail inputs and historical failure cases, not only simple requests. For each threshold set, compare success rate and cost, and choose the lowest-cost option that meets the quality floor.

    Fallback conditions also need to be explicit. If lightweight output fails validation—for example, JSON parsing fails, required fields are missing, or confidence is below the threshold—retry once. If it still fails, escalate to L2 or L3. Count every escalation in the routing error rate so thresholds can be adjusted later.

    3. Designing Stable Cache Keys

    A cache key consists of the following parts: model name and version, system prompt version number, hash of tool definitions, hash of static prefix content, and tenant identifier.

    Prefix stability is the key to cache hits. Put invariant content first, including the system prompt, tool descriptions, and fixed examples. Put dynamic content later, such as the user question, retrieval results, and current time. Do not insert timestamps, random IDs, or lists whose order changes per request into the prefix; otherwise the cache will be invalidated every time.

    When the prompt or tool definitions change, update the version number at the same time so the old cache is invalidated. Cache needs an expiration time, set according to the business update cadence; it should not be retained indefinitely.

    Privacy rules must be written into the design document. Do not put phone numbers, email addresses, ID numbers, access tokens, or API keys into the cache key or cached content. If cache isolation by user is required, use tenant identifiers generated by salted hashing; store the salt in a key management service, not in code or logs.

    4. Timeout and Fallback Method

    Set two timeouts for each model tier: a first-token timeout for streaming scenarios, and a total timeout for all scenarios. The threshold should be the P95 latency for that model tier on the evaluation set plus a margin. For example, if a task’s P95 on L3 is 12 seconds, the total timeout can be set to 18 seconds; use measured values for the specific number.

    Retry only idempotent requests, at most once, with random backoff before the retry. For non-idempotent tool calls such as database writes, sending messages, or charging, after a timeout first query the execution status, confirm the result, and then decide how to handle it; do not retry directly.

    Fallback order: after an L3 timeout, fall back to L2; after an L2 timeout, fall back to L1; if L1 still fails.