Cost Is Becoming a Core Variable in Model Selection

For some time, competition among large language models focused mainly on maximum capability. More recently, prices for mid- and lower-tier models have fallen significantly, while prompt caching has also become less expensive. Price gaps within the same model family are widening as well.

For developers, model selection is shifting from “Which model is the most powerful?” to “Which model delivers the best results for each unit of task cost?”

1. Three Major Pricing Changes

1. Lower Prices for Mid- and Lower-Tier Models

Mid- and lower-tier models are well suited to high-volume, structured, latency-sensitive tasks such as text classification, information extraction, format conversion, and simple question answering.

These tasks generally do not require the strongest reasoning capabilities, but they can generate a large number of API calls. As prices decline, use cases that were previously too expensive to operate at scale—such as preprocessing every support ticket, summarizing logs, and performing batch content moderation—become more practical to deploy.

2. Lower Prompt Caching Costs

Prompt caching allows a service to retain the computed results of repeated prompt prefixes. When a later request reuses the cached content, the matching tokens are billed at a lower rate.

Fixed prefixes such as system prompts, long documents, and tool definitions benefit most from caching. As cache read prices fall, the engineering practice of placing stable content at the beginning of a request translates more directly into cost savings.

3. Tiered Pricing

Service providers typically offer different prices for different model tiers. Input and output prices for premium models may be dozens of times higher than those for mid- and lower-tier models. Some services also provide a fast mode that charges more in exchange for lower response latency.

The purpose of tiered pricing is to allocate resources according to task value:

• Use premium models for high-value, infrequent tasks.
• Use lower-cost models for high-volume, routine workloads.
• Use fast modes for interactive steps where latency is especially important.

2. Why AI Agents Are More Sensitive to Cost

AI Agents have a different cost structure from ordinary question answering. A typical question-answering request consists of one input and one output. An Agent, by contrast, may plan, call a tool, read the result, and plan again in a loop. At every step, the accumulated context may need to be sent back to the model.

Three factors amplify costs:

• Number of calls: A task may require anywhere from a few to dozens of model calls. Total cost is roughly the cost of one call multiplied by the number of steps.
• Context accumulation: Each round may include previous conversation history and tool results, causing input token usage to grow as the task progresses.
• Output length: Reasoning content, structured tool arguments, and the final response all count as output tokens. Output pricing is often higher than input pricing.

As a result, reducing the cost of each step is just as important as eliminating unnecessary steps.

3. How to Estimate Costs

Building a cost model before deployment helps prevent estimates based only on intuition. The core formula is:

Cost per task = steps × (
    new input tokens × input price
    + cached input tokens × cached input price
    + output tokens × output price
)

The following example uses assumed prices for demonstration only. They do not represent the pricing of any specific service and should be replaced with the applicable current rates.

from dataclasses import dataclass
@dataclass
class PriceTier:
    input_per_m: float         # Price per million new input tokens
    cached_input_per_m: float  # Price per million cached input tokens
    output_per_m: float        # Price per million output tokens
def estimate_task_cost(tier: PriceTier, steps: int,
                       fresh_input: int, cached_input: int,
                       output: int) -> float:
    """Estimate the cost of a single Agent task.
    fresh_input, cached_input, and output are average tokens per step.
    """
    per_step = (
        fresh_input * tier.input_per_m
        + cached_input * tier.cached_input_per_m
        + output * tier.output_per_m
    ) / 1_000_000
    return steps * per_step
# Assumed prices in yuan per million tokens, for demonstration only
mid_tier = PriceTier(input_per_m=2.0, cached_input_per_m=0.4, output_per_m=8.0)
# 2,000 new input tokens, 20,000 cached input tokens,
# and 500 output tokens per step, across 10 steps
with_cache = estimate_task_cost(mid_tier, steps=10,
                                fresh_input=2000, cached_input=20000, output=500)
# The same task with no cache hits:
# all 22,000 input tokens are charged as new input
no_cache = estimate_task_cost(mid_tier, steps=10,
                              fresh_input=22000, cached_input=0, output=500)
print(f"With cache: {with_cache:.2f} yuan")   # Approximately 0.16 yuan
print(f"Without cache: {no_cache:.2f} yuan")  # Approximately 0.48 yuan

In this example, the same 10-step task costs approximately CNY 0.16 with cache hits and CNY 0.48 without them—a difference of roughly three times.

This shows that, for Agent workloads, cache design can sometimes deliver greater savings than simply switching to a cheaper model.

4. Engineering Strategies for Cost Optimization

1. Route Tasks Across Model Tiers

Use rules or a lightweight classifier to assess task difficulty. Assign simple tasks to mid- or lower-tier models, while sending complex reasoning and high-risk decisions to premium models.

Routing rules should be based on offline evaluation rather than price alone. A practical approach is to record the accuracy and pass rate of each model tier on an evaluation set, then calculate the cost per correct result. This metric can guide model selection.

2. Design Stable Prefixes for Caching

Place the system prompt, tool definitions, and fixed knowledge at the beginning of the message. Put dynamic content at the end.

If fields such as timestamps or request IDs change on every request and appear inside the prefix, the cache may not be hit. Reordering the template often requires only a small implementation change while producing meaningful cost savings.

3. Trim and Summarize Context

For conversation history, use a sliding window or generate a summary at the end of each stage. Retain only the information relevant to the current step.

Tool responses can also be large. Filter fields externally before sending the result back to the model, and include only the data required for the next decision.

4. Control Output Length and Step Count

Set a maximum number of Agent steps and a maximum output token limit for each step. Add a circuit breaker for repeated calls to the same tool with the same arguments to prevent loops.

Output formats should be explicitly constrained to structured results, reducing unnecessary explanatory text.

5. Use Fast Modes Selectively

Fast modes reduce latency but typically cost more. They are most appropriate for interactive steps where a user is waiting for a response.

Background jobs, batch processing, and offline analysis should use standard modes whenever latency is not a priority.

5. Risks and Limitations

Lower prices do not mean that lower-tier models can safely handle every task. They generally have higher failure rates on long-chain reasoning, complex tool orchestration, and edge cases. Retries after failures can offset the savings from a lower per-call price.

Before deployment, run regression evaluations and validate outputs programmatically. For example, use a JSON Schema to verify structure, followed by business rules to check value ranges.

Relying on a single provider also creates risks related to price changes and service interruptions. Encapsulating model calls behind a unified interface makes it easier to switch models and conduct A/B comparisons.

When user data is involved, verify that data retention and privacy terms meet internal compliance requirements.

6. Deployment Checklist

1. Classify Agent tasks and record the average number of steps and token distribution for each category.
2. Build a cost estimation sheet and calculate cost ranges under different cache hit-rate assumptions.
3. Configure candidate models for each task category and compare accuracy with unit cost on the same evaluation set.