Scaling AI Agents: Practical Per-Task Cost Optimization with Low-Cost Models

1. The signal has changed: from scores to cost per task

Recently, a vendor released a model positioned as “lightweight and high-throughput,” with public pricing of $0.10 per million input tokens, $0.004 per million cached input tokens, and $0.50 per million output tokens. A third-party evaluator reported that at the highest reasoning effort level, the model’s overall score was about 35.8, essentially tied with a comparable competitor at 36.2, while its per-task cost was only about $0.14, roughly one-third of the comparison target. Another evaluator listed it as the lowest per-task cost in the tested sample, about $0.15, with accuracy still above 60%.

These numbers will change as models are updated and should not be used as procurement criteria. The trend is worth attention: when baseline capability converges, competition shifts from “who is smarter” to “who is smart more cheaply.” The reason is direct: in chat, one request consumes one inference, while agent scenarios are long chains of multi-round tool calls, self-verification, and failure retries. Token consumption can be tens of times higher than ordinary dialogue. For an always-on agent, cost sensitivity is amplified to the same level as correctness.

2. Rebuilding the metric: replace “price per token” with “cost per task”

Most model services use similar billing structures: input, cached input, and output are priced separately, and cache hits are usually one to two orders of magnitude cheaper. Unit price is only a static parameter. The actual bill is determined by three dynamic factors: the number of rounds needed to complete a task, the cumulative context length in each round, and the number of failed retries.

A simple way to model the cost of one successful task is:

from dataclasses import dataclass
@dataclass
class RunSample:
    rounds: int               # Actual iterations, including retries
    input_tokens: int         # Cumulative input tokens
    cached_input_tokens: int  # Input tokens served from cache
    output_tokens: int        # Cumulative output tokens
    succeeded: bool
PRICE = {"input": 0.10, "cached_input": 0.004, "output": 0.50}  # USD per million tokens
def cost_per_success(samples: list[RunSample]) -> float:
    total = sum(
        (s.input_tokens * PRICE["input"]
         + s.cached_input_tokens * PRICE["cached_input"]
         + s.output_tokens * PRICE["output"]) / 1e6
        for s in samples
    )
    success = sum(1 for s in samples if s.succeeded)
    return total / max(success, 1)   # The denominator counts successful tasks only; failed-task costs are included in the numerator

The key point is that the denominator counts only successful tasks, while the numerator includes all failed-task overhead. This also explains an often-overlooked phenomenon: a model with a low unit price but a low first-round success rate may actually have a higher real cost. When evaluating models, retry count must be treated as a first-class metric, not as an engineering detail.

3. Why small models can catch up: trading reasoning-time compute for lower unit cost

A pattern widely observed over the past two years is that increasing reasoning effort at inference time can significantly improve complex reasoning and code repair performance at a lower parameter count. The so-called “highest reasoning effort level” essentially lets the model generate a longer chain of thought and perform more self-checks, trading time for accuracy.

This provides a plausible path for lightweight models to catch up with peers in the same score band: it is not that the model itself is stronger, but that a similar score of around 36 can be achieved with fewer total tokens and a lower unit price. From an engineering perspective, higher reasoning effort is not always better. The gains from high effort are often negative on simple tasks, and long reasoning traces increase output costs. Tiered usage is more cost-effective than setting everything to maximum.

ROUTES = {
    "rewrite_or_summarize": {"model": "lite", "effort": "low"},
    "structured_extraction": {"model": "lite", "effort": "low"},
    "code_repair": {"model": "lite", "effort": "high"},
    "high_risk_decision": {"model": "strong", "effort": "high"},
}
def dispatch(task):
    route = ROUTES[classify_intent(task)]   # The router can use a small model plus rules in a dual-channel design
    return route["model"], route["effort"]

Routing itself also needs cost constraints: the classifier should not be more expensive than the task it protects. In practice, the cost-optimal approach is to let regular expressions and keyword rules filter high-frequency deterministic intents first, then send the remaining samples to a lightweight classification model.

4. Long context and native multimodality: reduce the pipeline, not the per-call cost

Support for million-token contexts and native understanding of text, images, audio, and video has engineering value not in “stuffing more content in,” but in reducing preprocessing pipelines: no chunking, no external vector retrieval for recall, and no need to connect a separate model for each modality. One call can read a long document, inspect an image, and produce the final result. Removing one service from the pipeline reduces both latency and failure surface.

However, long context has two side effects that must be guarded against. First, there is the widely observed “lost in the middle” phenomenon: when critical information is buried in the middle, recall drops, so important instructions should be placed at the beginning or end. Second, cost grows linearly or even superlinearly with length, and repeated accumulation of historical messages is the primary reason agent bills get out of control. The mitigation is structured compression: keep the task goal, completed subgoals, and structured summaries of tool returns, and discard redundant raw outputs.

def compact(history, keep_last=3):
    goal, done = history.goal, history.done
    recent = history.tail(keep_last)
    dropped = len(history) - keep_last
    summary = f"Completed subgoals: {len(done)}; latest conclusion: {done[-1] if done else 'none'}"
    return [{"role": "system", "content": f"Task goal: {goal}\n{summary}"}] + recent

5. Training-side practice: distilling task-specific small models from flagship models

One useful part of the original release is the reuse of the training pipeline: first use a flagship model as the teacher, then train a task-specific small model for agent tasks using the same data and evaluation framework. In many cases, a usable version can be iterated within weeks. Common industrial approaches include:

• Using a stronger model to batch-generate “tool call → observation → re-decision” traces as supervised data.
• Applying quality filtering to traces, discarding samples that did not converge, repeated calls, or invalid parameters.
• Running data-mixing ablations on a validation set to find a balance among reasoning, code, and multimodal capabilities.
• Using executable tests rather than human evaluation as the reward signal for rejection-sampling fine-tuning.

The barrier to this path is the evaluation set and data pipeline, not compute. If a team already has a stable task-level evaluation set, a task-specific small model is often more cost-effective for its own scenario than a general-purpose large model.

6. Four engineering cost-reduction measures

1. Stabilize prompt prefixes and maximize cache usage. Cache hits depend on prefix matching. Inserting any dynamic content before the system prompt can reset the hit rate to zero. The standard practice is to keep invariant parts—tool definitions, system rules, and few-shot examples—at the very front, and put variables such as timestamps and user input at the end.

prompt = f"""{STATIC_SYSTEM_PROMPT}      # Never changes: tool schema + rules
{STATIC_FEW_SHOT}            # Never changes: examples
Task: {user_query}           # Dynamic section goes last
Current time: {now}"""

2. Use idempotent retries and budget circuit breakers. Retries must include exponential backoff and an upper limit. Each task should have a cost budget; when exceeded, the system should degrade or hand off to a human to avoid one exception consuming the daily budget.

3. Use batch processing and asynchronous execution. Non-real-time pipelines, such as offline document batch processing and nightly reconciliation, should be merged into batch requests. The cost per unit throughput is significantly lower than synchronous calls.

4. Validate results before downstream steps. Structured outputs should pass schema validation before entering the next tool call. Intercepting errors early is roughly one order of magnitude cheaper than fixing data afterward.

7. Observability: five metrics that must be monitored

Looking only at the total bill will not reveal the problem. At a minimum, collect cost per task, first-pass success rate, average retries, cache hit rate, and average context length, plus single-turn P95 latency. Persist each run’s inputs and outputs, tool-call traces, token usage, and cost details to object storage; route metrics to log services; and configure thresholds in monitoring alerts to form a “replayable and alertable” loop. On Baidu AI Cloud, this pipeline is typically built with model serving for inference, object storage for run traces, log services for metric aggregation, and monitoring alerts for anomaly notifications. The same general architecture can be self-built with equivalent services.

8. Rollout path and risk boundaries

A recommended four-stage rollout is: full regression on an offline evaluation set → shadow traffic comparison, where the same request is sent to two models but only observed and not switched → 5% canary release, bucketed by task type → full rollout. Each stage should have clear rollback conditions, such as cost per task increasing by more than 20% or success rate dropping by more than 2 percentage points.

Three boundaries need to be recognized clearly. First, there is a real trade-off between cost optimization and task success rate; the focus cannot be only on lowering price. Second, public benchmark scores have limited correlation with performance in your own business, so you must re-evaluate using your own evaluation set. Third, for scenarios involving user data, multimodality and long context mean a larger data surface; data transfer and retention should comply with organizational data-compliance requirements, and sensitive data should be desensitized before entering the inference pipeline. Model selection is only one variable in the cost equation; architecture design, cache strategy, and evaluation quality often have a larger impact.