Multi-Model Support Becomes Baseline: A Cost-Measurement Methodology for Agent Harnesses
This article explains why multi-model support is becoming a baseline capability for coding and office agents rather than a product differentiator. It presents a reusable cost accounting model, a six-step benchmarking method, and practical optimization levers such as layered routing, context governance, cache-friendly design, and convergence control. The core argument is that model price alone is misleading: real cost should be measured as cost per successful task under aligned task difficulty, quality thresholds, and context strategies.
1. Multi-Model Access Is Moving from Differentiator to Baseline
Recently, a coding and office Agent product announced support for external models. The available options include recent versions of several major open-source and commercial large models, with fairly broad coverage. The significance of this move is not that one product added another configuration switch. It signals a phase-level shift: multi-model access is moving from a differentiated selling point to a baseline capability.
1.1 Demand Is Inherently Tiered
For some time, tutorials about connecting coding tools to alternative models have circulated widely. Their continued popularity points to one basic fact: a single model cannot cover all scenarios. The reasons can be grouped into four categories:
- Task difficulty is tiered. For tasks such as file renaming, comment completion, or unit-test scaffolding, lightweight models are often sufficient. Cross-file refactoring, difficult bug localization, and long dependency-chain analysis are where top-tier models provide real value.
- Cost structures differ. When usage shifts from occasional individual use to thousands of automated tasks per day across a team, differences in input and output pricing are amplified linearly by call frequency. Even a small unit-price difference can become a meaningful cost gap.
- Existing procurement and compliance constraints matter. Enterprises often have already completed compliance reviews, data-transfer assessments, and procurement processes for specific model providers. They need Agents to reuse existing approved channels rather than introduce a new vendor.
- Preferences and evaluation results vary. The relative ranking of the same model can change across task sets. Teams often select models based on their own regression suites rather than generic public leaderboards.
1.2 Open Access Does Not Mean Low Confidence in a First-Party Model
A recurring industry argument is that product teams should win users through model capability, execution efficiency, and product experience, rather than forcing users into a closed stack through lock-in. This can be tested in engineering practice: if lock-in were truly effective, independent measurements of external-model performance would not become a topic of public discussion. In fact, it is precisely because switching is allowed that the quality of coordination between the Harness and different models can be tested under real conditions.
Another important point is two-way openness. On the model side, vendors provide standardized APIs and allow third-party applications to connect. On the Harness side, products support third-party models. Only when both sides are open does a healthy ecosystem emerge. Users can pair “this Harness with that model” or “that Harness with this model,” and the real competition becomes the efficiency of the combination rather than the height of ecosystem walls.
1.3 Engineering Implication: A Model-Agnostic Abstraction Layer Is Required
Multi-model support is not just a configuration option. It is a set of real architectural constraints. A maintainable implementation should at least unify the following layers:
- Message and role protocol: a unified representation for system, user, assistant, and tool roles, as well as the attachment of multimodal segments.
- Tool-call protocol: JSON Schema definitions for function declarations, support for parallel calls, and formats for returning tool results.
- Streaming and interruption: incremental event models, cancellation semantics, and retention of partial results after timeouts.
- Capability detection and fallback: models differ in long-context support, structured output, caching semantics, and maximum output length. The runtime should detect capabilities and choose a conservative path when a model does not support a feature.
- Metering and attribution: every call should be traceable to a specific task, tenant, stage, and model. Without this, later cost optimization has no reliable foundation.
2. What Is an Agent Harness, and Why Does It Directly Determine Cost?
An Agent Harness refers to the full engineering layer surrounding the model: system prompts, tool definitions, context management and compression, task planning and decomposition, execution sandboxes, result return, retry and convergence control, usage metering, and logging. The model is responsible for “thinking,” while the Harness determines what the model should think about, what it should see, how many rounds it should take, and when it should stop.
Putting the same model into two different Harnesses can produce measured cost differences of several times. This is not unusual. The main sources of overhead can be summarized into six categories:
| Cost item | Description | Typical amplification |
|---|---|---|
| Always-on prefix | System prompt plus all tool schemas must be resent with every request round | 1.5–4x input volume |
| Context strategy | Whether context is incremental, summarized/compressed, or retains historical tool outputs | 2–10x |
| Tool return payloads | Full logs, full file contents, or untruncated test output | Up to tens of thousands of tokens per turn |
| Reasoning rounds | Ineffective loops or repeated self-checking caused by poor stop conditions | 1.3–3x round count |
| Caching and concurrency | Prefix cache hit rate, request batching, and concurrent scheduling | 0.5–0.9x when caching works |
| Retry and fallback | Retry counts, model fallback chains, and timeout reruns | Proportional to failure rate |
The conclusion is straightforward: comparing model unit prices alone almost always leads to the wrong conclusion. What really determines the bill is the token consumption required per unit of effective output, and this is jointly determined by the model and the Harness.
3. How to Calculate Cost: Build a Reusable Accounting Model
3.1 Component Formula
A basic task cost model can be expressed as:
Cost_task = sum over i from 1 to N of (T_in_i × p_in + T_cached_i × p_cached + T_out_i × p_out) + Cost_side
Where:
- N is the number of reasoning turns for the task.
- Cost_side includes peripheral overheads such as vector retrieval and embedding calls, sandbox compute resources such as container CPU/memory duration, object storage and network traffic, and logging/monitoring storage.
Three pricing details often cause mistakes:
- Cache reads and cache writes are often priced separately. Some providers charge an additional fee for the first cache write.
- Long-context pricing may be tiered. Once token counts exceed a certain threshold, the unit price may jump.
- Output tokens are usually several times more expensive than input tokens. Reducing output is often more cost-effective than reducing input.
3.2 Example Accounting Implementation
from dataclasses import dataclass@dataclass(frozen=True)class Price:input_per_1k: float # regular input priceoutput_per_1k: float # output pricecached_input_per_1k: float | None = None # cache hit read pricecache_write_per_1k: float | None = None # cache write pricedef turn_cost(price: Price, usage: dict) -> float:"""Cost of a single API call (in the same unit as the unit prices)."""prompt = usage.get("prompt_tokens", 0)cached = min(usage.get("cached_tokens", 0), prompt)cache_write = usage.get("cache_write_tokens", 0)uncached = max(prompt - cached - cache_write, 0)p_cached = price.cached_input_per_1k if price.cached_input_per_1k is not None else price.input_per_1kp_write = price.cache_write_per_1k or 0.0return (uncached * price.input_per_1k+ cached * p_cached+ cache_write * p_write+ usage.get("completion_tokens", 0) * price.output_per_1k) / 1000.0def task_cost(price: Price, turns: list[dict]) -> dict:"""Aggregate by task and return structured metrics for attribution."""per_turn = [turn_cost(price, u) for u in turns]total = sum(per_turn)prompt_tokens = sum(u.get("prompt_tokens", 0) for u in turns)cached_tokens = sum(u.get("cached_tokens", 0) for u in turns)return {"cost": total,"turns": len(turns),"prompt_tokens": prompt_tokens,"output_tokens": sum(u.get("completion_tokens", 0) for u in turns),"cache_hit_rate": round(cached_tokens / prompt_tokens, 3) if prompt_tokens else 0.0,"cost_per_turn": round(total / len(turns), 6) if turns else 0.0,}
3.3 The Primary Metric Must Be Cost per Successful Task
If total spending is simply divided by the number of attempted tasks, failed tasks and retries contaminate the denominator. A recommended three-layer metric system is:
- Primary metric: Cost per Successful Task = total cost / number of tasks that pass acceptance criteria
- Supporting metric: cost per effective output, such as cost per 100 lines of effective change or cost per passing test case
- Health metric: first-pass rate, average repair rounds, and human intervention rate
Comparison is meaningful only when quality does not decline. If a cheaper option reduces success rate from 92% to 70%, the savings in API cost will be paid back many times over through human rework.
4. Benchmarking Method: Six Steps to Verify or Reject a “Cheaper” Claim
Step 1: Lock Down a Tiered Task Set
Do not use a single “average difficulty” task set. Split tasks into at least four tiers, with 20–50 samples per tier:
- L1 lightweight: single-file changes, format conversion, comment and documentation generation
- L2 medium: localized refactoring across 2–3 files, unit test generation
- L3 complex: cross-module changes, dependency upgrades, problems requiring multiple rounds of code reading to locate
- L4 long-chain: debugging tasks that require repeated execute-observe-fix cycles
Fix the code repository snapshot, dependency versions, and random seed to avoid environmental drift across runs.
Step 2: Align the Execution Environment
Use the same toolset, the same sandbox specifications, the same timeout and concurrency settings, the same maximum retry count, and the same maximum reasoning rounds. If these are not aligned, all later numbers are not comparable.
Step 3: Align Context and Prompt Strategies
- Keep system prompt length within a comparable range, for example within 20% of each other.
- Keep tool descriptions at a similar level of detail.
- Use the same history-trimming strategy, such as retaining the most recent K turns or applying the same summarization method.
- Record the baseline “always-on token” cost: how many input tokens are consumed by an empty conversation that does not invoke any tools. This is a fixed tax for every task.
Step 4: Repeat Runs and Use Statistical Reporting
Run each task at least three times. Report median and P90 values, not the best single run. Agent task costs have high variance, and some tasks may enter ineffective loops. The median represents typical daily experience, while P90 represents the upper bound for budgeting.
Step 5: Itemize Accounting to Create a Traceable Bill
Record and store input, cached, and output tokens per turn, along with latency and stage labels such as planning, tool calling, or summarization. Send these records to a logging service and store complete execution traces in object storage for offline replay. Cost optimization becomes possible only when every number can be traced back to a specific call.
Step 6: Common Anti-Patterns
| Anti-pattern | Why it is wrong |
|---|---|
| Comparing only input unit price | Ignores output pricing, caching differences, and context expansion |
| Using different task sets for two experiments | Results become impossible to attribute |
| Including failed cheap tasks in the denominator | Overestimates cost efficiency |
| Running each task only once | Conclusions are distorted by variance |
| Ignoring peripheral costs | Sandbox compute and retrieval calls may exceed model costs |
| Not separating model capability from Harness contribution | It becomes unclear whether optimization should focus on the model or the Harness |
5. Lower Cost Often Comes from the Harness Rather Than the Model
In practice, when one combination appears significantly cheaper, a large part of the difference may come from a leaner Harness rather than a cheaper model. Three typical differences stand out:
- Minimal Harness: short system prompt, fewer mounted tools, and no multi-round self-checking or regression verification. Token consumption per task is significantly lower, but the cost of verification is shifted to the user.
- Full Harness: multi-agent collaboration, parallel subtasks, and self-checking plus regression testing at each step. Quality and stability are better, but token consumption is clearly higher.
- Cache-friendly vs. cache-unfriendly: placing invariant content in a stable prefix can improve cache hits, while inserting timestamps or random IDs into every request invalidates the prefix. Input costs can differ by more than 2x.
Therefore, cross-comparison requires three alignments: task complexity alignment, quality threshold alignment, and context strategy alignment. Without any of them, the comparison conclusion is unusable.
6. Model-Agnostic Cost Reduction Techniques
6.1 Layered Routing
Select models dynamically based on task difficulty and historical success rate. Add a budget circuit breaker for each task.
PLANS = {"L1": {"model": "light", "max_turns": 4, "budget_tokens": 40_000},"L2": {"model": "mid", "max_turns": 8, "budget_tokens": 120_000},"L3": {"model": "strong", "max_turns": 16, "budget_tokens": 300_000},"L4": {"model": "strong", "max_turns": 24, "budget_tokens": 600_000},}def route(task, ledger: dict):plan = PLANS[task.level]if ledger.get(task.id, {}).get("failed_rounds", 0) >= 1:# Upgrade immediately after one failure to avoid idle loops on lightweight models.plan = {**plan, "model": "strong", "max_turns": plan["max_turns"] * 2}return plan
6.2 Context Governance
- Truncate first, then summarize tool output. Logs should return only error segments and key context. Files should return only relevant fragments rather than full content.
- Use retrieval-based injection instead of full injection. Turn “the entire repository” into “Top-K relevant fragments.”
- Compress historical messages. Distill conclusions from completed stages into structured summaries and discard process noise.
- Simplify structured output. When JSON is required, disable verbose explanation fields.
6.3 Cache-Friendly Design
Place invariant content—system prompts, tool schemas, and domain descriptions—at the beginning of the request and keep it byte-stable. Avoid inserting timestamps, session IDs, or random numbers into the prefix. This directly determines cache hit rate and is one of the simplest, most certain optimizations.
6.4 Convergence Control and Observability
- Set maximum rounds per task, maximum token budgets, and per-tool timeouts.
- When the budget is exhausted, degrade gracefully by returning current progress and suggesting human takeover instead of continuing to burn tokens.
- Build cost dashboards by model, tenant, and task type. Set alerts for abnormal period-over-period changes.
- Use fixed traces for offline shadow replay. Replaying the same execution traces across different models allows effect comparison at near-zero marginal task cost.
On general cloud platforms—Baidu AI Cloud is one example—these capabilities are commonly delivered through container platforms for isolated sandboxes and elastic concurrency, object storage for execution traces and evaluation sets, message queues for long-running task decoupling, and logging/monitoring services for token and cost attribution. Similar combinations are common across major cloud platforms. The selection criterion should be observability and data retention capability, not a single product name.
7. Co-Designing the Harness and the Model
The idea that “the same model performs better inside its own Harness” is not mysterious. It usually comes from four sources:
- Training distribution alignment. The model may have been adapted to the prompt style and tool protocol of a specific Harness, which naturally improves output-format validity.
- Protocol consistency. Tool-call behavior, parallel-call behavior, and structured-output boundaries are predictable, reducing parsing failures and ineffective retries.
- Failure-sample feedback loops. Production failure traces can be collected and converted into regression sets, driving prompt and tool-design iteration and creating a positive loop.
- Complete feature support. Long context, caching semantics, and reasoning-budget control can be fully used instead of being reduced to the lowest common denominator.
For multi-model access, the practical constraint of co-design is capability matrix and automatic fallback. Maintain a “model × capability” matrix. When a model does not support parallel tool calls or cache writes, the adaptation layer should automatically switch to a conservative path and record the degradation event. Otherwise, multi-model support can degrade into multiple hidden bugs.
8. Deployment Checklist
- Is the task set tiered, with enough samples per tier to support statistical conclusions?
- Are execution environment, context strategy, and quality thresholds aligned across three dimensions?
- Is every call itemized in accounting and traceable to a specific task and stage?
- Are median and P90 values reported instead of a single best run?
- Is the primary metric total cost per successful task?
- Are peripheral costs such as sandbox, retrieval, and storage included in the total bill?
- Has prefix-cache hit rate been optimized and continuously monitored?
- Are budget circuit breakers and failure-upgrade mechanisms in place?
- Is a “model × capability” matrix and fallback path maintained?
- Is offline replay available to avoid running full online experiments for every model selection?
Conclusion
Once multi-model access becomes standard, competition among Agent products shifts from “which models are supported” to “what efficiency is achieved on which models.” Efficiency is an engineering metric that can be measured rigorously. It is determined jointly by token structure, cache hits, context strategy, reasoning rounds, and quality thresholds.
For any team, the prerequisite work more important than “which model to choose” is establishing a trustworthy cost accounting and benchmarking method. When you can clearly explain, using a unified methodology, “where every dollar goes at each stage and how much success rate it buys back,” model selection stops being a matter of opinion and becomes a reproducible engineering decision.
