Model Selection in the Agent Era: Throughput, Reliability, and Cost

1. From Question Answering to Agents: The Task Has Changed

Large language models were once evaluated mainly by their accuracy on single-turn question answering. Agents work differently. An agent must break down a goal, call tools, read and write files, run tests, and repeatedly revise its work based on feedback until the task is complete. Every step consumes reasoning time and tokens, and a single error can invalidate all subsequent steps.

Model selection should therefore consider at least three metrics:

• Task completion rate: The percentage of tasks completed end to end and accepted during validation;
• Cost per task: The total cost of model calls, retries, tool use, and human review;
• End-to-end latency: The time from task submission to a usable result.

Looking at only one of these metrics can easily lead to misleading conclusions.

2. Two Engineering Paths: Throughput and Reliability

Most current agent optimization strategies follow one of two broad paths.

2.1 Parallel Decomposition: Trading Compute for Time

In this approach, an orchestrator divides a large task into several independent subtasks, assigns them to multiple workers for parallel execution, and then merges the results.

Typical use cases include:

• Large-scale code migration where modules are independent;
• Dependency-upgrade checks across multiple code repositories;
• Parallel analysis of large volumes of documents or logs.

The main cost is coordination. If subtasks have hidden dependencies, conflicts may appear during the merge step. Each worker also has to load relevant context independently, so token usage increases with the degree of parallelism. Decomposition should therefore match the task’s dependency structure rather than becoming as fine-grained as possible.

2.2 Self-Inspection and Verification: Trading Iteration for Reliability

The second approach emphasizes self-verification before delivery. A typical loop is:

1. Generate a solution;
2. Write or extend tests;
3. Run the tests;
4. Fix issues based on failure output;
5. Validate again.

The agent reports completion only after all checks pass.

This approach can reduce deliveries that look correct but fail in practice. However, it also increases the cost and duration of each task. Its effectiveness depends on the quality of the validation signals. If test coverage is insufficient or the tests themselves are flawed, the model may declare success against an incorrect standard, making problems harder to detect.

2.3 Controlling Reasoning Intensity and Budgets

The same model often offers multiple reasoning levels. Lower settings can be used for simple steps, while more complex planning can use higher settings. This allows resources to be allocated between quality and cost as needed.

Budget controls should be used alongside reasoning controls. Setting a token limit for the entire agent loop prevents the system from entering endless, unproductive retries. When the limit is exceeded, the task should stop and be reported for review.

The following example shows how self-verification and budget control can be combined. llm_client and run_tests are placeholder interfaces that must be replaced with actual implementations:

from dataclasses import dataclass
@dataclass
class TokenBudget:
    limit: int
    used: int = 0
    def charge(self, tokens: int) -> None:
        self.used += tokens
        if self.used > self.limit:
            raise RuntimeError("token budget exceeded")
def solve_task(task: str, llm_client, run_tests, budget: TokenBudget,
               max_rounds: int = 5) -> dict:
    patch = llm_client.generate(task)              # Generate the initial solution
    budget.charge(patch.tokens)
    for round_no in range(1, max_rounds + 1):
        result = run_tests(patch.code)             # External validation with trusted results
        if result.passed:
            return {"status": "done", "rounds": round_no, "code": patch.code}
        patch = llm_client.revise(task, patch.code, result.log)  # Fix issues from failure logs
        budget.charge(patch.tokens)
    return {"status": "needs_review", "rounds": max_rounds, "code": patch.code}

This design has two important characteristics. First, validation is performed by external tests rather than relying only on the model’s own assessment, making the result more trustworthy. Second, both a round limit and a token limit are enforced. When either limit is reached, the task is handed to a human instead of silently exhausting resources.

3. Cost Accounting: Unit Price Is Not the Total Cost

Service providers typically publish separate prices for input and output tokens. What enterprises actually need is the effective cost per successful task, which can be estimated with the following formula:

Effective cost ≈ (cost per call × average number of calls + retry and review costs) ÷ task success rate

Example numbers are shown only to demonstrate the calculation method. Suppose Plan A has a lower unit price, requires an average of 30 calls per task, and achieves a 60% success rate. Plan B has a higher unit price, requires an average of 12 calls, and achieves a 90% success rate. Even if each call under Plan A costs half as much, its effective cost may still be higher because failed tasks must be rerun or transferred to human operators.

Latency should also be analyzed by component. Output speed affects the interactive experience at each step, parallel decomposition affects wall-clock time, and the number of retries affects tail latency. These factors influence user experience in different ways and should be recorded separately.

4. An Evaluation Framework for Enterprise Model Selection

Public leaderboards can be useful for initial screening, but their evaluation criteria, task construction, and runtime environments vary. A single score cannot be mapped directly to an enterprise workload.

A better approach is to build an internal evaluation framework:

Dimension	Key question	Validation method
Task completion rate	Can real tasks pass acceptance checks on the first attempt?	Build a regression set from historical tickets and measure the pass rate
Cost per task	What is the total cost, including retries and human review?	Record token usage and human labor for each task
Latency	Are time to first token and end-to-end time acceptable?	Track P50 and P95 latency
Security boundaries	Are tool permissions minimized?	Execute tasks in an isolated environment and audit unauthorized calls
Observability	Can failure causes be identified quickly?	Preserve complete call traces and test logs
Migration cost	How much must interfaces and prompts change when replacing the model?	Abstract model calls behind a gateway layer

The regression set is one of the most valuable assets. Historical tickets can be used to build representative tasks and acceptance criteria, allowing models to be compared under the same workload and execution conditions.