LLM Agent Evaluation: Breaking Down Tasks, Cost, and Stability
A strong evaluation of LLM agents goes far beyond a single benchmark score. This article presents a three-dimensional framework for assessing whether model-driven coding agents can reliably complete engineering tasks. It explains how to control variables in the evaluation criteria (output-length limits, the agent harness as a co-author, and drifting rubrics), how to measure generalization across task categories, how to quantify stability through repeated runs and failure-mode analysis, and how to calculate the true total cost of producing a usable result rather than comparing per-token prices. It closes with a reusable evaluation plan and a practical rollout strategy built on task routing and validation gates, giving developers a repeatable way to turn model comparisons into reliable selection decisions.
1. Introduction: Faster Iterations Demand Better Evaluation Methods
Model services commonly follow a “preview first, then general availability (GA)” cadence: preview releases gather real-world feedback and are tuned quickly, while GA releases set the version for external use and long-term integration. Developers form two kinds of judgment from this:
- whether the jump from preview to GA is meaningful;
- whether that improvement translates into real efficiency and returns in production engineering.
Agent scenarios need this judgment especially, because the output is not just a piece of text but a full set of runnable engineering artifacts, spanning requirement understanding, code generation, tool calling, and multiple rounds of revision. A small improvement at the text level can be amplified into a make-or-break difference after multiple rounds of execution.
Comparing a single experience or a version score is far from enough. To make evaluation conclusions stable, reproducible, and ultimately useful for selection, you should observe at least five dimensions at once: evaluation criteria, task generalization, task stability, unit cost, and rollout strategy. This article uses “can a model-driven coding agent reliably complete engineering tasks?” as the entry point and breaks each one down.
2. Evaluation Criteria: Control Variables First, Then Compare
2.1 Output Length Limit Determines Whether a Task Is Solvable
The maximum output length of a single generation directly affects what a model “can do.” Take a typical problem statement: the prompt may run to tens of thousands of words, while a proper solution often needs thousands or even tens of thousands of tokens of text and formulas.
When output length is constrained, the model has only two options: compress its reasoning, or get cut off mid-way. The former produces a “conclusion without a completed derivation,” and the latter leaves an incomplete answer. Such failures are often lumped together as “insufficient reasoning ability,” when they are actually the combined result of interface constraints and task characteristics.
A recommended engineering practice is to place the prompt and the answer in different role messages:
- the prompt is provided as a preceding message and does not consume the single-generation output quota;
- only the answer needs to be output.
This practice is also common in the Tool-Call scenario of general APIs: content returned by retrieval or tools is sent back as a separate message, and the model’s output contains only the next decision.
# Illustration: the prompt does not consume the output quotamessages = [{"role": "system", "content": "You are a rigorous engineer..."},{"role": "user", "content": "<problem/requirement, possibly very long>"},{"role": "assistant", "content": ""},{"role": "assistant", "content": "<solution for this problem, generated by the model>"}]# If the API requires the last message to be user, append a short lead-in after the prompt
Even with sufficient output length, an over-long problem may exceed the context window. A common approach is to first retrieve relevant passages and then assemble the problem into the prompt. This yields three conclusions:
- Evaluation reports should state the task-solvability premise: whether the task can be completed in full under the interface constraints;
- For output-constrained models, a “step-by-step answering + result aggregation” pipeline can compensate, but it adds rounds and cost;
- Before comparing, verify how task length, context window, and single-output limit match up, to avoid distorted conclusions.
2.2 The Agent Framework Is a Co-Author of the Results
In agent tasks, the model is only one link in the execution chain. The system prompt, tool definitions, file-editing approach, context-compression strategy, and failure-retry mechanism all significantly affect the final artifact. The same model plugged into different coding agents may produce artifacts of quality in different volatility bands. In engineering terms, model capability can be approximated as:
Model capability ≈ output quality = f(model, coding agent, prompt strategy, sampling parameters)
Therefore, cross-model comparison must satisfy three preconditions:
- Fix the toolchain and its version, and record it in the report;
- Unify sampling parameters, such as temperature and maximum generation length;
- Use the same task set and the same scoring standard wherever possible.
A conclusion that applies to only one tool should not be extrapolated to all agent scenarios.
2.3 Evaluation Standards Drift and Need Retrospective Calibration
As model capability improves rapidly, the baseline for “what counts as a good artifact” moves up in parallel. The same artifact that might have been judged acceptable months ago today deserves only a middling rating.
Scoring with a fixed, unchanged old standard produces systematically optimistic results; but directly reusing historical scores accumulated across different periods for a longitudinal comparison creates inconsistency.
A more reasonable approach is:
- Record the version whenever you revise the scoring rubric for an evaluation;
- Re-score old samples of key models with the new rubric retrospectively;
- Always evaluate new models against the current rubric.
Only then does the horizontal comparison have a common baseline, and the vertical trend read accurately.
3. Task Generalization: From “Single-Point Excellence” to “Broad Usability”
Generalization is the ability to transfer the capability of solving one class of problems to new tasks with different structures and constraints.
Some models stand out early on front-end UI tasks but drop sharply once they move into data processing, API integration, and script automation. The cause is usually not insufficient coding ability itself, but shortcomings in tool-use strategy, domain knowledge, and requirement understanding.
The evaluation task set is recommended to cover the following categories:
- Project scaffolding: creating project skeletons and basic configuration from scratch;
- UI development: page structure and interaction implementation;
- Data processing: cleaning, transformation, and validation scripts;
- API development and service integration;
- Legacy code understanding and refactoring;
- Automated test writing and regression;
- Debugging: locating root causes from errors and logs.
Judging generalization should not rest only on “does it ultimately run,” but also on the quality of the process, observing three key checkpoints:
- Requirement understanding: can it identify implicit constraints such as compatibility requirements, input boundaries, and error-handling approaches;
- Information gathering: can it autonomously retrieve docs and inspect relevant code, rather than implementing on guesses;
- Self-correction: after hitting a problem, can it locate the cause from feedback and adjust effectively.
For harder tasks in relatively unfamiliar domains, relying on retrieval is perfectly reasonable, and retrieval-augmented approaches are widely used in engineering practice. The problem is when insufficient retrieval misses key details, or when the implementation is inconsistent with existing conventions. Real progress is when the model can autonomously fill in the logic after a deviation, rather than leaving the problem to the user.
It is also advisable to include non-standard tasks: colloquially phrased, incompletely specified, spanning multiple codebases. These tasks most easily expose the problem of “sloppily delivering without accurately understanding the goal.”
4. Task Stability: Both the Upper and Lower Bounds Matter
4.1 Where Volatility Comes From
The same model on the same task can produce noticeably different results across repeated runs. The sources are mainly three:
- the random sampling of generative models;
- state accumulation and branching from multi-round tool calls;
- changes in context length and composition, such as growing history and information loss after compression.
These differences ultimately manifest as three failure modes:
- Instruction-following breakdown: explicit requirements about output format, technical constraints, and implementation scope are ignored;
- Missing global consistency: local implementations are correct, but overall style, naming, or interface design is inconsistent;
- Hidden defects: issues the model’s own checks struggle to catch, such as poor handling of boundary conditions or missing error handling.
4.2 Distribution Characteristics to Watch
A single demo is easily affected by survivorship bias; it is recommended to run the same task multiple times and watch four things:
- Pass rate, the proportion of runs that meet the usable standard;
- Variance and lowest percentile, to gauge the lower bound;
- Failure-mode distribution, e.g., how much each of instruction-following, global consistency, requirement understanding, and code correctness accounts for;
- Resource-consumption distribution, including token usage and interaction rounds.
“High upper bound, low lower bound” means such a model may behave like this: the best run produces extremely high direct output quality — for example, refined front-end UI and animations — with modest token consumption; the worst run cannot even follow basic instructions, and the model’s self-check does not detect it.
From an engineering standpoint, the lower bound deserves more attention than the upper, because stability determines retry cost, and retry cost translates directly into task cost.
4.3 Build Stability Into the Evaluation Process
Three measures are recommended in the evaluation process:
- Multiple sampling: run each task 3 to 5 times. This is a common way to bring randomness into observation;
- Retain failure samples: for failure-mode classification and regression comparison between versions;
- Define a minimum acceptable level: beyond the average score, also assess the lowest-percentile performance.
On the model-call side, fixing sampling parameters such as temperature reduces some volatility but cannot eliminate the state branching from multi-round interaction, so external validation remains indispensable.
5. Task Cost: From “Looking at Unit Price” to “Calculating the Total Cost of Completing a Task”
5.1 Structural Differences in Cost
Model inference is usually priced separately for input and output tokens. But when comparing cost-effectiveness, looking only at the per-million-token price easily leads to wrong conclusions: the model with the lowest unit price does not necessarily have the lowest total cost for completing the same task.
Token consumption in agent tasks concentrates on three kinds of repeated reads:
- the system prompt and tool definitions are resent in every round of requests;
- the conversation history accumulates as rounds increase;
- every file view or retrieval re-injects large chunks of context.
As a result, the input tokens for a single task are often several times the output tokens. A model with many rounds and repeated trial-and-error may cost more in total even at a lower unit price.
5.2 Joint Measurement of Cost and Effectiveness
Cost per task = Σ(input tokens × input price + output tokens × output price)Average cost per "usable artifact" = cost per task / proportion of runs meeting the usable standardCost-effectiveness = effectiveness score of runs meeting the usable standard / average cost per usable artifact
For example (all data is hypothetical): Model A has a single-run success rate of 90% at a cost of $1; Model B has a single-run success rate of 50% at a cost of $0.4. Running repeatedly and stopping at first success:
- Model A average cost: 1 / 0.9 ≈ $1.11;
- Model B average cost: 0.4 / 0.5 = $0.8.
The cheaper model turns out more cost-effective; but once you factor in the rework time from failures, the conclusion may reverse. So the evaluation should present a cost-effectiveness distribution, not just a single ranking table.
5.3 Means to Reduce Unit Cost
- Compress context: trim irrelevant history and summarize tool-returned results;
- Streamline prompts and tool definitions: remove redundant descriptions to cut the fixed per-round overhead;
- Optimize task decomposition: avoid repeated exploration caused by vague goals;
- Task routing: hand simple tasks to lower-cost models and complex tasks to higher-capability models;
- Optimize the evaluation process itself: prefer services that support Prompt Caching billing, and fix the system prompt in the evaluation pipeline so the repeated part hits the cache, lowering the cost of batch evaluation. Cache hits are usually metered by cached-read tokens; refer to the provider’s billing documentation for specifics.
6. A Reusable Evaluation Plan: From Design to Reporting
6.1 Design Phase: Tasks and Rules
- Verify task solvability: check whether the problem length, context window, and output-length limit match;
- Build a layered task set that includes standard tasks and edge tasks;
- Define an actionable scoring rubric, e.g. (using “artifact can be merged directly without modification” as the baseline):
| Grade | Criterion | Meaning |
|---|---|---|
| A | Usable as-is | Meets requirements and constraints, passes tests |
| B | Usable with minor changes | Local adjustments suffice |
| C | Partially usable | Key logic needs rework |
| D | Basically unusable | Needs a rewrite |
6.2 Execution Phase: Repeated Runs and Cost Accounting
Fix the coding agent version and sampling parameters; run each task multiple times; record token usage, rounds, and elapsed time. You can refer to the following structure:
# Pseudocode: reproducible evaluation of a single taskdef evaluate_task(task, harness, config, runs):records = []for _ in range(runs):result = harness.run(task=task,max_output_tokens=config.output_limit,temperature=config.temperature,)records.append({"task_id": task.id,"category": task.category,"score": judge(result.artifacts, task.rubric), # A/B/C/D"tokens_in": result.usage.input_tokens,"tokens_out": result.usage.output_tokens,"rounds": result.rounds,"cost": cost(result.usage, price_table),"failures": tag_failures(result), # e.g. "instruction-following", "global consistency"})return records
6.3 Analysis Phase: Multi-Dimensional Summary
- Give the average grade and pass rate by task category;
- Compute the average cost and lowest-percentile performance of successful runs;
- Summarize the failure-mode distribution and compare it against historical versions.
It is advisable not to give a single overall score. You can normalize quality, cost, and stability to 0–1, then compute a weighted composite score, with weights set by business scenario:
Composite score = w_q × quality + w_c × cost + w_s × stability
| Business scenario | Focus | Weight tendency (illustrative) |
|---|---|---|
| Internal efficiency tools | Cost | Raise the cost weight |
| Customer-facing delivery | Stability | Raise the stability weight |
| Core system development | Quality | Raise the quality weight |
In the example, one model can match leading models “at its best,” but shows clear instruction-following problems on some tasks, while its per-task cost remains relatively high — leaving it short of the “best cost-effectiveness” overall. This shows that an advantage in a single dimension is not enough to decide selection; the trade-offs between dimensions are the core of the decision.
7. Rollout Strategy: Task Routing and Validation Gates
Evaluation results ultimately have to land in production. A combined “routing + validation” strategy is recommended.
7.1 Route by Task Characteristics
- Batch, low-risk, easily verified tasks — such as format conversion, boilerplate code generation, and supplementary unit tests — should preferentially use lower-cost models;
- High-risk tasks with strict delivery standards — such as core-module design and refactoring, and online incident troubleshooting — should use higher-capability models, combined with stricter human review;
- “High-ceiling” tasks such as UI work can be run with multiple sampling under human oversight, selecting the best result.
7.2 Set Validation Gates at Key Nodes
A typical failure of LLMs in agent scenarios is “generating without verifying” — declaring completion without running, testing, or checking. You can set several gates in the workflow:
- Code changes must pass compilation or type checking;
- Test cases must run and reach an agreed coverage level;
- Static analysis and dependency audits must run before submission;
- The change scope must match the task goal, preventing out-of-bounds modifications.
Validation gates are the most effective means against “hidden defects” and “too-low lower bounds”: problems the model cannot catch itself are backstopped by deterministic tools and processes.
7.3 Use Prompting Strategies to Cover Weaknesses
For instruction-following and global-consistency problems, an approach of “set the spec first, then implement” can be used:
1. First output requirement understanding, the constraint checklist, and the implementation plan, and wait for confirmation;2. List project-level conventions, such as naming style, directory structure, and error-handling approach, to constrain all subsequent changes;3. Self-check after each round: confirm item by item against the constraint checklist, and state the files affected and the impact scope of the changes.
In addition, a stable and reliable platform foundation is equally key:
- Containerize the evaluation pipeline to ensure a reproducible environment;
- Archive artifacts and run logs with object storage;
- Use message queues to smooth out peaks and troughs;
- Use monitoring and alerting to continuously track call volume and token consumption.
For example, cloud platforms such as Baidu AI Cloud offer general-purpose cloud product capabilities — containers, object storage, message queues, logging, and monitoring — that can serve as the foundation for hosting evaluation pipelines and agent applications.
7.4 Four Questions for Reading Third-Party Evaluations
When reading a model release evaluation, you can first ask four questions:
- Is the task solvable under the model’s output-length and context constraints?
- Are the agent tool and version unified and disclosed?
- Are the pass rate and failure modes under multiple runs reported?
- Is cost calculated by unit price, or by the total cost of completing a task?
Only when these four questions are answered clearly can an evaluation conclusion become a reliable basis for selection.
8. Conclusion
The “preview to GA” iteration cadence means model capability is still changing continuously, and any conclusion has a shelf life.
For developers, what is more valuable is not asking “which model scores highest,” but building your own evaluation system:
- Clarify interface constraints and evaluation criteria;
- Cover a sufficiently diverse set of tasks;
- Sample multiple times to observe stability;
- Account for the true cost of completing a task.
The triangle of capability, cost, and stability ultimately serves business goals. The supporting validation workflow and routing strategy are the key to taking an agent from “usable in a demo” to “reliable in production.”
