Evaluating a Production AI Coding Agent: Balancing Capability and Cost
A hands-on comparison of a general-purpose model’s production release with impressions of its preview version, along with a practical framework for evaluating coding agents. Learn how to assess task completion, error recovery, test results, token usage, latency, and human review requirements—while keeping the limits of informal observations in view.
1. What This Article Can—and Cannot—Show
This article reports hands-on observations, not the results of a controlled comparison. The author used a production release of a general-purpose model for AI coding-agent tasks and compared it with impressions of a preview version. The original account does not identify the model, version, test environment, task set, or sample size. Its observations should therefore be treated as indicative, not as standardized benchmark results.
Before relying on claims that one version is “significantly better,” establish two things: how many tasks were tested and whether each task had a reproducible pass criterion. Without both, such claims are difficult to verify.
2. Three Changes Worth Noting
More complete task execution. The preview version performed well on UI-related tasks, but was more likely to leave backend logic, project configuration, and data-processing work unfinished. In the author’s experience, the production version more often completed the full workflow for these tasks: analyzing requirements, modifying code, running checks, and fixing issues. This is a subjective observation; completion rates were not measured.
Better recovery from errors. Agent-based development rarely succeeds on the first attempt. During testing, the production version generally used runtime errors or failed tests to identify missing logic, and was less likely to repeat the same approach. That could matter for long, multi-step tasks, but more samples are needed to confirm how consistently it behaves this way.
Higher capability does not automatically mean lower cost. Tasks involving long context, multiple tool calls, and changes across many files can consume more tokens and take longer. Evaluations should track resource use alongside task outcomes.
3. The Nine Stages of an Agent Task
A coding agent typically moves through nine stages:
- Read the project structure.
- Understand the user’s requirements.
- Locate the relevant files.
- Plan the changes.
- Run code or tests.
- Analyze errors.
- Fix side effects.
- Verify the result again.
- Explain the changes.
A mistake at any stage can undermine the final result. Improving code generation alone does not guarantee a useful agent; context management, tool-use strategy, and error recovery matter too. During testing, record performance at each stage to help identify where problems occur.
4. A Suggested Scoring Rubric
Use the following rubric to record each test. Score every dimension from 0 to 2: 0 means unacceptable, 1 means usable with limitations, and 2 means ready for delivery. Calculate the weighted score for each dimension as (score ÷ 2) × weight, then add the six results to get a score out of 100.
| Dimension | Evaluation criteria | Fields to record | Weight |
|---|---|---|---|
| Requirement understanding | Did the agent identify implicit constraints such as API compatibility, permission checks, and error handling? | Number of missed constraints | 20% |
| Task decomposition | Did it work in stages and verify the result at the end of each stage? | Number of stages; verification result for each stage | 15% |
| Tool use | Were tool calls relevant to the goal, and did the agent change its approach after a failure? | Total calls; irrelevant calls | 15% |
| Error recovery | Could it fix the issue within two rounds after receiving an error? | Number of recovery rounds | 20% |
| Test results | Did all automated tests pass? | Pass rate | 20% |
| Scope of changes | Did it modify unrelated files or make unnecessary formatting changes? | Number of unrelated files changed | 10% |
Teams should set their own passing threshold. If there is no established baseline, 70 can serve as a starting point, subject to adjustment for the tasks being evaluated.
5. What to Record for Each Test
In addition to the rubric, keep a complete record for every task:
- Task ID and task type
- Technology stack and project size
- Whether the first output passed tests
- Number of files modified
- Number of tool calls and recovery rounds after failures
- Number of follow-up prompts from the user
- Completion time and token usage
- Whether unrelated changes were introduced
If you change the scoring criteria, record when and why the change was made and which historical scores it affects. This helps distinguish changes in model capability from changes in the evaluation scale. Comparing old and new scores without this context can lead to misleading conclusions.
6. Choose a Model Tier by Task Type
Whether the production version is worth the additional investment depends on the task. The following tiering approach is a starting point; teams should adapt it to their budgets and risk tolerance.
| Task type | Typical examples | Suggested model tier | Human review |
|---|---|---|---|
| Simple completion and formatting | Rename a variable; complete a single function | Lightweight model | Basic self-testing |
| Medium-complexity work | Add a feature to one module; adjust API parameters | General-purpose model; analyze before execution | Code review |
| High-risk work | Refactor across files; change permission logic; migrate data | Higher-capability model for planning and review | Testing and human review |
| Production-related changes | Update deployment configuration; process live data | Higher-capability model as an assistant; a human makes the decisions | Testing, review, and a rollback plan |
Calculate cost across the whole task, not from model price alone. Compare token usage, waiting time, and human rework time for similar tasks. If a lightweight model requires repeated fixes, its total cost may exceed that of a general-purpose model.
7. How Output Limits Affect Testing
The maximum output length varies across interfaces. Check the documentation for the interface or client you use; no specific limits are provided here.
Output limits can make it difficult to produce long reasoning traces, large files, or cross-module changes in a single response. A staged workflow can help:
- Ask the model for a change plan and have a human confirm the scope.
- Generate patches in batches, covering one or a small number of files at a time.
- Apply each patch with a patch tool and run tests.
- If tests fail, provide the model with the error and only the relevant code snippets.
This approach keeps individual outputs shorter and makes each change easier to review and roll back independently.
8. Limits of These Observations
The observations in this article have several limitations:
- The sample size is unknown, so results from individual tasks may skew the overall impression.
- Project documentation quality has a substantial effect. Missing documentation, inconsistent naming, or insufficient tests can lead to worse results.
- For less common frameworks, specialized protocols, or project-specific conventions, the model still depends on code search and runtime feedback.
- The observations cannot be generalized directly to other models, versions, or projects.
These limitations make the article more useful as a guide to test design than as a final basis for model selection.
9. How to Start Your Own Evaluation
To determine whether a model suits your project:
- Select 10 to 20 real tasks covering frontend, backend, configuration, data processing, and test authoring.
- For each task, specify the input, expected output, edge cases, and compatibility requirements.
- Use the same rubric to score the preview and production versions.
- Record completion time and token usage, then calculate average resource use by task type.
- Draw conclusions only after collecting enough samples, and document the sample size and test environment in your report.
An evaluation based on this process can provide a more reliable basis for practical model-selection decisions.
