LLM Coding Agents: Cost-Effectiveness Evaluation and Model Selection
As model capabilities converge, choosing a coding agent based on per-call price alone can be misleading. This article explains how to evaluate cost-effectiveness at the task level, covering cached and non-cached input pricing, output token control, internal evaluation datasets, and side-by-side comparison of capability and cost for practical model selection.
I. Background: Unit Price Is No Longer the Only Answer
As large model capabilities gradually converge, models in the same tier now perform fairly similarly on common coding tasks. At this point, price, output length, response stability, and agent capabilities become the main differentiators.
Coding agents do not work like ordinary question answering. They repeatedly read code, call tools, modify files, run tests, and replan based on errors. A single task may trigger dozens of model calls, and the context continues to grow. Therefore, the price per call is only one part of the cost. What teams really need to focus on is the total cost of completing a full task.
From an engineering perspective, this article discusses how to systematically evaluate the cost-effectiveness of these models.
II. Understanding the Pricing Structure
Pricing for mainstream large models usually consists of three parts:
- Non-cached input tokens: including prompts, conversation history, code context, and tool return results.
- Cached input tokens: when the same prefix is reused by the server, it is billed at a lower unit price.
- Output tokens: content generated by the model, such as answers and code. Which content counts as output depends on the provider’s rules.
Output unit prices are usually higher than input unit prices. For coding agents, input volume is often much larger than output volume, so the cache hit ratio has a major impact on total cost. For example, if a stable system prompt and project structure description can be reused continuously, actual spending will drop noticeably.
Pricing methods, cache validity periods, and cache hit conditions vary considerably across providers. Before selecting a model, rely on each provider’s official documentation rather than second-hand information.
2.1 Cost per Task Formula
Evaluation should be based on a “single task,” not a “single call.” The formula can be expressed as follows:
Cost per task = [(input tokens − cached tokens) × input unit price + cached tokens × cached unit price + output tokens × output unit price] ÷ 1,000,000
The unit prices in the formula are all per million tokens.
2.2 Example Code
The following code demonstrates how to compare the cost of two pricing plans using usage data from the same set of tasks. The prices in the code are assumptions for ease of calculation and are not pricing for any real product.
from dataclasses import dataclass@dataclassclass Pricing:input_per_m: float # Unit price per million non-cached input tokens (CNY)cached_per_m: float # Unit price per million cached input tokens (CNY)output_per_m: float # Unit price per million output tokens (CNY)@dataclassclass TaskUsage:input_tokens: intcached_tokens: intoutput_tokens: intdef task_cost(p: Pricing, u: TaskUsage) -> float:if u.cached_tokens > u.input_tokens:raise ValueError("Cached tokens cannot exceed total input tokens")fresh = u.input_tokens - u.cached_tokenstotal = (fresh * p.input_per_m+ u.cached_tokens * p.cached_per_m+ u.output_tokens * p.output_per_m)return total / 1_000_000if __name__ == "__main__":plan_a = Pricing(input_per_m=10, cached_per_m=1, output_per_m=20)plan_b = Pricing(input_per_m=8, cached_per_m=2, output_per_m=30)usage = TaskUsage(input_tokens=2_000_000, cached_tokens=1_600_000, output_tokens=50_000)print(f"Plan A cost per task: {task_cost(plan_a, usage):.4f} CNY")print(f"Plan B cost per task: {task_cost(plan_b, usage):.4f} CNY")
In this set of hypothetical data, Plan B has a lower non-cached input unit price, but its output unit price and cached unit price are higher. As a result, Plan A actually has a lower cost per task. This result shows that comparing only one unit price may lead to a conclusion that is the opposite of the real cost.
III. Output Length Determines Real Cost
Output tokens have a higher unit price, and output volume itself is often overlooked. If a model generates large blocks of irrelevant explanation, repeated code, or lengthy reasoning, total cost will rise even if the unit price is not high. Excessive output also increases response time and inflates the context in later rounds.
Output volume can be controlled through prompts. For example, requiring the model to return only modified code snippets or patches without additional explanation can usually reduce output tokens significantly.
Output constraints:1. Return only the files that need changes and their patches (unified diff format).2. Do not repeat unchanged code.3. Keep explanations to no more than three sentences, covering only the reason for the change.
Output constraints need to be validated together with evaluation. Constraints that are too strict may cause key information to be missing, so changes in success rate should also be monitored.
IV. Capability Evaluation: From Public Leaderboards to Real Workflows
Public leaderboards provide horizontal reference, but applying them directly has three limitations:
- The task distribution of a leaderboard may not match your own business.
- The same model can score very differently at different reasoning effort levels, so comparisons must state the configuration.
- Most leaderboards have limited coverage of tool use, multi-step task stability, and long-context performance.
Therefore, a more reliable approach for teams is to build an internal evaluation set. The evaluation set can cover three types of tasks:
- Isolated code generation, such as function implementation and unit test completion.
- Bug fixing: given error information and a code repository, require changes that pass specified tests.
- Multi-step engineering tasks, such as cross-file refactoring and API migration.
The following is an example structure for an evaluation configuration:
eval_suite:name: internal-coding-agent-v1tasks:- id: bugfix-001type: bugfixrepo_snapshot: "commit-hash"success_criteria: "all specified tests pass"max_steps: 30models:- name: candidate-Areasoning_level: mediumruns_per_task: 3metrics:- pass_rate- avg_input_tokens- avg_output_tokens- cache_hit_ratio- cost_per_task- wall_clock_seconds
Model outputs are stochastic, so each task should be run multiple times and the distribution of pass rates should be calculated. Cost and wall-clock time should also be recorded. Only when capability and cost are compared in the same table can a model selection conclusion be well grounded.
When comparing results at different reasoning effort levels, use the same level, or report data for each level separately.
