Offline Timed Evaluation of Code Agents: Calculating the Real Cost of Accepted Tasks

After a new generation of code agents is released, teams need to judge how capable the agent is and whether it actually saves money. Demos can overstate capability, while comparing only per-call prices can understate total cost. A more reliable approach is to run the model in a controlled task environment, with network access disabled, time limits enforced, and conditions close to real engineering constraints. Latency, manual review, failed retries, and risk costs should all be counted in the same framework.

Note that the figures in this article are illustrative. They were constructed to show the record format and calculation method, and they do not come from a verifiable public evaluation. Readers who want to cite specific conclusions should first find the original records, the evaluator, and the timing basis.

What an Offline Timed Evaluation Can and Cannot Show

When network access is disabled, external links cannot be reached. The agent can rely only on the model’s built-in knowledge, its reasoning, its code generation, and the information already in its context. This setup tests planning and code generation, but it cannot test retrieval-augmented capability. If the business scenario allows looking up documentation online, offline results will understate the agent’s performance.

The per-task time limit also shapes the conclusion. A 30-minute limit can show whether task planning is efficient, whether the agent knows when to stop, and what share of its effort goes to invalid attempts. It has limits too. If an agent does not finish a task within the limit, that does not mean it cannot do the task. Its approach to breaking down the task may simply not fit that time limit.

Small samples support only directional conclusions. Five or six tasks cannot produce a stable ranking. To compare two versions, a team should expand the sample to dozens of tasks that cover the main task types, and track variation across repeated runs.

Any judgment about real capability should therefore state its conditions: whether network access is allowed, the per-task time limit, the task types, whether human takeover is permitted, and whether test cases are run. Discussing strength or weakness without these conditions can mistake performance in one specific scenario for general capability.

Can Self-Reported Model Information Serve as Runtime Metadata?

During evaluations, when an agent states its identity, version, or knowledge cutoff, it often says that the information comes from the system prompt of its runtime environment, not from introspection. This explanation is reliable, because a model cannot inherently know which version it has been deployed as.

The engineering conclusion is straightforward. Model identifiers, version numbers, knowledge cutoff dates, system prompt versions, and tool permissions should be recorded by the deployment system. They should not depend on the model declaring them in a conversation. Otherwise, after the orchestration layer switches models or updates prompts, application teams will struggle to trace changes in behavior.

Knowledge cutoff dates should also be viewed in layers. Models usually remember earlier events fairly completely. Coverage thins for events close to the cutoff, and the model has no reliable memory of events after it. Time-sensitive content, such as news, prices, policies, dependency versions, and security advisories, must therefore be supplemented through controlled retrieval, an internal knowledge base, or human confirmation. Model memory is prior knowledge. It can serve as a reference, but it cannot replace a live source of facts.

What a Reproducible Evaluation Should Record

Evaluations often fail when they keep only conversation text, because conversations are hard to compare by task type, time limit, network status, and version. Structured records are more useful. The example record below is fictional. Its fields and values only illustrate the format:

task_record = {
  task_id: "T0",
  network: "disabled",
  time_limit_min: 30,
  elapsed_sec: 74,
  answer_source: "system_prompt",
  knowledge_cutoff: "2025-12",
  memory_confidence: "sparse_near_cutoff",
  human_takeover: false,
  validation: "manual_review"
}

Each field answers a specific question. elapsed_sec shows how long the task took. answer_source shows where the metadata came from. human_takeover shows whether the result depended on human intervention. Accumulating these fields over time lets a team compare how different versions perform on similar tasks.

Beyond single-task records, teams should also track time to first valid response, total elapsed time, task completion rate, test pass rate, lines of manual changes, retry count, timeout rate, share of invalid commands, and the merge rate of final patches. Recording only how long a task took is not enough. The team also needs to check whether the agent produced verifiable results within the time limit.

How to Calculate Cost per Effective Task

The denominator for cost-effectiveness should be the number of accepted tasks. Calculating cost by call count or price per thousand tokens ignores retries, human review, and regression fixes. A low-priced option that needs several retries, frequent human takeovers, and yields a low patch merge rate may cost more in practice. A slightly more expensive option with a high first-pass success rate, stable latency, and auditable results may be the better choice.

Teams can use the following formula for internal cost accounting:

effective_cost =
  model_cost
  + retry_cost
  + latency_cost
  + human_review_cost
  + regression_fix_cost
  + compliance_risk_cost
cost_per_accepted_task = effective_cost / accepted_task_count

accepted_task_count is the number of tasks that pass tests, pass review, and need no major rework. This formula is an internal accounting framework, not an industry standard. Each cost item must be calibrated to actual conditions. Convert human review using real labor rates. Estimate latency cost based on how sensitive the business is to waiting time. Set compliance risk cost together with the security or legal team. Until these items are calibrated, figures from different teams should not be compared directly.

How to Build a Layered Evaluation Plan

Divide the evaluation into four layers. The foundation layer checks identity metadata, knowledge cutoff, instruction following, and refusal boundaries. The code layer checks defect fixing, unit test generation, refactoring, dependency upgrades, and log troubleshooting. The engineering layer checks multi-file changes, long-context retention, safe tool use, and convergence before timeouts. The cost layer records latency, retries, human takeovers, and final merge rate.

Each task type should have fixed random seeds, environment images, dependency versions, and acceptance scripts. Measure offline and networked scenarios separately so their results do not mix.

The pseudocode below shows the execution order only. It cannot be run directly. The suite should come from a task set the team maintains, and every task needs an executable acceptance test.

# pseudocode: for illustration only
for task in suite:
    env = prepare_isolated_env(task)
    start = now()
    result = run_agent(task, env, network=off, timeout=30m)
    elapsed = now() - start
    quality = execute_tests(result.patch)
    record(task, result, elapsed, quality, human_takeover)
report(completion_rate, pass_rate, p95_latency,
       cost_per_accepted_task, retry_rate)

Checklist Before Selecting a Solution

Before selecting a solution, teams can use the table below to check each item. Each item should have a clear pass threshold, set according to the team’s business risk.

Evaluation layer	Required metrics	Example pass threshold
Foundation layer	Metadata source, knowledge cutoff, refusal records	Version information comes from the deployment system, and time-sensitive questions have external sources
Code layer	Test pass rate, patch merge rate	Test pass rate and merge rate meet the team’s set values
Engineering layer	Timeout rate, share of invalid commands, rollback rate	Timeout rate is acceptable, and rollbacks are traceable
Cost layer	cost_per_accepted_task, retry rate, human takeover rate	Cost per effective task is lower than the current solution

The thresholds in the table are examples. Set real thresholds from historical data and business risk tolerance rather than copying these values.

Conclusion

On the question of real capability and cost-effectiveness, a single offline, time-limited, small-sample evaluation can show whether the tested agent keeps handling tasks under restricted conditions, and whether it explains its own version and knowledge boundaries clearly. It cannot show full performance with network access, on large samples, or in real repositories. It also cannot show long-term stability.

This kind of evaluation fits as one part of a broader framework. A more reliable approach is to include it in continuous regression testing, track version changes with unified metrics, and make selection decisions based on the distribution of business tasks. Teams should first record reproducible structured data, then compare options by cost per accepted task. This turns agent capability into measurable, auditable engineering efficiency rather than a single demo or a unit price.