Long-Horizon Code Agents: Evaluation, Hallucination Control, and Cost Optimization Practices

    This article examines what sets long-horizon code agents apart and how to build reliable systems around them. It breaks long-horizon capability into three levels—single-step correctness, multi-turn consistency, and long-horizon convergence—and shows how to design trustworthy evaluation inside restricted sandboxes using pass@k. It then covers engineering governance of factual errors through layered metrics and generation-verification loops, how to measure abuse-risk tradeoffs, and a practical cost model built on context caching, caching-friendly prompt structure, TTL selection, and standard/high-speed tier routing. It closes with a pre-launch engineering checklist covering sandboxing, idempotency and rollback, cost guardrails, observability, and repeatable evaluation. You will come away with a concrete framework for evaluation criteria, quantifiable governance, and an optimizable bill structure.

    1. “Long-Horizon” Is the Real Dividing Line for Code Agents Today

    For a long time, ways of measuring a model’s coding ability stayed at “point-based” metrics such as single-turn code generation and unit test pass rates. But in real development and operations scenarios, tasks look more like this: given a vague goal, the agent must autonomously break it down, iterate through trial and error, call the terminal and tools, and converge to a verifiable result after dozens or even hundreds of steps. The difficulty here is not single-step correctness, but error accumulation and state management—a wrong environmental judgment made early on may only surface as a build failure twenty steps later.

    So when evaluating long-horizon capability, at least three levels need to be distinguished:

    1. Single-step correctness: syntactically correct, with locally sound logic;
    2. Multi-turn consistency: not losing the goal under context compression and noisy tool returns;
    3. Long-horizon convergence: reaching a verifiable “task complete” state within a limited number of steps.

    2. How to Design a Credible Evaluation System

    2.1 Terminal-Restricted Tasks: Put Evaluation Inside a Real Sandbox

    Reliable agent evaluation should satisfy three conditions: tasks have objective pass criteria (tests pass, file artifacts exist, a service port is reachable); the environment is restricted (controlled network policy, least privilege); the toolset is explicitly enumerable. Only under “restricted” conditions does a score reflect the model’s planning and reasoning ability rather than reliance on retrieval over an open network.

    It is worth recording the cost of passing alongside the pass rate:

    1. # Minimal scoring skeleton for terminal-task evaluation (illustrative)
    2. def score(task, run_result):
    3. passed = run_result["check_command"].returncode == 0
    4. return {
    5. "pass": passed,
    6. "k": run_result["steps"], # number of steps
    7. "tokens": run_result["total_tokens"],# token consumption
    8. "human_intervene": run_result["escalated"], # whether a human was escalated to
    9. "rollback": run_result["had_rollback"], # whether a destructive operation occurred
    10. }

    Report results for the same task family with pass@k, and always state the value of k and the sampling temperature; otherwise conclusions are not comparable across runs.

    2.2 Long-Horizon Tasks: Internal Benchmarks Should Serve Improvement, Not Marketing

    The value of a self-built long-horizon benchmark (for example, a long-horizon subset of an internal code benchmark) lies in longitudinal comparison across versions: judging whether a new release delivers substantive gains on tasks with “more steps and more complex constraints,” rather than padding scores on easy tasks. Two things need attention when using one:

    • Overfitting risk: once an internal benchmark is used for tuning, its absolute values become distorted, so keep a holdout portion that is never used in training;
    • Correlation check with public benchmarks: a conclusion has external validity only when the internal benchmark’s relative gains can be reproduced across several independent public benchmarks.

    2.3 Vertical Scenarios: Use “Toolchain Programmability” as a Proxy Metric

    Tasks such as parametric modeling, engineering drawings, and office document production are essentially “driving professional software through scripts.” Evaluating this capability does not have to rely on subjective impressions; you can use structured artifact metrics:

    • 3D modeling: the editability of geometry (whether a parameter tree is preserved rather than just an exported mesh), key dimension errors, and script re-executability;
    • Office documents: structured field completeness, formula and citation consistency, and ease of secondary editing;
    • General engineering: whether scripts are idempotent and whether they can be safely re-run after a failure.

    3. Engineering Governance of Factual Errors

    3.1 First, Break “Hallucination” Into Measurable Metrics

    A single “accuracy” figure cannot guide optimization; break it into at least three levels:

    • Single-point factual error rate: whether each independently verifiable statement is wrong;
    • Proportion of answers with errors: whether at least one factual error appears in a single answer;
    • Error severity: harmless deviation, misleading statements, and high-risk errors that can lead to wrong decisions.

    You must also state how the test set was constructed. A stress set deliberately built from error-prone cases (fine-grained numbers, time-sensitive information, long-citation reconstruction, cross-document aggregation) naturally has a higher error rate than the everyday distribution and must not be equated directly with everyday usage experience, or it will mislead readers about actual capability.

    3.2 Combined Governance on the Generation and Verification Sides

    1. # Lightweight verification loop for number and citation consistency (illustrative)
    2. def verify(answer, evidence):
    3. issues = []
    4. for num in extract_numbers(answer):
    5. if not number_supported(num, evidence): # does the number have a source
    6. issues.append(("unsupported_number", num))
    7. for cite in extract_citations(answer):
    8. if not citation_exists(cite, evidence): # does the citation actually exist
    9. issues.append(("bad_citation", cite))
    10. if issues:
    11. return regenerate(answer, issues) # rewrite with the list of issues
    12. return answer

    The more effective engineering combination is: retrieval augmentation to provide verifiable evidence + enforced citation constraints + re-checking of numbers and units + structured-output schema validation. In scenarios that have a test environment, unit tests themselves are the cheapest and most reliable factual checkers.

    4. Measuring Abuse-Risk Control and Capability Tradeoffs

    A common approach to governing high-risk capabilities is to identify and remove data with specific abuse tendencies during training-data construction, then reinforce behavioral constraints during alignment. The key engineering question is whether the cost can be quantified: compare the “unfiltered” and “final release” versions on the same evaluation set using pass@k, and record the difference as a public, trackable tradeoff metric rather than reporting only the conclusion.

    The usage side needs a closed loop too: secondary confirmation for high-risk operations, least-privilege accounts, complete audit logs (recording the caller, the target resource, and the state before and after a change), and a rollback-capable channel for changes.

    5. Cost Model: Caching, Routing, and Batching

    Context caching has become the core cost variable for long-horizon agents. A common industry structure has three tiers: a base input price, a cache-hit price (usually well below the base input price), and a cache-write price (tiered by short-term and long-term TTL, with longer TTL costing more). Cost can be estimated with the following formula (all unit prices are per million tokens):

    1. def cost(input_tok, output_tok, cached_tok, written_tok,
    2. p_in, p_out, p_hit, p_write):
    3. billable_new = max(input_tok - cached_tok, 0)
    4. return (billable_new * p_in
    5. + cached_tok * p_hit
    6. + written_tok * p_write
    7. + output_tok * p_out) / 1_000_000

    This yields three optimization principles you can apply directly:

    1. Caching-friendly prompt structure: put the stable system prompt, tool definitions, and codebase context in the prefix and dynamic content at the tail, maximizing the hit region;
    2. Match TTL to batching: long-TTL writes cost more and are only worthwhile for reuse across turns, while short-lived tasks should use a short TTL;
    3. Split standard and high-speed tiers: some services offer a low-latency tier at a fixed multiplier (the multiplier typically applies proportionally to each billing item, leaving the cache-hit price unchanged). Route by SLA—use the high-speed tier for interactive coding and the standard tier for overnight batch refactoring and batch script generation—to significantly reduce the overall bill.

    When deploying model services in the cloud, you can lean on general cloud capabilities: container platforms host the inference and sandbox execution environments, object storage holds evaluation artifacts and long-horizon task traces, message queues smooth out task peaks, and log services plus monitoring and alerting provide observability into token consumption, cache-hit rate, and task success rate. When building such a pipeline on Baidu AI Cloud, these same general product categories can be reused, without designing infrastructure separately for each scenario.

    6. Pre-Launch Engineering Checklist

    • Sandbox and isolation: the agent’s execution environment defaults to no external network and no production credentials, and destructive commands require an explicit whitelist;
    • Idempotency and rollback: all write operations are safe to re-run, with snapshots taken before and after each change;
    • Cost guardrails: per-task token and step limits, timeout circuit breaking, and alerts when the cache-hit rate is low;
    • Observability metrics: task success rate, average steps, human-intervention rate, cache-hit rate, and per-task cost—these five make up the day-to-day operations dashboard;
    • Evaluation readiness: run the same set of terminal tasks and the long-horizon holdout set on every version switch to guarantee longitudinal comparability.

    7. Conclusion

    The competitiveness of a long-horizon code agent ultimately rests on three things together: the ability to finish tasks in a restricted, realistic environment, the degree of control over factual errors and abuse risks, and the predictability of per-task cost. For developers, rather than chasing a single leaderboard number, it is better to build your own task set and cost model—controllable evaluation criteria, quantifiable governance measures, and an optimizable bill structure are the prerequisites for truly embedding agents into production workflows.