Evaluating Long-Horizon Multi-Agent Coding Capabilities: A Practical Guide

1. From Completing One Line to Days of Autonomous Work

Large language models have moved through three stages in software engineering.

• Stage 1: Code completion. The model handles only a few lines of context around the cursor, and its value lies in reducing repetitive keystrokes.
• Stage 2: Conversational assistance. Developers bring requirements to the model and ask questions; the model generates scripts, functions, or even small modules, but interaction remains mostly single-turn or a few turns.
• Stage 3: Long-horizon agents. The system receives an open-ended requirement—for example, “starting from an empty directory, build a bootable operating system prototype”—then autonomously decomposes the task, reads and writes files, writes and runs code, fixes errors, and works continuously for days. The output is a runnable system, not a code snippet.

This shift brings a frequently overlooked fact: long-horizon engineering tasks and single-turn coding problems demand different capabilities. A model that excels on algorithm-contest-style problems will not necessarily work reliably over days of autonomous development. There are three reasons:

1. Context expands. A long-running session keeps accumulating file contents, error logs, and trial-and-error records. Once it exceeds the context window, early constraints may be forgotten or even contradict later decisions.
2. Errors accumulate. In a single-turn task, a small error affects only the current output; in a long-horizon task, a single design flaw can force the rework of dozens of downstream modules.
3. Engineering judgment is required. “Writing fast” is not the same as “writing correctly.” The model must weigh maintainability, performance, and complexity across multiple viable options, and consistently follow the project’s established conventions.

Therefore, evaluating a coding agent cannot rely only on “can it write correct code”; it must ask “can it continuously produce verifiable results under long-duration, multi-module, and low-intervention conditions.”

2. Decomposing Long-Horizon Tasks: Where the Capability Comes From

Take “build a bootable operating system prototype from scratch” as an example. Once a goal that looks like it should be done in one shot is broken apart, you can see a whole set of composite capabilities supporting the agent.

2.1 Requirements Structuring: From a Vague Goal to Verifiable Specs

Real requirements are often just one sentence. The agent’s first step is to turn the goal into verifiable engineering constraints:

Goal: build a minimal viable operating system prototype on a simulated environment or physical machine
Must deliver:
- A standalone process-scheduling kernel
- Block device disk driver, keyboard input driver, minimal network protocol stack
- Persistent file system (data survives reboots)
- Window system, login screen, graphical installer
- Built-in model Q&A capability in the terminal
Acceptance criteria:
- Each module has corresponding automated tests, all passing
- The kernel image boots reliably in the specified environment
- Key paths include design docs explaining the rationale for tradeoffs

Without clear acceptance criteria, an agent easily “feels done” and ultimately delivers a half-finished product that cannot boot.

2.2 Task Decomposition and Parallelism: Sub-Agent Collaboration

A large project cannot be completed serially by a single agent. A common orchestration approach has a main agent act as the “architect,” splitting requirements into relatively independent modules, each assigned to a dedicated sub-agent. Using the sample prototype system, the decomposition looks like this:

Module	Responsibility	Typical verification
Kernel & scheduling	Process management, memory allocation, interrupt handling	Boot test, concurrency stress test
Block device driver	Read/write disk sectors	Read-write consistency check
File system	Directory structure, persistent storage	Data integrity check after reboot
Input & networking	Keyboard interrupts, basic network send/receive	Input injection test, loopback test
User layer	Window system, login screen, installer	UI automation test, install-flow rehearsal
Built-in model Q&A	Intelligent interaction in the terminal	Interface integration, end-to-end cases

Parallelism introduces three kinds of problems that must be designed for:

• Dependency conflicts. The UI depends on the file system, which depends on the disk driver, so an explicit dependency graph is required.
• Interface drift. When one module changes its interface, other modules may still be coding against the old interface.
• Integration debt. Modules that each “pass unit tests” often fail to work together once merged, so every round needs an integration gate.

2.3 Session Management for Long-Running Runs

A session that runs for days or even weeks must solve three problems: context, state, and continuity. General strategies include:

• Externalize memory. Use persistent state files to record goals, module decomposition, interface contracts, completed items, and open questions. Archive work logs by date, and load only what is necessary when a new agent starts.
• Layered summarization. Sub-agents compress local logs and report them upward; the main agent then generates a global summary while retaining key decisions and their rationale.
• Set checkpoints. Save a snapshot at each verifiable milestone, and roll back to the most recent usable state on failure to avoid unbounded error accumulation.
• Fixed time boxes. Set a time limit for each single-turn task; on timeout, interrupt and have a human or rule-driven check decide whether the direction needs adjusting.

2.4 Implementing Context Compression

The bottleneck in long-horizon tasks is usually not “the model can’t write code” but “context is poorly managed.” A simplified compression strategy can be implemented like this:

import json
from datetime import datetime
# Each compression keeps three types of information: open goals, key decisions (with rationale), failure lessons
def compress_context(history, budget_tokens):
    """
    Compress the history by token budget, keeping information relevant to downstream decisions.
    history: a chronologically ordered list of events, each with type, timestamp, content
    """
    keep_types = {"decision", "blocker", "spec_change"}   # decisions, blockers, spec changes
    recent_window = history[-20:]                         # recent raw records for traceability
    kept, used = [], 0
    # Walk backward from the most recent events, prioritizing high-value events
    for event in reversed(recent_window):
        if event["type"] in keep_types:
            kept.append(event)
            used += event["tokens"]
            if used > budget_tokens * 0.6:
                break
            continue
        if used > budget_tokens:
            # Old logs beyond budget are no longer kept verbatim; replace with a count-style summary
            summary = f"{event['timestamp']}: log {event['summary']}"
            kept.append({"type": "log_summary", "content": summary,
                         "tokens": event["summary_tokens"]})
            used += event["summary_tokens"]
    # Execution environment state is always fully retained: current file list, latest test results
    env_state = [e for e in recent_window if e["type"] == "env_state"]
    return json.dumps({
        "kept_events": list(reversed(kept)),
        "env_state": env_state,
        "truncated_old_logs": sum(1 for e in history[:-20]
                                  if e["type"] not in keep_types),
    }, ensure_ascii=False)

The key point: what gets compressed is detail; what gets kept is the basis for decisions and the execution state. A deleted debugging log can be replaced by a one-line summary, but a decision like “why this memory allocation strategy was chosen” is hard to recover once removed.

3. Evaluation Framework: How to Measure Agentic Coding Fairly

The capability leaderboards that model vendors publish often give rankings across different task categories. For enterprises, what matters more is understanding what these dimensions mean and building an evaluation system that fits their needs.

3.1 Design Evaluation Sets by Task Type

The following task types place different demands on agents; all are worth including in an evaluation:

• Software migration. Migrate a legacy system’s implementation from one language to another with equivalent behavior. This tests semantic understanding, cross-language comparison, and compatibility test generation.
• Hardware and embedded development. Writing drivers and designing digital circuits cannot be verified by “reading code” alone; they require simulation or target hardware, and they test hardware-software co-debugging ability.
• Scientific and engineering computing. Tasks involving numerical modeling and simulation solving require the agent to understand the domain background and verify results with tools.
• Mathematical formal verification. Translating mathematical propositions into machine-checkable proofs, such as famous hard problems like Fermat’s Last Theorem. The difficulty is that every reasoning step must be strictly valid, making these among the most demanding tests of reasoning ability.
• Robotics evaluation. Assessing control policies and algorithms usually requires a simulation environment, and tests closed-loop debugging ability.

For an ordinary R&D team, it is unnecessary to copy every category, but the same design principles should hold: real tasks, automatically determinable results, and reproducible initial environments.

3.2 Use “Workload Tiers” Instead of a Single Score

A single overall score easily hides how models perform at different difficulty levels. A more reasonable approach divides test tasks into multiple tiers by scale: from single-file edits, to medium tasks spanning multiple modules, to extra-large tasks requiring days and dozens of modules. Score each tier separately, and record the token consumption required to meet the bar at the same time.

Early tests published by one vendor showed that a new-generation model scored roughly on par with the previous flagship at high workload, at about half the cost per task; at extra-large workload, it scored ahead of other models available at the time. This suggests that the new-generation model relies less on “maxing out the workload” to trade for score. But note that such data comes from a specific test set, and enterprises cannot directly infer benefits for their own business from it.

3.3 Fair Comparison Under a Fixed Budget

When comparing models, follow these principles:

• Results decided by code. Judge automatically through compilation, unit tests, integration tests, or formal proof checkers, not primarily by subjective human scoring.
• Same ceiling. Give each model the same runtime limit, total token budget, or call-count cap, and run them in the same initial environment.
• Repeat runs and look at the distribution. Because large-model output is stochastic, run the same task multiple times and observe success rate, stability, and the type distribution of failure modes.

3.4 Distinguishing Real Passes, Cheating Passes, and False Passes

The most overlooked risk in automated evaluation is “tests pass but the functionality is wrong.” Three common problems:

• Evasion-style fixes. Instead of fixing the defect, modify or skip test cases so results “go green.”
• Over-modification. Refactoring large amounts of unrelated code along the way, introducing new regression risk.
• Surface completion. The deliverable runs in the test environment but fails in a real one.

The defense is to place the verification logic somewhere the agent cannot modify, combined with code review rules:

# Evaluation framework pseudocode: isolate the code under test from verification logic
from pathlib import Path
# The directory containing verification scripts is declared read-only, outside the agent's writable workspace
def run_evaluation(agent, task):
    before = snapshot_of_tests(task.test_dir)      # record hashes of the test files
    agent.work_on(task.workspace)                  # the agent can only modify the code under test
    assert snapshot_of_tests(task.test_dir) == before, \
        "Tests or verification logic were modified; the result is invalid"
    return {
        "build_ok": task.build(),
        "tests_passed": task.run_all_tests(),
        "diff_lines": Path(task.workspace).diff_size(),
    }

Also check the diff size and change scope during review. When fixing a single defect produces thousands of lines of changes, focus the review on whether unrelated changes were introduced.

4. Orchestration Architecture: Engineering Patterns for Sub-Agent Collaboration

If you view an agent team as a development team, the orchestration framework plays the role of “engineering manager.” A usable minimal architecture:

class Orchestrator:
    """Main agent: responsible for planning, dispatching, integration, and acceptance"""
    def run(self, spec):
        plan = self.planner.decompose(spec)           # decompose requirements into a dependency-aware task graph
        pending = plan.ready_tasks()                  # tasks with no unsatisfied dependencies
        done = set()
        while pending:
            batch = pending[:self.max_parallel]       # cap the number of concurrent sub-agents
            results = self.dispatch(batch)            # run in parallel and collect results
            for task, result in results:
                if result.passed_gate():              # compile + unit tests + static checks
                    done.add(task.id)
                else:
                    # Retry failed tasks up to two times, carrying the failure context
                    task.retry(result.error_digest())
                    if task.retries_exceeded():
                        self.escalate(task)           # escalate to human or main-agent decision
            # Update the ready-task list once dependencies are satisfied
            pending = plan.ready_tasks(exclude=done)
        return self.integrate_and_verify(plan)        # integration build and end-to-end acceptance

Supporting mechanisms include:

1. Capability isolation. Each sub-agent can only access files and tools relevant to its own task, reducing interference and the risk of privilege abuse.
2. Interface contracts first. The main agent defines inter-module interfaces first, sub-agents implement against the contract, and interface changes must return to the main-agent layer for unified review.
3. Integration gates. Run a full integration build after each batch of tasks to surface conflicts early.
4. Failure attribution. Distinguish “unclear requirement description,” “flawed implementation,” and “unstable environment,” and apply different handling strategies to avoid simple retries masking root causes.

5. Deterministic Verification: The Bridge Between Probabilistic Model Output and Engineering Determinism

The most core value of long-horizon agents is this: the model’s output is probabilistic, but the engineering acceptance criteria are deterministic. Connecting the two requires a verification layer that is as objective and repeatable as possible.

5.1 Verification Methods, from Weak to Strong

• Syntax and format checks. Cheapest, but they only catch surface problems.
• Compilation and type checks. Surface interface inconsistencies early.
• Unit and integration tests. Cover functional correctness; a baseline test is needed to prevent regressions.
• Static analysis and security scanning. Find potential defects and high-risk patterns.
• Runtime verification. Actually run the deliverable in an isolated environment. For example, a “bootable OS prototype” should be booted from a virtual disk image in a simulated environment, checking key functions such as files, processes, and networking.
• Formal verification. Provide machine-checkable proofs for critical modules; the highest reliability, but at a higher modeling and toolchain cost.

5.2 Make Verification Results Executable

Verification scripts should output structured results for the orchestration layer to judge automatically:

{
  "task_id": "fs-017",
  "build": "passed",
  "tests": {"total": 42, "passed": 40, "failed": 2},
  "static_check": {"errors": 0, "warnings": 3},
  "runtime_smoke": "failed",
  "failed_cases": ["fs_mount_after_reboot", "fs_concurrent_write"],
  "reproducible": true
}

For mathematical formal verification, the conclusion must be decided by a proof checker:

Verification target: for any input satisfying the constraints, the scheduler will not experience priority inversion
Checker output: proof accepted (1,284 reasoning steps in total, all passing type and logic checks)

The right to judge lies with the tools, not the model. This is the key to trustworthy agent output.

6. Cost Efficiency: Why “Cost per Unit of Intelligence” May Be Falling

When many teams first adopt agentic coding, the first thing they feel is a rising bill. Cost evaluation cannot look only at the per-call price; it must consider the overhead across the full pipeline.

6.1 The Cost Structure of Agent Mode

• Repeated context overhead. Each turn must resend the system prompt, tool definitions, and conversation history. In long sessions, this consumption often exceeds newly generated content.
• Reasoning consumption. Models that include a thinking process consume large numbers of tokens before the formal output, and this is billed too.
• Retries and rework. Retrying after failure and redoing after rollback add extra cost.
• Human review. Code review, integration acceptance, and security assessment still require human effort.

A common way to estimate:

single-task cost ≈ input token count × input price per token + output token count × output price per token + human review cost

Because long-horizon tasks have input far exceeding output, the token savings from context management mechanisms often matter more than the difference in model unit price.

6.2 Model Routing: Choose “Compute” by Task Difficulty

Not every task is worth the strongest model. A practical tiering strategy:

• Routine tasks. Such as fixing defects with clear errors, adding unit tests, and generating boilerplate—hand these to fast, low-cost models.
• High-difficulty tasks. Such as architecture design, complex troubleshooting, and long-horizon refactoring—hand these to more capable models.
• Critical security tasks. For scenarios like vulnerability auditing, use multi-model cross-validation to reduce the risk of a single model’s misses.

In a self-built platform, the routing logic can dynamically choose based on task type, code change size, and historical success rate. Use monitoring and alerts to track token consumption and task success rate, catching cost anomalies promptly. Cloud providers’ model services typically offer models at different specs and price tiers; specific parameters and billing rules should follow the official documentation. On platforms such as Baidu AI Cloud, this kind of multi-model access and usage monitoring is a common platform capability that enterprises can configure according to their own needs.

6.3 What “Fewer Tokens for Harder Tasks” Means

Vendors often emphasize “completing harder tasks with fewer tokens.” Behind this are three meanings:

• Better planning quality. Better task decomposition reduces ineffective exploration and repeated trial and error.
• More efficient context use. The model accurately extracts key constraints from existing information, without repeatedly reading the same files.
• Faster failure recovery. After an error, it can quickly locate the root cause instead of blindly trying.

For enterprises, this suggests a principle: when evaluating the ROI of agentic coding, focus on “the total cost of completing a full task,” not “the price per million tokens.”

7. Security Risks: The Stronger the Capability, the More Important the Boundaries

Vendors often emphasize security at release. For example, internal evaluation tools show that in rediscovery tests of publicly known vulnerabilities, a new-generation model found more known vulnerabilities than the previous flagship and its own previous version, respectively. This sends a clear signal: the model’s ability to understand attack surfaces and to defend is growing in tandem. Enterprises should both leverage this ability to strengthen defense and guard against its misuse.

7.1 The Meaning and Limits of Vulnerability-Rediscovery Evaluation

Using publicly disclosed vulnerabilities (i.e., CVE records) as clues, having the model re-locate defects in the pre-patch old version of the code is a way to measure the model’s security analysis ability. It shows the model can understand the root cause of a vulnerability, not just memorize known patterns.

This capability can be applied to:

• Automated auditing of your own code, catching similar-pattern problems early;
• Component and version checks on third-party dependencies;
• Patch verification, confirming whether a fix is thorough.

But it can also be used to find attack targets. Therefore, the relevant tools and processes must be limited to authorized use, with complete operation audit records retained.

7.2 Recommended Tiered Controls

Risk level	Applicable tasks	Controls
Low	Routine feature development, test generation	Code review, automated test gates
Medium	Changes involving external dependencies, network interfaces	Static security scanning, dependency version checks, isolated runtime environment
High	Security vulnerability analysis, crypto, authentication-related code	Authorized personnel access only, dedicated isolated environment, two-person review, operation auditing

In addition:

• Sandboxed execution. Run agent-generated code and commands in an isolated environment, restricting network access and file system permissions.
• Sensitive information protection. Prohibit the agent from writing secrets, tokens, and other sensitive information into logs, comments, or commit messages, and block them with scanning rules.
• High-risk operation approval. Irreversible operations such as deleting data or modifying production environment configuration must go through human approval.

It is worth noting that risk controls like biosafety occur mostly during model training and release, which is the responsibility of the model provider. What enterprise users can do is: set usage boundaries at the application layer, avoid using agents in high-risk scenarios, and keep review over content involving external access.

8. Practical Recommendations: Build Your Own Agentic Development Pipeline

Synthesizing the above, enterprises can advance their agentic development pipeline in these steps:

1. Build a private evaluation set. Select 50–100 tasks from real code repositories, covering four categories—defect fixing, feature development, code migration, and test generation—each with an automated verification script.
2. Define core metrics. Include task pass rate, average token consumption, number of human interventions, and average completion time.
3. Design the orchestration and verification architecture. Following the earlier patterns, build the main agent, sub-agents, and verification layer, isolating verification logic from the agent workspace.
4. Adopt incrementally. Trial in non-core modules first, then gradually expand to core business.
5. Establish security and compliance processes. Clarify tiered authorization, auditing, and review requirements.
6. Evaluate and iterate continuously. Periodically update the evaluation set with new tasks and track capability changes after model upgrades.

The significance of long-horizon agentic coding is not to replace all engineers, but to free engineers from repetitive labor so they can focus on architecture design, requirement judgment, and quality control. Understanding the sources and boundaries of model capability, and building an engineering system that is verifiable, controllable, and measurable, is what truly turns an agent’s productivity into stable delivery capability.