How to Evaluate the Real Productivity of Long-Horizon Coding Agents

    Switch the evaluation axis from leaderboard scores to end-to-end performance on long-horizon tasks and the picture becomes clear: the current generation of open-source code models has pushed the ceiling of long-horizon coding further, carrying changes across files, layers, and modules to a compilable, runnable, testable state. They are not yet safe to ship on a green test run — a legacy-project stress test surfaced a P1 object-level authorization flaw that passed every automated case. This article lays out a reproducible two-track evaluation method, a six-dimension scorecard, two hands-on cases (legacy modification and from-scratch build), and a capability tiering that separates what agents can own from what must stay with humans.

    1. The conclusion first: coding ability needs a new coordinate system

    If you replace “leaderboard score” with “end-to-end performance on long-horizon tasks” as the yardstick for evaluating coding agents, the picture becomes fairly clear: this generation of open-source code models has pushed the ceiling of long-horizon coding noticeably higher. It can read complex context inside a large codebase, keep driving connected changes across the frontend and backend and across modules, and carry a task to a state that is compilable, runnable, and testable. That is already a practical dividing line: the move from “generating code snippets” to “delivering a feature change.”

    But it is still clearly short of “all tests green, ship with confidence.” In a deliberately stressed hands-on run, both tasks went through: one added a capability to a legacy project carrying historical baggage and security constraints; the other implemented, entirely from zero, a knowledge-learning app that can actually be played to the end. Both held up at the functional level, yet the legacy project exposed a P1 defect during manual acceptance — object-level authorization isolation had not been done properly. Every automated test was green and every case passed, and the flaw was still triggerable as an unauthorized access path.

    The more pragmatic positioning is therefore this: long-horizon coding capability is now strong enough to be stress-tested on real projects, and the payoff is largest on cross-file, cross-layer coordinated changes; but wherever permissions, secrets, audit trails, and final acceptance are involved, human judgment cannot be absent.

    2. Why long-horizon tasks expose capability boundaries better than one-shot generation

    Short tasks — writing a function, filling in a unit test — produce almost no “capability amplifier” effect. Long-horizon tasks are different: they force several hidden capabilities out into the open.

    Error accumulation. In a long task, every decision becomes a premise for the steps that follow: interface signatures, field naming, state machine values, error-return conventions. One arbitrary convention chosen early turns into debt requiring wide rework dozens of steps later. Whether the model can stop and converge on a design at the right moment distinguishes skill far better than single-step generation quality.

    Repository-level context management. In a real system, “change one feature” usually means touching contract definitions, data tables, task state, server APIs, background workers, frontend pages, and the rendering layer at the same time. The model has to do its own retrieval and localization: find the real write point rather than the call site, identify which callers depend on the old behavior, and judge which table needs a compatible migration. However large the context window, an active retrieval strategy is required rather than stuffing the whole repository in.

    Toolchain closure. Progress on a long task depends on whether the compile → run → read logs → locate → fix loop can spin automatically. If the model cannot read build failure output, cannot locate a fault from a stack trace, and cannot write a minimal reproduction case, it can only guess at the semantic level instead of converging at the engineering level.

    Explicitness of constraints. Business constraints and security baselines are often not in the code; they live in documents, review records, and sometimes verbal agreements. A constraint the model cannot see is a constraint that does not exist — which is precisely the root cause of the security defect in Section 4.

    3. A reproducible evaluation method

    For an evaluation to be reusable and comparable across models, it cannot rest on subjective impressions. Organize it in the four steps below.

    3.1 Task set design: two complementary tracks

    • Track A: legacy system modification. Add a medium-complexity feature to a project that has legacy code, existing dependencies, and explicit security constraints. What is under test: reading an old system, respecting established conventions, and introducing no regressions.
    • Track B: build from zero. Supply only a requirements document and a reference document, and require an application with a closed feature loop that can complete the main flow end to end. What is under test: completeness from information analysis through product delivery, including scope trimming and prioritization.

    The two tracks have almost no overlap in failure modes: Track A typically loses points on regressions and constraint violations, Track B on scope explosion and “looks finished” pseudo-features. Relying on a single track yields a skewed conclusion.

    3.2 Environment isolation and observability

    • Run every task in a containerized sandbox with fixed repository snapshot, dependency versions, and build parameters, so results are reproducible.
    • Retain the full set of artifacts: change sets, execution logs, command history, test reports, rendered screenshots or screen recordings.
    • Record objective quantities: end-to-end elapsed time, number of human intervention rounds, build failure counts, rework counts. These are far more useful than “it feels pretty strong.”

    3.3 Metric definitions and scorecard

    Dimension How it is observed Pass criteria High-frequency failure points
    Runnability First-pass and final build results Final state must be runnable; first-pass compile rate reflects one-shot success rate Declaring completion without verification
    Cross-module consistency Alignment checks on contracts, storage, services, frontend fields Fields and semantics consistent end to end, no leftover TODO breakpoints Changing only one side; caller types do not match
    Regression risk Existing test suite and smoke cases New features do not turn existing tests red Editing old assertions to make new tests pass
    Security Dedicated cases for unauthorized access, injection, upload, path handling Object-level authorization enforced at the data access layer Only checking “is the user logged in”
    Maintainability Change set review Naming, layering, and error handling follow the project’s existing style Reinventing wheels, direct cross-layer access
    Wrap-up quality Documentation and self-test notes Change description, migration steps, and rollback plan all present No migration or rollback notes

    3.4 One thing that is easy to overlook

    Do not look only at the final result; preserve the record of how the model itself decided it was done. Did it add verification cases, or did it simply declare completion? That difference directly determines whether it can be trusted to run unattended on a long task.

    4. Case 1: how the security boundary leaked in a legacy modification

    4.1 Task setup

    Add one capability to the legacy project: let users upload their own image assets and let animation scenes reference those assets during rendering. The change spans the upload entry point, the asset metadata table, background processing jobs, read authorization for private storage, and reference resolution at scene render time. The task statement explicitly required “must not break existing tenant isolation.”

    4.2 Tests all green, yet an authorization bypass exists

    The feature worked: uploads succeeded, the asset list was correct, scenes referenced and rendered the assets properly, and every automated case passed. Only during manual acceptance did a P1 problem surface — authorization checked “is the user logged in” but never “does this resource belong to the current user.” Swap in any arbitrary asset ID and you can read someone else’s content.

    1. # Counter-example: verifies identity only, never resource ownership
    2. @route("POST /assets/import")
    3. def import_asset(req, user):
    4. # user is used for authentication only, and plays no part in any ownership check
    5. record = db.assets.find(req.asset_id) # an attacker can pass any ID
    6. db.scenes.append_ref(record.id)
    7. return {"url": record.public_url} # public direct link, downloadable by anyone

    Problems of this kind tend to arrive in clusters and are worth working through as a checklist:

    1. Missing object-level authorization. Login state is verified, but there is no “subject–resource” ownership check, which opens up unauthorized read and write.
    2. Exposed storage direct links. Returning a long-lived public URL amounts to bypassing application-layer authorization.
    3. Careless path handling. Concatenating a client-supplied filename into the storage path without normalization or ownership isolation can allow directory traversal or overwriting other users’ files.
    4. Type validation based on extension or request header only. Disguised files can be uploaded and later treated as executable content at the rendering stage.
    5. User content entering the rendering context directly. Scene references splice external content into page templates without escaping or sandbox isolation, creating a script injection surface.
    6. No quotas or resource limits. Without size, count, or decode-dimension limits, the service is exposed to resource exhaustion.

    4.3 Root cause: why models systematically miss this class of problem

    Tests are written along the path the model itself implemented. Models tend to construct cases following the call order in their own heads, which naturally covers the happy path. An authorization test requires composing “legitimate identity + someone else’s resource,” and that attacker’s perspective does not arise on its own.

    A positional preference for permission logic. Models tend to place checks at the entry layer — routes, gateways, decorators — to decide “is this caller allowed through the door,” whereas object-level isolation must be expressed at the data access layer together with ownership conditions. An entry-layer check stops applying the moment the code moves away from that entry point.

    The constraint never entered the context. Requirements like “tenant isolation” and “assets are private,” if they exist only in team conventions, are requirements the model has no way to honor. A security baseline that cannot be seen is equivalent to one that does not exist.

    4.4 Fix direction

    1. # Fix: enforce ownership and tenancy at the data access layer
    2. def get_asset(actor, asset_id):
    3. row = db.assets.find_one(
    4. _id=asset_id,
    5. tenant_id=actor.tenant_id, # tenant condition goes into the query, not an after-the-fact check
    6. owner_id=actor.id, # object-level ownership condition
    7. )
    8. if row is None:
    9. raise NotFound() # uniformly return "not found" to avoid resource enumeration
    10. return row

    Three supporting measures: switch storage to private reads with short-lived signed URLs; generate all upload and read paths server-side and never accept a path from the client; escape user content on the rendering side and place it in an isolated context, disallowing external links and inline scripts.

    5. Case 2: building an app from zero that can actually reach the end

    5.1 Task setup

    Supply only a requirements document and a domain reference document, and ask for a knowledge-learning application: questions and knowledge entries, level progression and unlocking, scoring and progress persistence — and it must be possible to go from the first level all the way to completion.

    5.2 Typical failure modes

    • Scope explosion. Early effort goes into a visually polished home page while the core question data structure and progress state machine stay rough, and the main flow does not actually work.
    • Pseudo-features. Every UI element is present but nothing real happens on click, and data is not connected across pages; the first acceptance test sees straight through it.
    • The completion path is unreachable. Unlock conditions contradict the level data (for example, prerequisite levels never issue an unlock token), so the flow deadlocks partway through.
    • Missing state consistency. Score, unlock state, and saved progress are each written in several places, so state drifts after a refresh or re-entry.

    5.3 Verification: turn “reaching the end” into an executable assertion

    From-scratch builds are the easiest tasks to be fooled by “it looks usable in the demo.” A reliable criterion is to write the main flow as automated cases, have a script drive from the initial state all the way to the completion state, and assert the key state transitions:

    1. Case: the main flow can be completed end to end
    2. 1. Enter as a new user and assert the progress is in its initial state
    3. 2. Complete levels 1..N in order, asserting per level: score delta, next level unlocked, save persisted
    4. 3. Inject two kinds of anomaly: resubmitting the same level, and jumping to an unlocked level out of order
    5. Assert: resubmission does not score twice; out-of-order access is rejected and does not corrupt existing progress
    6. 4. Log in again and assert progress and score match the values at logout
    7. 5. Reach the final level and assert the completion state is achieved and cannot be triggered again

    The value of this case set is that it turns “can it be finished” from a subjective impression into an objective conclusion, while also covering the two high-frequency defect areas: state consistency and unauthorized access.

    6. Capability tiering: what you can hand over, and what must stay with people

    Safe to hand over (highest payoff)

    • Cross-file, cross-layer feature delivery and integration, including synchronized changes to contracts, storage, service interfaces, background jobs, and the frontend
    • Boilerplate code, repetitive refactors, bulk replacements, and cascading fixes after dependency upgrades
    • Adding tests, documentation, migration scripts, logging, and observability instrumentation
    • Targeted refactoring and defect fixing, especially problems with a clear reproduction path

    Requires close human involvement

    • Architecture-level trade-offs, data models and migration strategy, compatibility and gradual-rollout plans
    • Changes that rest on assumptions about external dependency behavior, or on performance and capacity assessments

    Must be owned by humans; the model cannot certify itself

    • Object-level authorization and multi-tenant isolation, key and credential management, audit log integrity
    • Attack-surface code: upload and download, rich text and template rendering, externally exposed APIs
    • Final acceptance and security review sign-off

    On the division of labor, the more accurate phrasing is “human on the loop” rather than “human in the loop”: people no longer write code line by line, but they remain accountable for task definition, constraint setting, and acceptance criteria.

    7. Five practical recommendations for team adoption

    1. Treat context as an engineering problem. Provide structured requirement specs, module responsibility descriptions, a data dictionary, existing tests, and local run scripts. Context quality usually has a larger effect on output than the fine-grained differences between models.
    2. Express constraints as machine-checkable rules. Do not leave the team security baseline in a document; encode it as static analysis rules, policy-as-code, and CI gates so that unauthorized access, plaintext secrets, and insecure dependency versions fail directly in the pipeline:
    1. deny[msg] {
    2. input.kind == "data_access"
    3. not contains(input.filters, "tenant_id")
    4. msg := "missing tenant-dimension access filter"
    5. }
    1. Build a layered acceptance pipeline. Layer 1: compilation and unit tests. Layer 2: static scanning and dependency vulnerability checks. Layer 3: dedicated security cases (unauthorized access, injection, upload, path handling). Layer 4: human review and acceptance. The model works to keep the first three layers as green as possible; the fourth always belongs to people.
    2. Design checkpoints for long tasks. Require the model to produce a change set summary, migration steps, and a rollback plan at key points, and to run verification on its own initiative. Trust in a long task comes from being inspectable midway, not from the final result.
    3. Cover deployment and operations with general-purpose capabilities. For self-hosted or private deployments: put model weights and artifacts in object storage, use a message queue to absorb spikes in task distribution, run inference and builds on a container platform with elastic scaling, funnel call chains and execution logs into a unified logging service, track quality metrics (success rate, average latency, intervention rounds, defect distribution) continuously through monitoring and alerting, and route credentials through a unified key management capability. Only this way do evaluation conclusions settle into long-term observable production data instead of a one-off experiment record.

    8. Closing thoughts

    Evaluating long-horizon coding agents ultimately answers one question: which stages have had their marginal cost driven down far enough to hand over, and which stages still cost human effort? The answer so far is fairly consistent — the cost of implementation and integration has dropped substantially, while the cost of security and acceptance has not dropped at all, and is even easier to underestimate behind the illusion of all-green tests. Using long-horizon tasks as the evaluation baseline, covering legacy modification and from-scratch builds with two complementary tracks, and turning security constraints into machine-checkable gates is currently the highest-value approach.