Engineering Long-Horizon Coding Agents in Real Software Projects

    Evaluating generative coding systems for software engineering by single-problem pass rates or public leaderboards is unreliable. This article explains how to assess and operationalize long-horizon coding agents in real projects: keeping context, decomposing goals, changing multiple modules, verifying with tests, and delivering runnable, testable, reviewable work—especially on legacy-system changes and greenfield applications.

    Evaluating generative coding systems for software engineering by single-problem pass rates or public leaderboard rank is one of the least reliable methods. Engineering value is determined by whether a system can keep context across a long task, decompose goals, modify multiple modules, run verification, fix regressions, and drive the work to a runnable, testable, and reviewable state.

    That standard is simple, but it shifts evaluation from “can write code” to “can complete a delivery chain.” Code generation covers only one step. Long tasks also require understanding constraints of an existing system, honoring interface contracts, maintaining task state, identifying dependency impact, writing or repairing tests, and continuing to diagnose after failure. Progress in long-horizon coding agents is therefore worth attention, but they remain clearly short of “all tests green means safe to ship.”

    This article does not discuss specific products or model versions. It focuses on engineering evaluation and practice in real projects. One task class is modernization of legacy systems with substantial historical code. The other is building an end-to-end interactive learning application from requirement materials. The first stresses context understanding and enforcement of safety constraints. The second stresses completeness from analysis of source material through a shippable product.

    1. Shift evaluation from “correct snippets” to “reliable task closure”

    Short tasks are relatively easy to score: given a function signature, inputs, outputs, and edge cases, passing tests is enough. Long tasks are different. They typically span data models, interface contracts, task state, background processing, frontend interaction, and rendering. A miss in any layer can amplify later.

    Long-horizon coding ability should cover at least four layers:

    1. Context understanding: Read existing directory structure, shared components, historical conventions, exception handling, and test style—rather than inventing a parallel implementation.
    2. Cross-module progress: Identify contracts, tables, state machines, APIs, background jobs, and presentation layers affected by a change, and keep semantics consistent across layers.
    3. Verification and repair: Run tests, read failures, locate regressions, and add unit, integration, and end-to-end coverage as needed.
    4. Engineering convergence: Control change scope and complete configuration, migrations, logging, monitoring, documentation, and rollback—not only a demo that happens to run.

    Metrics should therefore move from “does the generated code look decent” to “did the task close.” Record whether the first implementation ran, how often humans intervened, regression count, severity of security defects, test effectiveness, maintainability of the diff, rollback cost, and final acceptance. Distinguish model capability success from systems-engineering success. The same model can behave very differently under different toolchains, context organization, permission boundaries, and test feedback.

    2. Where long-horizon ability comes from—and where it stops

    Industry practice suggests gains are not only from swapping a base model. Post-training that enlarges long-task environments, diversifies environment types, extends training time, and strengthens tool use, plan decomposition, test feedback, and error recovery all matter. Systems trained and optimized in longer, more complex, more realistic engineering settings are more likely to keep working.

    The progress is real: agents can read large code ranges, understand module relationships, make cross-layer changes, and continue repairing after test failures. For repetitive refactoring, interface integration, scaffolding, test supplementation, and defect localization, they can already cut human cost.

    The boundary is equally clear. They are efficient executors, not accountable owners. They can extract patterns from code but may not infer unwritten internal security rules. They can produce implementations that look reasonable without knowing that a permission check must fail closed on every read path. They can generate tests that cover happy paths while missing privilege escalation, tenant isolation, races, and rollback failure.

    A typical lesson: adding safe user media import and rich-media scene references in a legacy system may run and pass ordinary tests, yet human acceptance still finds high-priority isolation bugs. For example, asset reads may check only login, not tenant ownership, asset ownership, and scene context together. Passing tests proves covered paths run; it does not prove the permission model is correct and cannot replace security review.

    3. Two complementary tasks: legacy change and greenfield delivery

    To judge engineering usefulness, design at least two clearly different task types.

    3.1 Legacy-system change

    In a project with substantial historical code, add “safe user media import and rich-media scene references.” On the surface this is upload, storage, and display. In practice it touches contracts, data models, task state, APIs, background workers, admin UI, and rendering.

    The point is not “can it upload,” but whether upload is safe, references are authorized, display is auditable, and publish is reversible. The agent must understand existing authorization, exception handling, migration norms, and deploy constraints. Bypassing existing abstractions with a parallel stack raises maintenance cost even if the feature works.

    3.2 Greenfield delivery

    Given domain learning materials and a product requirements document, complete a truly completable interactive knowledge-learning application. Scope includes content modeling, level structure, learning progress, persistence, scoring feedback, exception recovery, and basic observability.

    This tests completeness from material analysis to product. Agents often scaffold pages and flows quickly, then silently fill unspecified requirements: retry after failure, multi-device progress consistency, compatibility after content updates, and UI state when permission is denied. Humans must define acceptance criteria in advance; “it runs” is not “it ships.”

    4. Legacy change: write safety invariants into the task contract first

    For media import and scene references, the priority is not more code from the agent. Put non-negotiable safety invariants in the task statement, with executable tests.

    4.1 Input and media processing

    • Identify upload types; do not trust extension or request headers alone.
    • Cap file size, dimensions, count, rate, and storage quota.
    • Decode, validate, and re-encode images or media as needed; strip untrusted metadata.
    • Quarantine suspicious files until security checks complete, then promote to the usable zone.
    • Use random, unguessable object IDs to avoid path traversal and enumeration.
    • Separate original and display files; do not expose internal storage paths to clients.

    4.2 Authorization and tenant isolation

    Read APIs must check identity, tenant, resource ownership, operation type, and scene context together. Use fail-closed: any failed condition denies; do not default to visible. For unauthorized resources, returning “not found” uniformly reduces enumeration.

    Pseudocode:

    1. function getAsset(assetId, requester, sceneContext):
    2. asset = store.find(assetId)
    3. if asset is null:
    4. return notFound()
    5. if asset.ownerTenant != requester.tenantId:
    6. return notFound()
    7. if not policy.allow(requester, asset, "read", sceneContext):
    8. return forbidden()
    9. if not sceneContext.isActive():
    10. return forbidden()
    11. return storage.signRead(asset.storageKey, shortTtl)

    The point is check order and failure semantics: tenant mismatch must not leak existence; policy must include scene context; signed URLs should be short-lived and scoped.

    4.3 Scene references and render safety

    A reference is not merely storing an asset ID. On create, validate availability, type match, tenant consistency, and allowed scene state. On delete or takedown, define dangling-reference handling. The render layer must escape, sanitize, and isolate untrusted content against script injection, CSRF, and CSP bypass. Prefer controlled templates or allowlisted capabilities over executing arbitrary uploaded fragments.

    4.4 Background work and observability

    Import, transcode, security scan, and cleanup jobs need idempotency, retry limits, timeouts, and dead-letter handling. Audit every import, grant, reference, and denial with request ID, tenant, actor, resource ID, result, and failure reason. Alerts should cover abnormal failure rate, quarantine backlog, spikes in authorization denials, and quota anomalies.

    4.5 Tests and acceptance

    Beyond unit and end-to-end tests, add authorization-matrix tests across roles, tenants, scene states, ownership, and operations. Negative cases matter: forged IDs, cross-tenant reads, expired signed URLs, concurrent reference deletes, duplicate submits, and rollback failure. Security acceptance is an independent quality gate; do not skip it because functional tests are green.

    5. From requirements to a complete app: process quality drives outcome quality

    On a greenfield interactive learning app, agents start fast and cover a wide surface: pages, data, APIs, and deploy skeleton. The common failure is a complete demo path with missing real boundaries.

    Split work into milestones with human quality gates:

    1. Clarify requirements and domain model: Learning goals, roles, level structure, scoring, progress semantics, and non-goals. The agent should emit an assumption list, not silently invent requirements.
    2. End-to-end skeleton: Minimal loop: start learning, enter a level, submit an answer, get feedback, record progress. Completeness is not the goal; it must run.
    3. Content pipeline and validation: Structure, versioning, and validation so invalid items, wrong answers, or broken references never reach runtime.
    4. Progress and recovery: Reconnect, duplicate submit, progress rollback, compatibility after content upgrades, and error messages.
    5. Quality and observability: Key logs, metrics, alerts, accessibility, and basic performance checks.
    6. Delivery and runbook: Deploy notes, configuration, data initialization, rollback steps, and known limits.

    Acceptance is not only “can the learner finish.” Define completion, exception, and recovery scripts: finish once, continue after refresh, no double scoring on resubmit, reject illegal content, recover after service faults, keep old progress after upgrades. For learning products, humans still judge whether paths are reasonable, feedback is clear, and difficulty progresses—automation cannot fully replace that.

    6. Evaluation metrics and hard gates

    Use hard gates plus a weighted score. Privilege bypass, data leak, secret exposure, irreversible migration, or severe unauthorized writes should fail delivery outright; averages must not cancel them.

    Dimension Focus Notes
    Task completeness End-to-end runnable Not generation volume; whether real flows close
    Correctness Main and exception paths Edges, concurrency, failure recovery
    Regression control Existing behavior preserved Change scope, dependency impact, test effectiveness
    Security and permissions Least privilege and isolation Permission issues are hard gates
    Testability Valuable tests added Negative paths and safety invariants
    Maintainability Fits existing architecture and style Avoid a parallel abstraction
    Observability Logs, metrics, alerts usable Needed to debug real issues
    Rollback Reversible migration and release High-risk changes need a rollback plan
    Human intervention Count and location of fixes Directional vs. detail fixes
    Reproducibility Stable results on a fixed task Record context, commands, environment, acceptance cases

    Public leaderboards capture some base skill; they do not replace these metrics. Real-project value usually depends on whether the system keeps converging on long tasks, not whether a single generation looks impressive.

    7. Human–agent split: high-autonomy development, low-risk delivery

    A better collaboration model is neither full hands-off nor constant hovering. Put the agent in a high-autonomy development loop and keep high-risk decisions with humans.

    Work that fits agents includes reading legacy code and mapping dependencies, drafting implementations, adding routine tests, fixing well-specified defects, refactoring, documenting, drafting migration scripts, and locating test failures.

    Work that must stay human includes threat modeling, permission semantics, data classification and compliance, architecture tradeoffs, migration approval, production release, final acceptance, and high-risk security review.

    Use three checkpoints:

    • Before the task: Goals, scope, non-goals, safety invariants, interface contracts, acceptance cases, and forbidden actions.
    • During execution: Human approval for architecture, data-model, and permission-policy changes; the agent runs in an isolated environment with least-privilege credentials.
    • Before delivery: After automated quality gates, humans still do security review and business acceptance. Require an evidence chain: change summary, test results, log samples, rollback plan, and known limits.

    Reusable task brief:

    1. Goal:
    2. Scope:
    3. Non-goals:
    4. Existing constraints:
    5. Safety invariants:
    6. Interface and data contracts:
    7. Acceptance cases:
    8. Forbidden actions:
    9. Delivery evidence required:

    The template turns implicit experience into executable constraints. Agents handle complexity better than vague constraints. Permission, security, and final acceptance must not be vague.

    8. Conclusion: ready for real projects, not a substitute for delivery ownership

    Long-horizon coding agents have raised the ceiling of software-engineering assistance. They no longer only complete snippets; they can keep moving cross-module changes in complex context toward a runnable, testable state. For legacy change, feature-chain construction, test supplementation, and defect repair, they are worth validating on real projects, especially long coding tasks.

    “All tests green” is the start of delivery, not the end. Tenant isolation, data security, migration rollback, and business acceptance still need humans as the last line. A more robust engineering stance is high-autonomy development with low-risk delivery, scoring success on runnable, testable, auditable, and rollback-capable outcomes—not on leaderboard scores or demo polish.