Long-Horizon Code Agents: Keeping LLMs on Track During Long-Running Coding Tasks

    This article examines how to build long-horizon code agents that stay on track during multi-hour, multi-file engineering tasks. It contrasts short-horizon benchmark work with real engineering, catalogs four common failure patterns (context drift, plan collapse, local-optimum traps, and missing verification), and proposes a set of long-horizon metrics. It then covers actively managing context in layers with never-compressed constraints, combining symbol-level with semantic retrieval, verifiable gates with checkpoints and structured feedback, tool-call design principles, a four-step profiling loop for performance tuning, strategies for obscure tech stacks, cost-control levers, general cloud deployment patterns, and an implementation checklist.

    1. Define the Problem First: “Writing Code” and “Delivering” Are Not the Same Thing

    There’s a long-standing mismatch in how the industry evaluates code LLMs: leaderboard scores are high, demos look impressive, but when you drop the model into a real enterprise repository to run a long task, the failure rate jumps sharply. This isn’t a problem with a single model’s capability—it’s that the task shape has changed. Typical tasks in benchmark sets are short-horizon: a single file, a single function, clear acceptance criteria, convergence within a few turns. Real engineering tasks are long-horizon: spanning dozens of files, with deep dependency chains, ambiguous acceptance criteria, and repeated trial and error along the way.

    Failures on long-horizon tasks typically fall into four patterns, each worth addressing directly:

    1. Context drift. As turns accumulate, early hard constraints (interface signatures that can’t change, a library version locked down, a performance baseline that can’t regress) get diluted in the context. The model starts pushing forward in whatever way “looks most reasonable now,” and the final deliverable violates the original constraints.
    2. Plan collapse. A long plan generated in one shot lacks feedback anchors. Tiny deviations in the first few steps get amplified in subsequent steps, and things drift further and further until the output can’t compile or pass tests at all.
    3. Local-optimum trap. The model tends to change whatever is easiest to change, while the real bottleneck often sits at the architecture or scheduling layer. The symptom: “many lines changed, throughput didn’t move at all.”
    4. Missing verification. Without executable acceptance criteria, the model can’t self-correct and can only deliver something that “looks right.” This is the most fatal pattern, because it produces no failure signal.

    So when we measure a code agent, what matters more is not “can it write this one function,” but a set of long-horizon metrics:

    • Average steps and tool calls per task;
    • Number of mid-task rollbacks and the recovery success rate after rollback;
    • The proportion of tasks that still satisfy the original constraints after long runs;
    • Resource and cost overhead per task;
    • Whether it can resume from a checkpoint after failure, rather than starting over.

    The rest of this discussion revolves around this set of metrics.

    2. Effective Context: Window Length ≠ Usable Context

    A common misconception is that “once the context window is big enough, the problem is solved.” In practice, the industry has widely observed that as context grows longer, recall of details in the middle section drops noticeably—and the constraints that matter most in engineering tasks (like an interface that can’t be touched) are often written near the front. The evaluation method is simple: place key information at different depths in the context and measure the proportion correctly cited, giving you an “effective context” curve.

    A more reliable engineering practice is to treat context as an actively managed resource, organized in layers:

    Layer Content Update policy
    Current step Target files to change this step, failure errors, pending tool calls Refreshed every step
    Task state Hard-constraint list, acceptance criteria, done/todo, excluded approaches Append-only; no compression allowed
    Repo state Module map, symbol index, dependencies, key interface definitions Retrieved and injected on demand
    History state Full conversation and raw tool outputs Rolling summary + archive; not in current context

    Two practical points:

    First, constraints must live in a separate block and never be compressed. Summarization algorithms drop modifiers in order to compress, and a constraint like “do not change the public method signatures of OrderBook“ becomes a disaster once its qualifiers are lost. Putting hard constraints in a dedicated block, preserved verbatim and re-injected every step, is the lowest-cost defense against drift.

    Second, retrieval should follow code structure, not just semantics. Pure vector retrieval tends to recall fragments that are “textually similar” but “structurally irrelevant.” A more robust combination is symbol-level indexing (functions, classes, interfaces, config items) plus semantic retrieval: once you locate a target function, expand along the call graph and dependency graph to pull in its callers, callees, and interface definitions, filling in “who will be affected by this change” in one pass.

    1. # Context assembly sketch (pseudocode)
    2. task_state = load_state(task_id) # hard constraints, acceptance criteria, done items, excluded approaches
    3. budget = context_budget()
    4. block_hard = task_state.hard_constraints # never compressed, always injected
    5. block_repo = symbol_index.lookup(target) # symbol-level recall
    6. block_repo += dependency_graph.expand(target, depth=2)
    7. block_sem = vector_search(query, k=8) # semantic supplement
    8. assembled = [block_hard,
    9. render(repo_map(block_repo)),
    10. render(recent_errors(limit=3)),
    11. summarize(history, budget - cost(above))] # only history state is compressed

    3. The Source of Stability: Every Intermediate State Must Be Verifiable

    For long tasks to stay on track, the core isn’t making the model “smarter”—it’s ensuring that every small step has an objective right/wrong signal. That requires writing acceptance criteria as something executable before the task begins, not a one-sentence description.

    The recommended gate order is: compile/type check → unit tests → relevant integration tests → static and security scanning → performance baseline regression (where applicable). Four design points are key:

    • Feedback must be structured. Don’t paste entire build logs back into the context—that’s the fastest way to pollute it. Instead, parse them into structured records like “file, line number, error category, minimal reproduction command,” and only send back the first N.
    • Small commits + checkpoints. Every time a gate passes, drop a rollback-able checkpoint and persist the task state. This way, even if a long task is interrupted, it can resume from the most recent successful state rather than losing hours of progress.
    • Distinguish retryable from non-retryable failures. Network jitter and temporary resource shortages can be retried; compile errors and assertion failures are signals that the code needs to change—retrying only burns money.
    • Serial writes, parallel reads. Concurrent writes to the same workspace easily overwrite each other; reads and retrievals can be parallelized freely.
    1. # Long-horizon task main loop sketch (pseudocode)
    2. state = load_or_init(task_id)
    3. while not state.done and state.steps < state.step_limit:
    4. obs = assemble_context(state) # see previous section
    5. act = model.plan(state, obs) # outputs: structured actions, not free text
    6. if act.kind == "edit":
    7. workspace.checkpoint() # rollback-able
    8. apply_patch(act.patch)
    9. elif act.kind == "tool":
    10. result = tools.call(act.tool, act.args, timeout=act.timeout)
    11. gate = run_gates(scope=act.affected_scope) # compile/test/scan
    12. if gate.passed:
    13. state.commit(act, evidence=gate.evidence) # record evidence chain, incl. passing tests
    14. else:
    15. workspace.rollback()
    16. state.record_failure(structured(gate.errors, limit=5)) # structured, capped
    17. state.exclude(act) # excluded approach, avoid hitting the same wall repeatedly
    18. persist(state) # resume-from-checkpoint

    The state.exclude action is often overlooked but very valuable: it turns “this path was tried and didn’t work” into persistent knowledge. Without it, the model easily oscillates between two wrong approaches, burning through the budget.

    4. Tool-Call Design Quality Determines the Ceiling of Long-Horizon Capability

    Long autonomous iteration is essentially the system behavior of model + tool environment. If the tools are poorly designed, even a strong model degrades into blind guessing. A few repeatedly validated principles:

    • Few tools, semantically orthogonal. The more tools there are, the larger the selection space and the higher the chance of a wrong pick. Better to build one semantically clear “read/write code” tool with complete parameters than to split into a pile of fine-grained actions.
    • Return bodies must have a budget. Every tool call needs an output cap; the overflow should give a summary plus a handle for continued querying (like a result set ID), so the model can page on demand rather than dumping megabytes of logs into the context at once.
    • Idempotent. Repeating the same action shouldn’t produce a difference in side effects; otherwise auto-retry becomes auto-destruction.
    • Least privilege + sandbox isolation. Code execution, dependency installation, and network access belong in an isolated environment; credentials are issued briefly on demand; and destructive operations get a second confirmation. This layer is the security baseline and can’t be replaced by “the model will judge.”
    • Make long-running tasks asynchronous. Compilation, stress tests, and long builds shouldn’t block the main loop; submit, get a task ID, poll or receive a callback for results, and let the main loop keep advancing other steps.

    5. Performance Optimization Tasks: From “Changing Code” to “Finding Bottlenecks”

    Refactoring and tuning enterprise-grade engines is among the hardest long-horizon tasks, because it simultaneously requires reading legacy code, understanding runtime behavior, and quantifying gains. A reusable methodology is a four-step loop:

    1. Profile first, refuse to guess. Start with sampling-based profiling to get structured data: hot functions and their self/cumulative time share, lock wait time, memory allocation and GC frequency, cache miss rate. Flame graphs are a great visualization, but for an agent, what’s fed to the model should be a structured Top-N hotspot table extracted from the flame graph or profile file, not an image or a raw sampling file.

    2. Make hypotheses explicit. Before each optimization, write a falsifiable hypothesis: “The bottleneck is X, because X accounts for Y% of total time; I expect it to drop to Z% after the change.” If you can’t write the numbers, it’s not yet time to act.

    3. Minimal change + same-baseline regression. Change one variable at a time; re-test with a fixed input set, fixed warm-up duration, and fixed concurrency; compare percentile latency (p50/p95/p99) and throughput, not just the average.

    4. Record conclusions, including failures. Optimization history is itself the most valuable knowledge asset: which parameter combinations were tried, how much they gained, why they were rolled back. Without this record, a long task repeatedly burns steps in the same dead end.

    Concurrency orchestration and thread topology are high-frequency optimization points in this class of tasks and belong to general engineering, so they’re worth expanding. Take a common multi-stage processing pipeline (say, market-data parsing + matching + settlement) as an example. Adjusting the thread allocation from “independent pool per stage, symmetric counts” to “proportional to actual load” often yields more than shaving code at a single point:

    1. # Thread/executor topology comparison (generic sketch; parameter names per actual implementation)
    2. # Before: symmetric thread counts across stages
    3. stage.parse : threads = 4, queue = unbounded
    4. stage.match : threads = 4, queue = unbounded
    5. stage.settle : threads = 2, queue = unbounded
    6. affinity : none
    7. # After: allocate by measured load, pin cores, bounded queues + backpressure
    8. stage.parse : threads = 2, queue = bounded(4096)
    9. stage.match : threads = 1, queue = bounded(2048)
    10. stage.settle : threads = 1, queue = bounded(2048)
    11. affinity : pin_to_cores, numa_local

    The general elements to watch alongside this include: making queues bounded to create backpressure, avoiding unbounded queues that eat memory and hide latency inside the queue; reducing shared mutable state across threads to lower lock contention; watching for cache-line false sharing and NUMA affinity; and the tradeoff between batch granularity and single-item latency. This kind of adjustment often brings order-of-magnitude throughput improvement while the code change might be only a few dozen lines—this is the classic “local change, global gain” scenario, and exactly the kind of optimization that’s hardest to find by manually reading code line by line.

    One point of methodological honesty needs emphasis: any gain figure must be measured on the same machine, the same input set, under the same warm-up conditions, and must be able to rule out noise (fix CPU frequency, close interfering processes, repeat multiple times and take the median). Multiplier relationships compared across environments have no engineering meaning and will instead skew later decisions.

    6. Obscure Tech Stacks: Three Remedies When the Corpus Is Sparse

    When a task involves a language, framework, or private DSL that rarely appears in training corpora, the model’s priors are nearly useless. These tasks happen to be good test cases for long-horizon capability, and there are three remedies:

    1. Spec-driven: turn language details into executable assertions. Don’t count on the model to “remember” the syntax of an obscure language; instead, write the semantic requirements as test cases. Pass/fail is a hard signal, more reliable than any syntax explanation.

    2. Differential testing against a reference implementation. If a mature implementation exists in another language, use it as the baseline for output comparison (golden test): given the same inputs, the new implementation’s output must match the reference bit for bit. This turns “is the implementation correct” from a subjective judgment into an objective check, and is the most effective guardrail for porting tasks on obscure stacks.

    3. Use compile-time feedback as high-density supervision. Some systems programming languages are designed around “explicit, zero hidden behavior”: no hidden allocations, no implicit copies, error handling must be written explicitly, memory layout is fully controllable. These traits are actually favorable for automated iteration—the compiler points out missing allocation frees, unhandled error paths, and type mismatches, forming dense and cheap correction signals. The prerequisite is that the tool environment must structurally return the compiler’s diagnostics (file, line, error code, suggestion) rather than pasting the whole thing.

    Take a general engineering sequence as an example: implementing local inference for an open-weight model from scratch, a reasonable breakdown is: memory layout and ownership model → tensor and shape system → basic operator kernels → quantization path → scheduling and memory pool → end-to-end numerical consistency check → performance tuning. Note that the numerical consistency check must come before performance tuning: correctness first, speed second. Reverse the order, and locating precision problems later becomes extremely painful.

    7. Cost Control: Where the Money Goes in Long Tasks

    The cost structure of long tasks is completely different from short ones, worth a separate calculation:

    1. single-task cost ≈ avg steps × per-step (input tokens + output tokens) × unit price + retry cost

    The two items most likely to spiral out of control aren’t the model’s unit price, but step count and input tokens. And the biggest source of input tokens is often not the conversation history, but tool return bodies—dumping an entire build log or the whole file tree back into the context in a single step can multiply that step’s cost several times over. Control measures, in order of cost-effectiveness:

    1. Cap tool return body size, output a summary + handle, page on demand;
    2. Compress error info structurally, returning only the first N most informative entries;
    3. Use cheaper models for routing, summarization, and retrieval rewriting; use costlier models only for key decisions;
    4. Reuse existing artifacts: persist and archive compile results, test reports, and profiling data, retrieve and reuse rather than re-run;
    5. Set budget circuit breakers: dual caps on both step count and cost per task, and on breach, output the current best checkpoint and hand off to a human;
    6. Enable prompt prefix caching, keeping stable system prompts and the repo map in fixed positions.

    There’s also a clear dividing line between self-hosting and managed services: scenarios with high data-compliance requirements, stable load, and a need to control the model iteration cadence lean toward self-hosting; scenarios with large peak fluctuations, a need for rapid validation, and tight ops headcount lean toward managed services. The real value of open weights in this tradeoff is controllability and reproducibility—you can pin a version and run long-term regression on the same pipeline, without worrying that a silent backend model upgrade will cause behavior drift. The cost is that evaluation, tuning, and ops responsibility all fall on you, so before self-hosting you must first have your own evaluation set, otherwise “is it good enough” becomes an unquantifiable account.

    8. General Patterns for Cloud Deployment

    Whether self-hosting or managed, productionizing a long-horizon agent typically requires several general capabilities working together. Here I only discuss capability categories and responsibility boundaries; specific specs, interfaces, and pricing should follow the official documentation:

    • Model serving: hosts inference, provides elastic scaling and multi-version management to support canary release and rollback;
    • Container platform: runs the agent main loop and isolated sandboxes, keeping code execution and dependency installation within a controlled boundary;
    • Object storage: holds code artifacts, build caches, profiling files, and task checkpoints to support resume-from-checkpoint;
    • Message queue: handles async decoupling of long tasks, with compile and stress-test results flowing back via callback or subscription;
    • Logging and monitoring/alerting: correlates the full chain by task ID, monitoring success rate, rollback rate, average steps, and per-task cost;
    • Key management: centralized credential hosting and short-lived issuance, preventing keys from entering the context or logs.

    When implementing this kind of solution on Baidu AI Cloud, a fairly natural layering is: model capabilities accessed via model serving, the agent runtime and sandbox running on the container platform, artifacts and checkpoints in object storage, long-running subtasks made asynchronous via the message queue, and observability handled by the logging service and monitoring/alerting. The purpose of the layering is clear—to make “agent behavior reproducible, task progress recoverable, and cost changes attributable”—and these three things are the real threshold for whether a long-horizon task can go into production.

    9. Implementation Checklist

    Before introducing a long-horizon code agent into a real engineering environment, it’s worth confirming each item:

    • Is there an evaluation set for your own repository, rather than just looking at public leaderboards?
    • Are hard constraints stored separately, preserved verbatim, and re-injected every step?
    • Does every step’s change have an executable gate, with automatic rollback on failure?
    • Do tool return bodies have a size budget and pagination handles?
    • Is task state persisted to support resume-from-checkpoint?
    • Are retryable and non-retryable failures distinguished, to avoid wasted retries burning the budget?
    • Are dual circuit breakers on step count and cost set?
    • Is code execution in an isolated environment, with short-lived, least-privilege credentials?
    • Is there a measurement standard for performance optimization: same environment, same input set, multiple repetitions?
    • Is optimization history (including failed conclusions) retained structurally and reused?

    10. Conclusion

    “Open-source, cheap, and reliably long-horizon” carry weight together because they correspond to three real pain points: controllability, economy, and deliverability. But to be clear, none of these three is delivered all at once by any single model version; they are supported collectively by engineering elements—context management, verifiable gates, tool design, the profiling loop, and cost governance—as a system capability.

    The most practical way to judge whether a code agent is usable isn’t to see whether it can write an elegant function, but to give it a real task that spans a dozen-plus files, runs for hours, and has clear acceptance criteria, then observe: will it drift? If it drifts, can it detect it itself? If it detects it, can it recover? And is the final spend within budget? Only when all five questions get a “yes” has it truly made the leap from “can write code” to “delivers reliably.”