Evaluating and Tuning LLMs for Long-Horizon Coding Tasks

    This article looks at how large language models perform on long-horizon coding work, where tasks span hundreds of millions of tokens, dozens of stages, and an evolving codebase. It explains why single-turn evaluations fail to predict long-horizon quality, then walks through five evaluation dimensions: requirement coverage, context management, long-horizon stability, responsiveness, and human intervention cost. It examines the most damaging failure mode, silent requirement loss caused by context compaction, and shows how an external task checklist, explicit checkpointing, and re-anchoring after each compaction can compensate for a model's memory decay. It also covers trade-offs between context window size and compaction frequency, how specification adherence fades over a long run, how to orchestrate multiple models with complementary strengths, and closes with a practical six-point tuning checklist.

    1. Introduction: When Coding Moves from a Single Conversation to a Long-Running Project

    Conventionally, we bring large language models into coding work for short-range interactions: “write this function,” “fix that bug.” But the real limits of a model only surface once the task grows large enough to require processing hundreds of millions of tokens continuously, moving through dozens of stages, and maintaining a codebase that keeps evolving. Long-horizon coding is not about whether a model can produce correct code in isolation; it is about whether it can avoid getting lost, avoid forgetting, and avoid drifting off course across an extended context.

    Many developers have seen the same thing. On isolated tasks, the leading models perform fairly similarly, and their final answers are all usable. But the moment work enters a long-horizon workflow, the ranking is reshuffled, and it can even invert. Clearly, conclusions drawn from single-point evaluations do not represent long-horizon capability. What we need is an evaluation perspective that sits much closer to real engineering work.

    2. Evaluation Dimensions: More Than “Was the Answer Correct?”

    To assess a model’s behavior in long-horizon coding scientifically, consider at least the following dimensions together:

    1. Requirement coverage — Does the model implement every item on the task list in full, or does it quietly skip part of the list?
    2. Context management — As the context approaches its limit, does compaction lose critical information or drop unfinished feature points?
    3. Long-horizon stability — Over a long run, does the model fall into a loop of its own making, and does its adherence to conventions gradually decay?
    4. Response speed and experience — Does it slow down noticeably under peak load, or does it agonize repeatedly over a single step and burn tokens for nothing?
    5. Human intervention cost — How much of the final output needs to be identified and fixed by a person? This is the practical anchor for “usability.”

    Requirement coverage and human intervention cost sit closest to engineering value, because they map directly to one question: can this code ship as-is?

    3. Core Pain Point: Silent Requirement Loss Caused by Context Compaction

    In long-horizon tasks, the most subtle and most damaging problem is requirements being silently dropped. Typical symptoms include:

    • When the model reaches an expensive feature to implement, such as a complex algorithmic module, it writes an empty function as a placeholder, skips over it, and never comes back to complete it;
    • Even under strict test-driven development (TDD) constraints, more than 20% of requirements can still be missed;
    • Context compaction amplifies the problem: once the conversation history is summarized, early-registered requirement items can be “diluted” by the summarization algorithm, leaving the model “forgetting” that tasks remain outstanding.

    The cause is that a model’s attention resources spread thinner as context grows, and compaction is inherently a lossy trade-off. To mitigate it, engineering practice can introduce an explicit requirement list plus a checkpoint mechanism, so that “degree of completion” becomes an external state that can be re-verified repeatedly, rather than depending on the model’s memory.

    1. # tasks.md -- make requirements a verifiable external state
    2. - [x] Define core data structures
    3. - [x] Implement the data persistence interface
    4. - [ ] Implement the rate-limiting module under high concurrency # tag expensive tasks separately
    5. - [ ] Fill in edge-case tests for the rate-limiting module
    6. # rules.md -- make conventions continuously referencable
    7. 1. Every function must have its test written before its implementation (TDD).
    8. 2. Never paper over an [ ] unfinished item with an empty implementation.
    9. 3. At the end of each stage, update tasks.md and self-assess coverage.

    The key move is to relocate to-dos, progress, and conventions out of the model’s memory and into external files, so that after every context compaction the goal can be re-anchored.

    4. Context Strategy: Balancing Compaction Frequency Against Window Size

    Models do not agree on when to compact context. Some habitually trigger compaction once the conversation reaches roughly 75%–80% of capacity; a long chain of thought then burns tokens quickly, driving even more frequent compaction and producing a negative feedback loop: the more it thinks, the more it compacts, and the more it compacts, the more it forgets.

    A few lessons from practice are worth borrowing:

    • Enlarging the context window appropriately reduces the number of compactions significantly and therefore relieves requirement loss. This is also why running long-horizon tasks in a larger window visibly improves quality.
    • Keep the chain of thought for each step under control. Avoid opening an excessively long self-debate over a small problem; it saves tokens and lowers compaction frequency at the same time.
    • Advance in segments rather than in one continuous burst: at the end of each stage, proactively write a structured summary and persist a checkpoint.
    1. # Pseudocode: an explicit checkpoint mechanism
    2. def run_stage(task, context, checkpoint):
    3. context = load(checkpoint) + task # re-anchor the goal after compaction
    4. result = model.generate(context)
    5. coverage = verify(task.requirements, result) # check requirement coverage
    6. save(checkpoint, context, result, coverage) # persist so nothing is forgotten
    7. if coverage < threshold:
    8. return run_stage(retry_missing(task), context, checkpoint)
    9. return result

    This loop of reload, verify coverage, and persist is fundamentally an external engineering substitute for the model’s memory decay over long contexts.

    5. Prompt Engineering and Specification Constraints: From “Adding Rules” to “Keeping Rules”

    One easily overlooked behavior: even when detailed conventions are written down, a model’s adherence to them decays as the task progresses. A rule does not stay in force just because it was stated once; it needs to be restated at key points. Recommended practices:

    • Place core constraints at a fixed position in every prompt round (such as the beginning or the end) to raise the probability they are followed;
    • Express constraints in structured formats (checklists, tables, JSON schema) rather than natural language, since they are parsed more reliably;
    • Add a verification step for critical outputs, for example requiring the model to self-certify before submission that the requirement list is fully covered.

    6. Architectural Coordination: Combining a Single Model’s Capability into a Team

    Rather than relying on one model being good at everything, divide the work according to strengths and have them collaborate:

    • A more objective model with sound reasoning produces the overall plan and skeleton;
    • A more detail-oriented model that is good at spotting gaps handles review and gap-filling;
    • Watch out for another tendency, over-focusing on details. It will keep circling back over trivia and slow the whole pace down, so the prompt needs to draw explicit boundaries around what it should attend to.

    From an engineering standpoint, good post-training lets a model reach requirement coverage above 90%, and very little of its output then needs manual repair. This shows that model selection cannot be based on parameter count alone; it must weigh the model’s completion rate and its willingness to follow instructions on real long-horizon tasks. Orchestrating several models into the same pipeline and converging the result through voting or cross-verification is usually more robust than betting on a single model.

    7. An Actionable Tuning Checklist

    1. Build an external requirement list, forbid empty implementations, and verify coverage at the end of every stage;
    2. Enlarge the context window appropriately to lower compaction frequency and reduce forgetting;
    3. Keep per-step chains of thought short to avoid pointless self-debate burning tokens;
    4. Restate key conventions at a fixed position in every round, expressed in structured formats;
    5. Introduce checkpoint persistence so long-horizon tasks become interruptible, resumable, and auditable;
    6. Divide work across multiple models: an objective model for the plan plus a detail-oriented model for review, converging through cross-verification.

    8. Closing Thoughts

    Long-horizon coding capability is not measured by whether a model can get one piece of code right, but by whether it holds on to the goal across a vast context and a long process. The real gap usually lies not in how clever a model is at any single question, but in requirement coverage and human intervention cost. Understanding the trade-offs of context compaction, using external mechanisms to compensate for memory decay, and constraining model behavior with engineering means are what actually convert a model’s potential into results. No matter how large that potential is, it stays unrealized without tuning; meanwhile, a rigorous context management strategy and coordinated architecture often decide the outcome more than switching models does.