Agent Harness Effects on Coding Model Performance: Why the Same Model Scores Differently Across Scaffolds

    A controlled experiment shows that a flagship coding model, with weights and version held fixed, can move from a mediocre score band to near-perfect results simply by swapping its agent harness. This article explains what an agent harness (scaffold) is, what a minimal orchestration setup actually changes, and why the effect happens: modern coding models are post-trained inside a specific agentic orchestration environment, so a mismatched harness at inference time causes distribution shift rather than a real gain in model intelligence. It also gives practical signals for detecting this hidden coupling and concrete best practices for evaluation and deployment.

    1. A Counterintuitive Controlled Experiment

    Picture a controlled experiment: hold one fixed set of real engineering maintenance tasks constant, and change only the agent harness on the outside that drives the model. The model weights and version stay exactly the same.

    Under a fully featured set of coding tools and a standard orchestration mode, a flagship coding model lands consistently in a middling-to-above-average band. Switch to a near-bare-bones minimal orchestration, and the same tasks come back twice in a row with near-perfect scores.

    This is not random noise. It is a visible tier change — from “didn’t seem to show up” to “fully switched on.” More intriguing still, the minimal-mode results line up with the model’s internal grey-test numbers from before its official release. When the model, the tasks, and the evaluation criteria are all locked down, the only remaining variable is the execution environment the model runs in.

    That points to a core issue that is easy to overlook: a model’s demonstrated capability is not determined by its weights alone. It depends heavily on the “exam hall” it runs in.

    2. What an Agent Harness (Scaffold) Actually Is

    A concept needs clarifying first. The “intelligent coding assistant” people use today is essentially a large model plus a layer of orchestration logic. That orchestration layer is the harness. It typically includes:

    • Tool definitions: which actions the model can invoke (reading and writing files, executing commands, code retrieval, network access, and so on)
    • System prompt: constraints on the model’s role, behavior rules, and output style
    • Reasoning and loop control: whether multiple rounds of explicit planning are allowed, how context is assembled, how tool results are fed back in, and when to stop
    • Context management strategy: how history is trimmed, how it is summarized, and how project structure is injected

    The model itself is only a function that predicts the next step from its context. The harness decides what that context looks like, which actions are available, and in what style the output is produced. In other words, putting the same model into a different harness is like sending the same candidate into a different exam hall, with a different answer sheet and different stationery.

    3. What the Minimal Mode Actually Changed

    Back to the experiment. The minimal mode does very little, and does it very deliberately:

    • Only two tools are kept — command execution (Bash) and file editing. Everything else is switched off.
    • The system prompt is compressed to almost a single sentence, for example keeping only “You are a helpful software engineer assistant.”
    • The reasoning mode changes from multi-round explicit planning to direct execution.

    The difference between the two configurations can be sketched like this:

    1. # Standard harness (illustrative, not real product syntax)
    2. system_prompt: "You are a fully featured software engineering assistant. Plan
    3. thoroughly first, then explain step by step, and make
    4. reasonable use of all available tools..."
    5. tools: [file read, file write, file edit, command execution, code search, network access, ...]
    6. reasoning_mode: multi-round explicit planning
    7. # Minimal harness (illustrative)
    8. system_prompt: "You are a helpful software engineer assistant"
    9. tools: [command execution, file edit]
    10. reasoning_mode: direct execution

    Intuitively, fewer tools and a shorter prompt should produce a weaker result. The outcome was the opposite. This suggests the model’s latent capability on these kinds of tasks was there all along — it was simply disrupted by the standard harness’s layered wrapping, verbose instructions, and excessive planning, and never released smoothly.

    An easily overlooked piece of supporting evidence is the change in language style: in standard mode, the model frequently uses phrasing of the “let me first take a look at…” variety; after switching to minimal mode, that kind of expression almost disappears, replaced by a more direct collaborative tone. This wording difference was initially misread as “the vendor quietly swapped the model,” but a more reasonable explanation is that the same set of weights presents two different reasoning modes under two different orchestrations.

    4. Root Cause: Post-Training Is Deeply Coupled to the Harness

    To understand this phenomenon, you have to go back to the training pipeline of large models. Contemporary coding models usually go through a round of reinforcement learning and post-training aimed at agentic behavior after pretraining. In that stage, the model does not learn “writing code” in isolation. It learns, inside a specific agentic orchestration environment, how to use tools, how to interact with the environment, and how to complete tasks.

    This means the model forms a strong conditional dependence on a particular harness: it internalizes the combination of “tool set + system prompt + interaction rhythm” as part of its input distribution. At inference time, once you switch to a harness that does not match, you get the classic distribution shift — the input distribution during training differs from the input distribution during inference, so performance naturally drops.

    In one sentence: this is not the model “getting smarter,” it is the model “recognizing its environment.” It was trained to score well in a specific exam hall; move it out of that familiar hall and its answering ability is discounted. The reason minimal mode stands out is precisely that it closely matches the agentic orchestration used during post-training — it went back to its home field.

    5. How to Spot This “Exam Hall Effect”

    For engineering teams and evaluators, a few practical signals reveal this kind of hidden binding:

    1. A step-change jump in scores before and after swapping the harness. Not gradual, small fluctuations, but a tier change — and consistent across different task sets, which indicates the environment rather than task difficulty is doing the work.
    2. Official grey-test or internal-test numbers that match one specific configuration. If a publicly claimed high score can only be reproduced under the same harness, be wary that the score is bound to familiarity with the training environment.
    3. Systematic differences in behavioral style. Wording, planning rhythm, and tool-call preferences that shift wholesale along with the harness are a direct external manifestation of distribution shift.
    4. Documentation or changelogs that expose the configuration. If the model’s API documentation explicitly states that the code tasks in the public benchmark use a “minimal” or “official” orchestration, and the framework repository contains changes aligning the minimal agent with the agentic orchestration used during reinforcement learning training, that essentially confirms the diagnosis.

    6. Takeaways and Best Practices for Developers

    This finding has direct implications for both users and evaluators:

    • Always publish the harness alongside the score. Any model capability number should be reported together with the tool set, system prompt, and reasoning mode. Talking about scores without the harness is like talking about speed without the track — there is no comparability.
    • Be careful with cross-harness comparisons. Differences between orchestrations may reflect only how well the exam environment matches, not the model’s real capability. Try to align the harness when running head-to-head comparisons.
    • Move closer to the training configuration when deploying. If you want the model to perform at its best, follow the vendor’s recommended orchestration so that the tool set and interaction mode at inference time stay close to the post-training environment and unnecessary “capability loss” is avoided.
    • Separate capability from familiarity. When one configuration clearly lifts performance, first ask whether it improved reasoning quality or merely returned the model to a familiar input distribution. The two have completely different implications for selection.
    • Build your own task-level baseline. On your team’s real codebase, compare harnesses head-to-head using a fixed task set to get a reproducible baseline that fits your own scenario. That is more reliable than reading public leaderboards alone.

    7. Conclusion

    The claim that “just changing the shell can substantially raise a model’s level” is both right and wrong. The right part: a well-designed harness really can release a large share of a model’s potential. The part that needs correcting: this improvement is not a growth in the model’s intelligence, but a repair of distribution shift caused by realigning the post-training orchestration with the inference environment.

    Understanding this helps us avoid two mistakes. One is blindly chasing some “magic configuration” and mistaking it for a universal silver bullet. The other is being misled by a mismatched harness during selection and evaluation, underestimating or overestimating a model. For engineering practice, the real work is to treat the harness as part of the model’s capability — to design it, document it, and evaluate it that way, so that it works with the model’s post-training shape instead of against intuition.