Why the Execution Framework Decides Coding-Agent Benchmarks

    Coding agents should be evaluated as a model paired with an execution framework, not as isolated weights. This article explains how context management, stop conditions, and token or step budgets change pass rates, why framework rankings often fail to transfer across models, and how to make evaluations comparable while designing agent workflows.

    Coding agents are no longer a matter of handing a problem to a model and waiting for a block of code. The repository must be opened, symbols located, files read, patches applied, tests run, and failure messages fed back before anyone can confirm that the diff actually fixes the issue. If those actions are not connected, even a strong model remains a static reasoner. The fairer object of evaluation is therefore the combination of model and execution framework, not an isolated weight file.

    Public technical discussion keeps showing the same pattern: changing only the outer task orchestration, not the model, can nearly double the one-shot pass rate on code-repair benchmarks. Ranking several frameworks on two different models can produce rank correlation so low that the orderings look almost unrelated. An orchestration that works best for model A may fall to the back of the pack on model B. That matters for how coding leaderboards are read. The numbers usually measure a system under one wiring, not the model’s inherent ceiling.

    Taking the agent apart

    A coding agent can be written as:

    coding agent = model + execution framework

    The model judges within a given context: what an error means, which function to change, and whether the next turn should keep searching. The execution framework is the eyes, hands, workbench, memory, and project manager. It decides what the model sees, what it can call, when it stops, and where the budget goes.

    A typical repair is not a single completion. It is a controlled loop:

    1. Receive the issue description and repository constraints.
    2. Search or browse relevant paths and read file excerpts.
    3. Form a change hypothesis and land a patch.
    4. Run the agreed checks or tests.
    5. Write a failure summary back into the context.
    6. Within the budget and stop conditions, decide whether to continue or finish.
    7. Emit the diff and check it against the definition of done.

    The model usually cannot “open the repository” by itself. The execution framework is what turns those steps into a state machine. It is not a UI shell. It is an amplifier or a limiter on how much of the model’s intelligence can be used.

    Three control surfaces that are easy to miss

    Context management decides which history is still sent to the model. Replaying everything crowds the window and buries the key error. Cutting too hard drops paths already tried and causes repeated exploration. Common tactics include storing messages by role, summarizing tool returns, keeping only the latest failure and the current hypothesis, and replacing long files with line-numbered excerpts. Different policies change whether the model “remembers what it already failed at.”

    Stop conditions decide when the task is declared done. Some frameworks require all tests to pass. Some accept the model’s own claim of completion. Some also require a non-empty diff that passes static checks. Stopping too early counts unfinished work as a pass. Stopping too late spends budget on loops that no longer add information. If an evaluation does not publish its stop definition, the numbers are not comparable.

    Token and step budgets decide how much room the model has to explore. Under a tight budget, a strong model may never reach the module that actually matters. Under a loose budget, a weakly constrained loop can idle in unrelated directories. A larger budget is not automatically better. It has to match task granularity and retrieval quality.

    Stacked together, these three surfaces can make the same model look entirely different as an author of engineering code.

    Tokens are spent mostly on exploration, not the final patch

    Many code-repair experiments suggest that the bulk of total consumption is not the last patch write. It is reading, locating, and confirming back and forth. In some experiments that pair a general execution loop with the model, read-style operations can account for roughly 60 to 80 percent of total tokens. How the model obtains context is itself part of capability.

    If retrieval can only scan blindly by file name, the model spends many turns buying information. If the framework can first supply symbol references, the failing test stack, and candidate files, the model can spend reasoning on causal judgment. Optimization should not only chase “a slightly nicer write.” It should also ask:

    • Whether each tool return is short, accurate, and traceable
    • Whether failures are structured instead of raw logs poured back in
    • Whether visited paths are deduplicated so the same file is not reread
    • Whether the patch and the tests share the same working copy, so state does not drift

    A product-agnostic sketch of the loop can align a team on state without tying the idea to any one tool’s interface:

    1. state = init(issue, repo, budget)
    2. while not should_stop(state):
    3. observation = select_context(state) # trim history and file excerpts
    4. action = model.decide(observation) # search, read, edit, or run checks
    5. result = tools.apply(action)
    6. state = update(state, summarize(result))
    7. report(diff(state), verdict(state))

    Replacing select_context, should_stop, and summarize often moves scores more than changing the sampling temperature.

    Why framework rankings barely transfer across models

    Models and frameworks are coupled. They are not freely interchangeable parts. Some models handle long tool-call chains and tolerate verbose observations. Others depend on short instructions and a clear completion signal, and drift once the context gets noisy. Some models propose the next retrieval on their own. Others need the framework to hard-code “reproduce, then locate, then edit” in a template.

    The result is familiar: framework X gives model A an observation density and a stop signal that match its habits, and the score leads. The same framework X on model B makes observations too fragmented or stops too rigidly, and the rank falls. A correlation near zero means there is often no single best universal shell. What exists is a match.

    That also explains a common leaderboard mistake: treating a submitted system’s score as a model score, and treating a comparison that does not disclose orchestration, retry counts, test timeouts, and context policy as a fair comparison. The numbers can differ a lot. The difference is not necessarily intelligence.

    How to make evaluations comparable

    If the goal is to judge the model itself, freeze the execution framework and publish the following: the task set and leakage controls, the allowed tool set, maximum steps and token caps, context-trimming rules, stop and retry policy, whether the test environment is isolated, and whether scoring is one pass or multiple samples. Change only the model. With everything else frozen, the gap is closer to a model difference.

    If the goal is to judge the engineering system, publish the model and framework as one unit and report cost alongside quality: pass rate, average steps, the token share of reads versus edits, and the distribution of failure types. Pass rate alone rewards “search harder by adding budget.” A Pareto view is more useful: who is more stable at similar cost.

    Evaluations also need to guard against two kinds of inflated scores. One is a framework that smuggles in task-set answers or an overly strong repository index, so the model never really locates the code. The other is tests and patches that share state they should not share, so a fluke pass is recorded as capability. When reproducing an experiment, isolate the working copy, dependency versions, and time-sensitive tests.

    What this means for a development workflow

    When building a coding assistant internally, treat the execution framework as a first-class design, not a thin wrapper over the model API.

    First define task state: the issue, the range already read, the current hypothesis, the checks already run, and the remaining budget. The state should be serializable so failed cases can be replayed.

    Then define a minimal tool set: list directories, search by symbol or text, read excerpts by line number, apply a patch, and run a specified check. Every return should carry a length cap and an error code so the whole repository is not pushed into the window.

    Next, prepare switchable policy packs for different models instead of one prompt for every model. A pack only needs observation granularity, whether reproduction is mandatory first, whether a failure should prefer the stack or a reference search, and how many steps of no progress trigger a stop. Before launch, compare them on the same internal defects. Look at stable passes and cost, not demo cases.

    Finally, wire evaluation into continuous integration. When the framework or prompt policy changes, rerun a fixed subset so a change that “looks smarter” cannot silently regress the benchmark. Logging and monitoring are a natural place to watch steps, failure types, and timeouts. Object storage is a natural place to keep replayable traces. These are general engineering capabilities and do not have to be locked to one interface. If a team already uses development and operations components on Baidu AI Cloud, traces, metrics, and permission boundaries should still be governed separately so the evaluation environment is not mixed with production repository permissions.

    Takeaway

    The execution framework decides the world the model sees, the order in which it acts, and the standard for finishing, so it can rewrite measured scores. When reading a coding leaderboard, first ask which model-times-framework pair was measured. When building a coding agent, treat context, stop conditions, budget, and tool-return quality as design surfaces equal to model choice. That avoids mistaking an orchestration gain for a model leap, and it avoids keeping a mismatched workflow after the model changes.