Coding Agent Offense-Defense Evaluation: The Capability Gap Static Benchmarks Miss
Static security benchmarks measure one-shot problem solving, while multi-round offense-defense evaluation measures adaptation under continuous patching. This article explains why high static scores can overstate practical capability, outlines evaluation design choices such as patch budgets and reproducible evidence, and shows how to interpret score tiers, runtime effects, and common biases.
When security evaluation shifts from “solving a task correctly” to “making progress while an opponent keeps patching,” coding agents begin to reveal their real capability gap. A static benchmark measures one-time problem solving. Dynamic adversarial evaluation measures strategy adjustment, evidence use, and the ability to remain effective under changing constraints. The two are related, but they cannot replace each other.
Why High Static Scores Can Be Misleading
A common single-player setup is: the target has no defenses, each task allows multiple attempts, and correctness is judged by whether the expected result is reached. This design is suitable for regression on knowledge coverage, but it assumes a static environment, reusable answers, and no opponent feedback on failure. For vulnerability-analysis tasks, models often rely on known patterns: dangerous functions, missing validation, overly broad permissions, insecure deserialization, and so on. As long as the task is close to the training distribution, several agents at similar levels may all pass.
Passing everything does not mean the capabilities are homogeneous. It only shows that, under conditions with no adversary, sufficient information, and repeated trial and error, these systems can all reach the endpoint. Once the defender modifies code, tightens permissions, adds validation, or changes interface semantics based on attack traces, previously effective paths stop working. At that point, the gap comes from three things: whether the agent can extract actionable evidence from failure; whether it can rewrite assumptions without breaking business constraints; and whether it can organize local findings into a sustainable verification loop. Static accuracy barely records these processes.
What Dynamic Adversarial Evaluation Measures
An effective match includes at least offense, defense, and adjudication. The offensive side finds and validates exploitable conditions within an authorized scope. The defensive side patches based on evidence that is public or disclosed with a delay. The judge scores only reproducible evidence, not narrative explanations. When the match runs for multiple rounds, the environment changes each round, and the previous exploit script cannot be used directly as the current round’s answer.
Four types of observations can be abstracted without binding to any specific model:
- Initial capture: whether effective verification can be established when there is no defense or only weak defense.
- Survival after repair: whether the original path still holds after defenses are applied, or whether the agent can pivot within the rules.
- False positives and disruption: whether patching introduces new failures, and whether offense treats non-reproducible guesses as success.
- Stability: whether conclusions remain in the same range when the same task is run with a different random seed or an equivalent implementation.
A stable industry phenomenon is already visible: leading systems pull away in match win rate, while mid-pack and lower systems are nearly tied on static tasks. More diagnostically useful is a comparison on the same set of tasks: without defenses and with limited attempts, most tasks are solved by multiple agents; after entering head-to-head matches with continuous patching, the offensive capture rate in later rounds falls sharply, and the size of the drop is uneven. This shows that “everyone can solve the task” masks “who can still perform when conditions change.” Specific scores will vary with tasks, rounds, and adjudication rules, so a single leaderboard should not be treated as a capability ceiling.
Evaluation Design Considerations
Tasks should come from reproducible engineering defects, not vague instructions to “find all issues.” Each task needs fixed initial code, a runtime environment, allowed operation boundaries, success criteria, and prohibited actions. Success criteria should be automatically checkable states wherever possible—for example, a specific invariant is violated and can be stably reproduced—rather than the model saying “I have successfully exploited it.”
Defense cannot be indefinite manual improvisation, or matches will not be comparable. A more reliable approach is to predefine a patch budget: how many files may be modified per round, which regression suite must continue to pass, and which external contracts cannot change. After a patch is generated by another agent or by fixed rules, it must pass the build and regression tests before the next offensive round begins. This measures offense-defense strategy, not one side secretly changing the task.
The number of rounds does not need to be large. Three rounds are often enough to show decay: the first round is close to static capability, the second tests whether the agent can read the patch, and the third tests whether it keeps hitting the same invalid path. There must be enough batches to separate task difficulty from randomness. Only at the scale of twenty-plus batches and hundreds of valid matches do tiers begin to be credible; with fewer samples, a single unusual task can easily dominate the results.
For scoring, keep both outcomes and process, but use process only for explanation, not as the main ranking basis. Outcomes should use win rate, capture rate, and survival rate after repair. Process should record the number of hypothesis changes, the share of invalid attempts, and regression failures caused by patches. Avoid relying on a single total score; otherwise teams cannot tell whether the weakness is in offense, defense, or environment setup.
A schematic adjudication logic can be very simple; the key is determinism:
for round in rounds:attack_ok = reproduce(evidence, current_target)if attack_ok:offense_score[round] += 1patch = defend(current_target, evidence, budget)current_target = apply_if_regression_passes(patch)report(capture_rate_by_round, win_rate)
Here, reproduce must be executed by the evaluator; participant logs should not be trusted. Evidence format, timeouts, and network isolation should all be written into the rules in advance.
How to Read Tiers, Not Myths
A match win rate significantly higher than that of other players usually means a more stable strategy, not that the agent “can autonomously break any system.” A drop in capture rate after defense shows that the task has begun to measure adaptation, and that many apparent successes depend on a specific unpatched implementation. When reading results, do at least three things:
- Look at the difference for the same task between single-player mode and adversarial mode. A large difference indicates inflated static scores.
- Look at the slope of capture rate across rounds. A flatter slope indicates that the agent can reorganize verification in the face of patches.
- Check whether win rate is consistent with absolute score. High win rate but average total score may mean the agent is good at extended matches but not at first-solving hard tasks; the reverse may indicate reliance on opening luck.
It is also unwise to use open source versus closed source, or parameter scale, as a direct explanation. The same agent connected to different coding tools, context lengths, test feedback, and retry strategies will score differently. Evaluation reports should describe “model” and “runtime” separately: who is reasoning, who is executing commands, and who decides when to stop. Otherwise, the tiers reflect the whole pipeline but are misread as an inherent property of a single model.
Implications for Development Workflows
In security-related development, agents are suitable for hypothesis generation and patch drafts first, not for directly serving as a release gate. A more reasonable usage is: static scanning and single-task regression cover known patterns; small-scale adversarial matches spot-check high-risk modules; humans confirm the evidence chain and business impact. If an assistant performs well only on an undefended benchmark, but its capture rate halves after two rounds of patching and it cannot explain the failures, its automatic modification permissions should not be expanded.
Organizational rollout can be risk-tiered. Low-risk repositories can allow agents to submit patches, but the patches must include reproducible tests. Changes involving authentication, permissions, payments, and data isolation should default to human review, and adversarial evaluation should be used as pre-release sampling rather than daily full coverage. The evaluation set itself must also prevent leakage: public tasks quickly become new static memory, so unpublished same-distribution tasks should be retained, and defect variants should be rotated regularly.
If cloud object storage is used to hold tasks and evidence, a container platform is used to isolate each match, and logs and monitoring record commands and timeouts, the focus should be on isolation and audit, not on stacking product names. Network egress should be denied by default, the root file system should be read-only with a writable workspace, and keys should not enter the participant context. These constraints are themselves part of evaluation validity: a match in which answers can be freely inspected or external solving services can be called has no comparative meaning.
Common Biases
Treating one leaderboard as the final word on capability is the most common bias. If the task distribution changes, the tiers will change. Next is rewarding only offense without penalizing destructive patching, which leads defenders to remove functionality in exchange for “security.” Another is still awarding points when evidence is not reproducible; models will learn to write as if they succeeded. Finally, declaring a gap from too few samples: 0.8 versus 0.6 without confidence intervals and task-difficulty notes may be nothing but noise.
A safer conclusion is this: dynamic adversarial evaluation can reveal differences in adaptability that static question answering cannot see; these differences have direct implications for whether to grant coding agents more autonomy; but the evaluation measures relative performance under specific rules, not authorization to test real production systems, and certainly not an intrusion guide. Any probing of external systems must have prior written authorization, and evaluation should be limited to self-built, isolated, resettable targets.
Practical Checklist
- Keep both a single-player baseline and at least three offense-defense rounds for the same task.
- Count as success only automatically reproducible evidence, not self-reporting.
- Give defense a patch budget and regression tests to prevent task changes or feature deletion.
- Record the model, execution environment, and retry strategy separately.
- Use multiple batches of matches to examine tiers, and use unpublished variants to prevent memorization.
- Keep human adjudication for high-risk changes; treat agent output only as a candidate.
When evaluation moves from answer accuracy to multi-round offense and defense, teams do not get a flashier ranking. They get a capability-decay curve. The curve tells you how much effectiveness the agent has left after an opponent starts fixing vulnerabilities—and whether you are ready to let it touch real code.
