# What 466 Agent-vs-Agent Cybersecurity Matches Reveal About the Inflection Point in Adversarial Evaluation

    Static benchmarks no longer predict real-world offensive security performance. Based on 466 matches across 22 batches of AI agent versus agent cyber competitions, this article examines why capture rates collapse when defenders actively patch vulnerabilities, how top agents build a decisive win-rate gap, what methodology makes adversarial evaluation trustworthy, and practical steps for teams building red-blue confrontation platforms.

    What 466 Agent-vs-Agent Cybersecurity Matches Reveal About the Inflection Point in Adversarial Evaluation

    Why Static Benchmarks Fail to Represent Real Security Operations

    For years, security capability evaluation has relied heavily on question-bank style testing: a target environment is provided, the target is assumed to be undefended or only minimally configured, a limited number of attempts is allowed, and scoring is based on whether the challenge is solved. This approach can measure knowledge coverage, single-step exploitation skills, and recognition of common vulnerability patterns. But it misses one critical difference from real-world operations: actual attack and defense is a two-way dynamic game.

    In a real environment, defenders continuously patch vulnerabilities, add detection rules, and adjust permission boundaries based on alerts, logs, and attack traces. Attackers, in turn, change their paths, split their actions, and bypass detection based on feedback. A successful exploit is not the end—it is the starting point of the next round of confrontation. Evaluating only static solve rates therefore tends to significantly overestimate an agent’s actual capability in sustained adversarial scenarios.

    A recent public evaluation adopted a more realistic approach: 22 batches and 466 valid matches were completed, with a leaderboard and win rates used to observe how participating agents stratified. The results showed that capability gaps under dynamic adversarial conditions are far larger than under static testing. Even more telling: in a controlled experiment where the target was undefended and each task allowed two attempts in single-player mode, 30 out of 40 tasks could be fully solved by all three participating agent types. But when those same 30 tasks were moved into a two-way adversarial competition, and the defender started patching vulnerabilities, by round 3 the three agents’ attack capture rates had dropped to 52%, 37%, and 35% respectively. In other words, attacks that seemed nearly guaranteed in an isolated environment saw nearly half—or more than 60% in some cases—fall to defense or miss entirely once placed into a dynamic game.

    This reflects a paradigm shift in cybersecurity evaluation: from “can it solve this task” to “can it still reliably achieve its objective while the opponent actively responds.”

    Stratified Results Across 466 Matches: A Clear Lead and a Win-Rate Gap

    The leaderboard showed clear stratification among participating agents. The top score reached 78.6, the second-place agent scored 73.8, and the leading open-source representative scored 68.5, with remaining participants ranging from 56.3 to 67.1. On scores alone, a gap between the top two and the rest is already visible. Win-rate data makes it even more striking: the top two agents achieved match win rates of 0.83 and 0.86, while the highest win rate among all other participants was just 0.59.

    Scores and win rates together indicate that the top agents’ advantage is not accidental success on a particular category of task, but consistent performance across multiple rounds of confrontation. Dynamic adversarial evaluation tests at least the following capabilities:

    • Strategy adaptation: Can the agent adjust attack paths in response to the defender’s patching actions?
    • Feedback utilization: Can the agent extract useful information from failed attempts, environment changes, and partial successes?
    • Long-horizon planning: Can the agent allocate actions across multiple rounds of budget rather than pursuing only single-step success?
    • Error recovery: When a path is blocked, can the agent quickly switch to an alternative plan?
    • Stability and reproducibility: Can the agent maintain low variance across similar tasks?

    When these capabilities compound, top agents build a “discontinuous advantage.” It is worth noting that 466 valid matches provide a meaningful observation window, but results should still be interpreted in light of batch divisions, task sampling, budget settings, and statistical confidence intervals. The value of the evaluation lies not in permanently labeling any agent, but in revealing how capability structure changes under dynamic confrontation.

    High Static Scores Don’t Equal Real-World Capability: Lessons from the 30-Task Controlled Experiment

    The 30-task controlled experiment is key to understanding the value of dynamic evaluation. It ran in two modes:

    1. Single-player mode: Undefended target, two attempts per task, focusing on whether agents could solve the tasks.
    2. Two-way adversarial mode: A defender was added who patched vulnerabilities, added detections, and adjusted strategy based on attack outcomes, with attack capture rates recorded per round.

    In single-player mode, 30 out of 40 tasks could be fully solved by all three participating agent types, showing they already had strong coverage of common vulnerability patterns and exploitation paths. Yet when the same 30 tasks entered the adversarial competition and the defender began patching, by round 3 the three agents’ capture rates had fallen to 52%, 37%, and 35%.

    Here, “capture rate” means the proportion of attacks that successfully achieved the objective within the given rounds and budget. It differs from the static solve rate: a static solve rate assumes the environment never changes, while capture rate is jointly affected by defensive patching, detection interception, path blocking, and round budgets.

    This result makes at least three points clear:

    • Static capability is an entry ticket, not a real-world verdict. Solving tasks only demonstrates baseline ability to find and exploit paths.
    • Defensive patching rapidly shrinks the attack window. When defenders can patch in response to feedback, the fixed paths attackers depend on quickly become ineffective.
    • Dynamic evaluation better exposes strategy weaknesses. High scores in isolated mode can magnify into obvious gaps in multi-round games due to path dependency, poor feedback utilization, and bad resource management.

    Put simply: static question banks measure “knowledge stock,” while dynamic confrontation measures “strategy gains.”

    Where the Gap Comes From: Five Key Variables in Dynamic Games

    1. Strategy Adaptation and Learning from Feedback

    In dynamic confrontation, every attacker action changes the environment, and defenders leave patching traces behind. Top agents are better at extracting constraints from failures—for example, after one path is blocked, pivoting to an adjacent entry point, changing trigger conditions, or reassembling the attack chain. Agents lacking feedback learning tend to repeatedly attempt similar paths, causing capture rates to collapse quickly.

    2. Patching Speed and Detection Coverage

    The shorter the defender’s patching loop, the smaller the attack window. The sharp drop in round-3 capture rates essentially reflects how quickly the defender patched vulnerabilities and added detection rules after observing attacks. Without defensive patching actions in the evaluation, this window contraction cannot be observed.

    3. Long-Horizon Planning and Resource Management

    Real-world operations do not allow unlimited attempts. Rounds, time, request counts, and observation capability all constitute budgets. Top agents allocate exploration and exploitation within budget, avoiding exhausting opportunities on low-value paths. Weaker agents may over-consume in early rounds and fail to organize effective actions later.

    4. Information Asymmetry and Uncertainty

    Defenders hold the perspectives of assets, permissions, and logs; attackers can only infer the environment through limited feedback. Information asymmetry amplifies strategy differences. Agents that can actively probe, validate hypotheses, and control noise are more likely to maintain stable capture rates.

    5. Stability and Reproducibility

    A single success may be luck; winning consistently across multiple batches demonstrates capability. The gap between win rates of 0.83/0.86 and the next-best 0.59 reflects not just peak performance but variance control. For enterprises, security capability must be repeatable and auditable, not dependent on occasional success.

    Evaluation Methodology: Making Adversarial Results Trustworthy

    For evaluations like this 466-match series to be meaningful, the evaluation itself must be designed as a repeatable, auditable, and fair adversarial system. Key elements include:

    • Batch division: Group tasks by type, difficulty, and environment configuration to avoid a single scenario dominating results;
    • Valid match determination: Define clear rules for handling abnormal interruptions, environment errors, timeouts, and invalid actions;
    • Balancing score and win rate: Scores reflect overall performance; win rates reflect adversarial consistency;
    • Rounds and patching windows: Fix attack-defense round counts, defender patching timing, and observation windows;
    • Budget control: Apply uniform constraints on attack steps, request counts, and time;
    • Randomization and blind testing: Reduce overfitting to task ordering or known answers;
    • Log auditing: Record actions, patches, and outcomes per round to support reproduction and attribution;
    • Statistical confidence intervals: Avoid interpreting small-sample fluctuations as capability discontinuities.

    A simplified adversarial evaluation loop can be expressed in the following pseudocode:

    1. for batch in batches:
    2. for task in sample_tasks(batch):
    3. state = init_environment(task, defense_policy, attack_policy)
    4. for round_id in range(max_rounds):
    5. defense_patch = defense.observe_and_fix(state)
    6. attack_result = attack.run(state, budget=round_budget)
    7. state = apply_patch_and_result(
    8. state,
    9. defense_patch,
    10. attack_result
    11. )
    12. record(
    13. batch=batch,
    14. task=task,
    15. round_id=round_id,
    16. captured=attack_result.captured,
    17. defense_action=defense_patch.action,
    18. attack_path=attack_result.path
    19. )

    The point of this pseudocode is not the specific interfaces, but the evaluation loop: the defender observes and patches, the attacker acts within budget, and the system records results per round. Only when this loop exists can static tasks be transformed into dynamic adversarial samples.

    Engineering Implementation: Key Modules of a Red-Blue Confrontation Platform

    Enterprise teams don’t need to build a large-scale evaluation system from day one, but should progressively build around the following modules:

    1. Task and environment orchestration: Support batch creation of target ranges, initialization, network isolation, and resource cleanup;
    2. Range isolation and access control: Ensure attack actions never affect production environments, with sensitive data masked;
    3. Defender patching interface: Allow defenders to submit patch actions and record state before and after;
    4. Attack budget and round control: Uniformly constrain time, steps, and attempt counts;
    5. Observation and logging services: Collect action logs, alert logs, and patch records, with playback support;
    6. Scoring and leaderboards: Compute score, win rate, capture rate, and patch time together;
    7. Regression set management: Turn historical failure cases into regression samples to prevent capability regression;
    8. Anti-cheating and fairness checks: Prevent direct answer reading, evaluation boundary bypassing, or environment exploitation.

    When deploying in the cloud, general-purpose service categories can be combined—for example, container platforms for isolated ranges, logging services for confrontation records, monitoring and alerting for anomalous behavior, and object storage for evaluation results and playback data. The key is to make “rounds, patching actions, observation evidence, and scoring results” form an auditable chain.

    Recommendations for Developers and Enterprise Teams

    1. Establish a Dual-Track Evaluation System

    Use static question banks for quick screening of knowledge coverage, and dynamic confrontation for validating real-world strategy. Neither can replace the other. Treat static solve rates as a baseline metric, and multi-round capture rates, win rates, and post-patch recurrence rates as core metrics.

    2. Bring the Patching Loop into Security Metrics

    Counting only successful attacks overestimates risk. Also track patching time, detection coverage, false positives and negatives, and recurrence rates of similar issues. The round-3 capture rate drop shows defensive patching works—enterprises should incorporate this capability into routine exercises.

    3. Control Budgets and Rounds to Simulate Real Constraints

    Unlimited attempts distort results. Set round, time, and request budgets close to real scenarios, and observe agent strategy quality under constraint.

    4. Use Historical Failure Cases for Regression

    Accumulate failed attack paths, intercepted actions, and post-patch retest samples into a regression set, and automatically rerun them after version iterations and strategy updates—avoiding “high static scores, dynamic regression.”

    5. Focus on Stability, Not Single-Run Peaks

    The win-rate gap matters more than a single highest score. When selecting vendors or running internal evaluations, watch multi-batch variance, long-round performance, and exception recovery capability.

    6. Maintain Isolation and Compliance Boundaries

    Red-blue confrontation must run in isolated environments, with strict control over permissions, data, and network boundaries. Evaluation results should be auditable and replayable, and adversarial samples should never be used directly in production environments.

    Conclusion

    The biggest insight from 22 batches and 466 valid matches is not how many points any agent scored, but that cybersecurity capability evaluation is shifting from static testing to dynamic adversarial games. High solve rates in static mode can quickly convert into much lower capture rates once defensive patching and two-way feedback are introduced. Top agents build a win-rate gap not through single-task hits, but through strategy adaptation, feedback utilization, long-horizon planning, and stability.

    For developers and enterprise teams, what truly needs to be built is a sustainable red-blue confrontation evaluation loop: with defensive patching, round budgets, log auditing, regression sets, and a reproducible scoring system. Cybersecurity is entering a stage of continuous confrontation. Static high scores are only the entry ticket—dynamic adversarial capability is what determines real-world performance.