Adversarial Task Generation for Coding Agent Benchmarks: Breaking the Saturation Trap

    Static coding benchmarks are rapidly losing discriminative power as data contamination and memorization inflate scores. This article explains why simply adding fresh problems is not enough, then presents an adversarial task-generation and co-evolution approach in which coding agents create and solve tasks for one another so that difficulty grows with capability. It also covers practical design requirements such as runnable verification, sandboxed execution, anti-collusion safeguards, difficulty calibration, and infrastructure needs for reliable continuous evaluation.

    1. Why Coding Benchmarks Expire

    When evaluating the real capabilities of AI coding agents, the industry has long relied on a set of classic code-repair benchmarks. These benchmarks are typically built from historical issues in real repositories: a model is given a buggy codebase and a problem description, and it must produce a patch that passes all tests.

    At first, this approach worked well. But as the strongest models began to post increasingly impressive results, an awkward fact emerged: they were not truly “solving” the problems so much as having “seen” the answers before. The cause is data contamination. As training corpora expanded dramatically, the public benchmark problems and their reference patches were quietly absorbed into training data. Models remembered the solutions themselves rather than demonstrating reasoning ability.

    It is like an exam whose question bank has not changed for three years. When more and more students score perfectly, it is often not because they suddenly understand the subject better, but because the answers have long been printed somewhere in the textbook. Once a benchmark can be “memorized,” its discriminative power collapses quickly.

    2. Why Replacing Problems Is Not Enough

    The most intuitive response is to keep replacing the problems. A class of time-sensitive benchmarks is designed exactly this way: they only collect new issues that appeared after the model’s training cutoff. This works in the short term, but it hides a deeper structural weakness.

    Problem 1: The difficulty ceiling is set by the task author, not by the system being tested.
    New tasks usually come from bugs and feature requests that human developers submit in their daily work. That means task difficulty is tightly capped by “the bugs humans happen to write.” Consider an analogy: suppose you want to test a group of sprinters, and every year you change the track and the starting gun. The setup feels fresh, but as long as the race distance remains fixed at 100 meters, increasing training intensity will eventually allow everyone to run under 10 seconds, and the test loses its ability to distinguish between athletes. The race distance itself must grow along with the athletes’ capabilities.

    Problem 2: Letting models generate their own tasks does not necessarily decouple difficulty.
    Having models generate test tasks is another explored path, but it also has two limitations:

    • One-off static task sets: The tasks are generated by AI, but they are packaged as a static collection. They do not automatically become harder as the tested model becomes stronger, so they soon fall back into saturation.
    • Adversarial mechanisms with limited source material: Even if a “propose-and-solve” adversarial structure is introduced, if the task material still comes from a fixed pool of human problems and AI is only asked to fill in test cases, the difficulty ceiling still cannot escape the question of “how hard human bugs are.”

    The conclusion is clear: the real issue is not whether tasks are new, but whether difficulty has a mechanism for evolving in step with model capability.

    3. Core Idea: Adversarial Self-Generation and Co-evolution

    The key to solving the saturation trap is to turn evaluation from a “static exam” into a “dynamic game.” The core idea is to let multiple coding agents generate tasks for one another and solve one another’s tasks, with difficulty rising naturally through the game. This is co-evolution.

    Imagine two roles: a Proposer constructs a runnable programming task and provides a reference solution and validation tests; a Solver attempts to repair or implement the target under constraints. The stronger the Proposer, the more challenging the tasks it can construct. The stronger the Solver, the better it can handle challenging tasks. As the two apply pressure alternately, the difficulty ceiling is no longer preset by humans; it is dynamically raised by the agents’ level of play.

    The fundamental difference from a fixed task set is this: difficulty becomes an endogenous variable that grows automatically with capability, allowing the evaluation to retain discriminative power even as capabilities continue to improve.

    4. Key Points for a Practical Design

    Adversarial evaluation sounds elegant, but to be truly usable it must solve two major engineering problems: task validity and anti-cheating.

    1. Tasks Must Be Runnable and Verifiable

    Every generated task should have a clear pass criterion, usually a set of automated tests such as unit tests. During scoring, the agent’s patch is executed in a sandbox environment, and an objective pass/fail result replaces subjective grading:

    1. def run_duel(proposer, solver, rounds=100):
    2. stats = {"solved": 0, "invalid": 0}
    3. for _ in range(rounds):
    4. # 1) Proposer constructs task + reference solution + validation tests
    5. task = proposer.generate_task()
    6. # 2) Validate task: reference solution must pass tests, baseline code must fail
    7. if not is_valid(task): # Filter invalid or giveaway tasks
    8. stats["invalid"] += 1
    9. continue
    10. # 3) Solver attempts to generate a patch in sandbox
    11. patch = solver.solve(task)
    12. # 4) Run validation tests in isolated environment for objective result
    13. if verify_in_sandbox(patch, task.tests):
    14. stats["solved"] += 1
    15. # 5) Dynamically adjust subsequent task difficulty based on results (co-evolution)
    16. proposer.adapt_difficulty(stats)
    17. solver.adapt_difficulty(stats)
    18. return stats

    2. Prevent Collusion and Reward Hacking

    Adversarial mechanisms have inherent loopholes: the Proposer and Solver may “collude,” constructing overly simple tasks or tasks with shortcuts so that both can inflate their scores. Possible countermeasures include:

    • Validity threshold: Require the reference solution to pass all tests and the initial code to fail; otherwise, discard the task.
    • Difficulty calibration: Use historical pass rates as difficulty labels and filter out tasks with abnormal pass rates that are too high or too low.
    • Cross-validation: Have a third-party solver agent blind-test tasks to check whether their difficulty and discriminative power are real.
    • Resource isolation: Run patches in an independent sandbox to prevent agents from cheating by reading or writing test files.

    3. Engineering and Compute Support

    Continuously generating tasks and repeatedly compiling and running tests in containers carries non-trivial compute overhead. General cloud capabilities can support this: use a container platform to host isolated sandboxes, a message queue to schedule the “propose–solve–verify” pipeline, object storage to archive tasks and patch artifacts, and monitoring and alerting to track evaluation pass rates and resource usage. Cloud platforms such as Baidu AI Cloud provide general-purpose infrastructure capabilities of this kind.

    5. Practical Recommendations and Caveats

    • Maintain diversity: A single domain, such as only covering one class of algorithm problems, encourages overfitting. Tasks should be distributed across languages, scenarios, and difficulty levels.
    • Beware of overfitting the evaluation method itself: Once agents learn to cater to the “propose–verify” rules, metrics can become artificially inflated. Regularly introduce manual spot checks and external benchmarks for cross-checks.
    • Separate capability from memorization: The advantage of dynamic games is precisely that tasks are continually new and difficult to “memorize answers for” in advance, bringing evaluation closer to real engineering capability.
    • Control the pace: Difficulty should increase gradually to avoid evaluation noise caused by unsolvable tasks.

    6. Conclusion

    Evaluation for coding agents is moving from “fixed exam papers” to “dynamic games.” Data contamination causes static benchmarks to saturate quickly, while simply replacing problems remains constrained by the difficulty ceiling of human task authors. Adversarial self-generation combined with co-evolution makes difficulty an endogenous variable that grows with capability, offering a viable direction for breaking the saturation trap. In production, engineering controls are still needed around task validity, anti-cheating, compute isolation, and diversity. Only then can evaluation continuously and credibly measure whether agents can actually write code, rather than whether they have memorized answers.