Contamination-Resistant Coding Agent Benchmark Design: A Practical Playbook

1. Where the Problem Starts: Once Capability Moves Up, Distortion Becomes the Main Risk

Code capability used to be measured with function-level completion tasks: given a signature and a docstring, produce the implementation. That format was reasonably discriminative while models only had basic syntax competence. As model capability keeps moving upward, the evaluation task itself changed in two ways.

First, the objective shifted from “can it write syntactically correct code” to “can it complete multi-step work in a real codebase” — reading an unfamiliar repository, locating a defect, editing across files, getting the test suite to pass, handling failures and retrying. Second, the scarcity of evaluation data inverted: public problem sets are resampled over and over and gradually degrade from a measurement instrument into a memory-retrieval quiz.

Three classes of distortion follow from this, and any serious agent benchmark has to address them head-on:

• Memory contamination. Task prototypes come from public historical records (commits, defect tickets, discussion threads). A model may already have seen the problem statement, or even the reference answer, during training, so a high score reflects recall rather than reasoning.
• Harness confusion. The same model running under different execution harnesses can show score gaps larger than the gaps between models. If the two variables are not separated, the leaderboard is effectively measuring a model × harness mixture.
• Environment leakage. Allowing web retrieval, allowing questions back to the user, or allowing direct modification of the host environment quietly swaps “task completed” for “external resource obtained,” and the measurement target drifts with it.

The rest of this design is organized around these three distortions.

2. Design Goals: Fix What You Measure as Interpretable Independent Variables

A usable agent benchmark should state, at design time, which quantities are held fixed and which are being measured. The categories below are worth aligning on:

Category	Item	Handling
Independent variable	Model	The only factor allowed to vary
Controlled variables	Execution harness, system prompt, tool set, context window, reasoning depth, step and time budget	All locked and disclosed in the report
Measured quantities	Task completion, resource consumption, stability, security compliance	Reported separately; never collapsed into one composite score
Environment	Runtime image, dependency versions, network policy, permission boundary	Versioned and published together with the task set

The step and time budget is the control that most often slips. If models actually take very different numbers of interaction turns but share a single result threshold, that is equivalent to handing some models an invisible quota. A workable approach is to set a uniform cap while recording a per-task resource-consumption curve, so that facts like “ten times the compute for a 3% gain” become visible instead of buried.

3. Task Set Construction: Reverse-Engineering Real Engineering Records into Natural-Language Requests

3.1 Sources and Desensitization

Prefer real engineering traces as task sources: defect fix records, incremental feature changes, refactoring changes, production incident handling records, internal tickets, and business requirement descriptions. Once collected, every source must pass through a desensitization pipeline first:

1. Remove internal identifiers, secrets, accounts, domain names, metric names, and proprietary terminology.
2. Strip context clues that could locate the original record directly (issue numbers, branch names, timestamps, author information).
3. Discard any sample that cannot be desensitized without losing the essence of the task. Shrinking the set is preferable to introducing leakage.

3.2 Reverse Engineering: Rewriting from the Answer Side Back to the Question Side

The key move is reversing direction. A raw record is usually a complete chain of “problem + solution + discussion,” so lifting the question side verbatim also publishes the search entry point to the answer. Reverse engineering instead keeps the repository state and the test expectations that correspond to the solution, and rewrites the question side so that it covers the same technical requirement semantically while being unmatchable against the original record in wording and structure.

3.2 Conversational Rewriting and Role Assignment

Rewriting layers three techniques:

• De-templating. Drop specification-sheet phrasing such as “please implement feature X subject to requirement Y,” and write short requests in the voice of the person who needs the work done.
• Role assignment. Inject roles like “I own this service” or “I just inherited this code.” The objective shifts plausibly with the role: for the same repository, an owner cares about regression risk while a newcomer cares about comprehension cost.
• Deliberate incompleteness. Omit some constraints on purpose and see whether the model reads the code, runs the build, and checks configuration to fill the gaps itself. This matches real work and sits closer to the intent of measuring reasoning.

One clarification: conversational rewriting is not about creating ambiguity or setting traps. Its purpose is to cut the retrieval path from “prompt text → original record.” After rewriting, manual spot checks must confirm that the task objective is still uniquely determinable and the acceptance criteria are still objective.

3.4 Workspace Assembly and Verifiability

Every task maps to a self-contained workspace: a source snapshot, a dependency manifest, build scripts, baseline tests, and hidden tests used for scoring. Prefer executable assertions over subjective human grading:

# Task specification (illustrative)
task_id: refactor-auth-timeout-014
domain: code            # code | web | office | security
prompt: |
  The timeout settings on our auth middleware keep getting complaints.
  Latency swings, and we see occasional 5xx in production.
  Please sort it out without breaking existing behavior.
workspace:
  snapshot: repo@sha256:...        # immutable snapshot
  install: ["install_deps", "bootstrap_local_cache"]
  baseline_test: "run_test_suite --select existing"
score:
  hidden_tests: "run_test_suite --select holdout"
  required_pass_ratio: 1.0
  max_wall_clock_sec: 1800
  max_steps: 120
  network: none
constraints:
  allow_user_question: false

Scoring looks only at the hidden-test pass rate plus a few necessary static constraints (for example, the test files themselves must not be modified, and no new external dependencies may be introduced). That keeps the scorer hard to bypass by “editing tests until they turn green.”

4. Anti-Contamination: Rewriting and Version Management Must Work Together

Rewriting alone is not enough. Rewriting raises the cost of retrieval matching, but if the task set stays frozen, a model can still absorb problem statements gradually through community discussion, secondhand excerpts, and iteration feedback. Two layers are therefore stacked.

Layer one: rewriting blocks literal matching. The goal is to make the path “prompt fragment → public record” dead. An operational criterion: run the rewritten prompt through a general web search, and nothing in the results should point back to the original engineering record.

Layer two: version management blocks time accumulation. Publish the task set as immutable snapshots under semantic version numbers, with a full change log for every release. The set used during training and development is physically separated from the set used for the final report, and a held-out set is kept private for the long term to check whether scores are inflated by overfitting. Empirically, a systematic rank reversal between the public set and the held-out set means the benchmark has saturated or been adapted to.

A pragmatic supplement is dynamically generated task families: maintain parameterized templates for one technical intent (different repositories, defect variants, constraint combinations) and rotate samples per version. Difficulty distribution stays stable, and the payoff from memorizing one specific task approaches zero.

5. Isolated Execution Environment: Making Pure Reasoning and Execution Measurable

The execution environment should be controlled, reproducible, and auditable — not a simulation of a real developer desktop.

• Process and filesystem isolation. Each task runs in its own container instance. The task snapshot is mounted read-only, the writable layer covers only the working copy, and the container is destroyed when the task ends, with no state carrying across tasks.
• Network policy. Default to fully offline. If a task class genuinely depends on internal resources, replace that with preloaded local caches and a private mirror, converting “network access” into the static condition “local resource availability.” That removes the measurement drift caused by live retrieval.
• No human questions. Interactive clarification injects an irreproducible human factor. Instead, keep the necessary incompleteness in the prompt and measure whether the model verifies things itself. Separately, track “did the model ask a question it could have answered on its own” as an independent observation metric.
• Least privilege. Deny access to sensitive host paths, deny privilege escalation, and deny outbound data transfer by default. Intercept and separately audit destructive actions (deletion, force-push style operations, permission configuration changes) as part of the safety observations.
• Resource and time budget. Apply uniform CPU and memory limits and a wall-clock limit. Exceeding either is a failure, so slow implementations and efficient ones are not lumped under one threshold.
• Reproducible records. Retain the full command trace, file diffs, build and test logs, and resource samples throughout, so any run can be replayed frame by frame afterward.

6. Cross-Harness Validation: Treat the Harness as an Independent Variable

This is the core technique for separating model capability from harness capability. Run the same batch of models on at least two mutually independent execution harnesses to form a model × harness matrix experiment.

Real harness differences usually show up in these details: the granularity of the tool set (structured search, patch application, terminal execution), the context management policy (when to compact, how to preserve key conclusions), the failure recovery logic (retry strategy and rollback approach), and the planning style (plan first then execute, versus adjust while working).

Three points of experimental discipline:

1. Parameter alignment. Align everything alignable — set reasoning depth to the highest setting for all runs, expand the context window to the same magnitude (for example, around 200k tokens), and keep the tool permission set identical. Anything not alignable (harness-specific tools) must be disclosed explicitly and covered by a sensitivity analysis.
2. Repeated sampling. Run each task multiple times under each harness and report the mean and dispersion of the pass rate rather than a single result. On multi-step tasks, single-sample variance alone is enough to flip a ranking.
3. Attribution reading. If a model clearly beats another under harness A and the reverse holds under harness B, the conclusion should be phrased as “this model is a better fit for that harness,” not “model X is stronger than model Y.” Only when rankings agree across harnesses is there grounds to discuss absolute model capability.

# Scheduling skeleton for the matrix experiment (illustrative; names and
# interfaces can be substituted for your own)
def run_matrix(models, harnesses, tasks, repeats=3):
    results = []
    for m in models:
        for h in harnesses:
            for t in tasks:
                for i in range(repeats):
                    env = Environment(
                        snapshot=t.snapshot,
                        network=False,
                        allow_user_question=False,
                        max_steps=t.max_steps,
                        max_wall_clock_sec=t.max_wall_clock_sec,
                        reasoning_depth="max",
                        context_window=200_000,
                    )
                    trace = h.run(model=m, prompt=t.prompt, env=env)
                    results.append(score(t, trace))
    return aggregate(results)   # aggregate per (model, harness) group

7. Task Domain Design: Differentiated Assessment Across Four Scenarios

A single task type hides capability gaps. Cover at least four domains, each scored according to its own characteristics.

Code. The focus is multi-file coordination and regression safety. Score primarily on hidden unit tests, supplemented by a regression check that existing public behavior is preserved. Also record change size (files edited, diff lines): at equal pass rates, a smaller change is usually the more controllable one.

Web and frontend. The focus is correctness of visual and interaction results. Beyond functional assertions, use rendered screenshot comparison, DOM structure assertions, and replay of key interaction scripts. Layout-type requirements should tolerate reasonable pixel differences so the scorer is not brittle.

Office and document processing. Aimed at structured output: spreadsheet calculations, document layout, data format conversion, batch file processing. Score by structural validation of the output files (field completeness, formula correctness, style properties, batch consistency) rather than text similarity.

Security. Separate two subtasks explicitly: vulnerability identification and vulnerability fixing. Identification tasks require a verifiable location and an explanation of the exploitation path. Fixing tasks must remove the defect without breaking functionality, and must additionally check whether dangerous patterns were introduced (hardcoded credentials, insecure default configuration, unvalidated input paths). Tasks in this domain must run in an isolated environment so the evaluation itself does not create real risk.

8. Metrics, Statistics, and Reporting Standards

Avoid the idea that one composite score explains everything. Report in layers:

• Primary metric: hidden-test pass rate, with confidence intervals.
• Efficiency metrics: steps, tool calls, wall-clock duration, token and compute consumption.
• Stability metrics: variance of pass rate across repeated runs, and the distribution of failure modes (timeout, privilege violation, incorrect edits, giving up).
• Security compliance metrics: number of blocked privilege-escalation attempts, number of attempted destructive operations.
• Explainability metrics: root-cause classification of failures — the most valuable part for driving improvements in both models and harnesses.

Two statistical cautions matter. With a few hundred tasks, differences of a few percentage points usually fall inside the noise band, so run significance tests instead of ranking directly. And when comparing multiple models, apply multiple-comparison correction to avoid false positives from cherry-picking the best run.

9. Common Pitfalls

• Benchmark overfitting. Iterating against one fixed task set for a long time raises the score while the real experience stays flat. Keep a held-out set and rotate it periodically.
• Brittle scorer. Assertions are coupled too tightly to a reference implementation, so a correct implementation written differently is marked as failed. Judge only externally observable behavior.
• Hidden environment differences. Different dependency versions in the base image, or different cache-warming states, make results incomparable. Hash the image contents and publish them alongside the results.
• Contamination reflow. Once a public report is out, the prompt text spreads and gets absorbed by later training. Counter it with versioning plus parameterized rotation of task families.
• Metric synthesis fallacy. Weighting pass rate, cost, and latency into a single number hides the cost of a doubled bill. Present metrics separately and include the Pareto frontier.

10. Engineering: Turn the Evaluation Pipeline into Standing Infrastructure

Evaluation should not be a one-off exercise. It should be a pipeline that can be triggered daily. A typical implementation looks like this:

• Storage layer. Task set snapshots, image artifacts, and run outputs all go to object storage, isolated in versioned directories with retention policies.
• Scheduling layer. A container platform handles concurrency isolation and elastic scaling, allocating resource specifications dynamically against each task spec, with a queue service for queuing and priority control.
• Observability layer. Run logs are collected centrally, with monitoring and alerting covering anomalies such as a sudden rise in failure rate, exhausted resources, and task backlog.
• Configuration management. Model configurations, harness parameters, and scoring rules all live under version control, so any result can be traced back to the exact configuration combination.
• Reporting layer. Generate per-metric breakdowns and failure root-cause distributions automatically, and keep a comparison view across historical versions to see whether score growth comes with an efficiency regression.

On a cloud provider, each of these maps onto generic product categories — object storage, message queues, container platforms, log services, monitoring and alerting, and configuration and secrets management — without locking into a specific implementation. Baidu AI Cloud offers supporting capabilities in containerized execution, log monitoring, and configuration governance; the concrete selection should be evaluated against team size, task concurrency, and compliance requirements.

Conclusion

The value of a contamination-resistant agent benchmark is not a flattering ranking. It is decomposing “capability” into components that are verifiable, attributable, and reproducible: whether tasks come from real engineering work, whether the prompts block retrieval shortcuts, whether the environment excludes external resources, whether the model and the harness are treated as two independent variables, and whether the conclusions survive statistical testing.

Get these things solid, and benchmark results become an input to improvement rather than material for marketing. For teams planning to build their own evaluation system, the minimal starting path is this: pick one domain (usually code), accumulate 50 to 100 tasks with executable scoring, get the minimal closed loop running — isolated containers, two harnesses, repeated sampling — and only then expand to other task domains and larger scale. A trustworthy measure comes first; steady progress comes after.