1. Why Reproducible Evaluation of Code LLMs Matters

When selecting a code LLM, the biggest pitfall is not choosing the wrong model; it is comparing models in a non-reproducible way. If you try a few ad hoc prompts and score based on subjective impressions, the conclusion is often neither reproducible nor attributable: change the prompt or context length once, and the ranking changes.

A qualified hands-on evaluation should fix at least four categories of variables:

• Model side: the model identifier, version, and inference parameters, such as temperature and maximum output length, remain stable.
• Harness side: whether the model runs in a plain chat interface or in an agent runtime with tool-calling capabilities makes a huge difference. The same model performs significantly better in an environment where it can read files, edit multiple files, and run tests than in pure text question answering.
• Environment side: whether network access is allowed, whether command execution is permitted, and what sandbox resource quotas apply.
• Scoring side: unified task prompts, unified scoring criteria, and unified timeout limits.

The benchmark sample used in this article uses the following configuration: a single fixed model version; an agent runtime with tool calling; an evaluation concentrated within a continuous window of about 50 minutes; network access disabled for the entire run, meaning that none of the external links included in the tasks were accessed; a 30-minute time limit per task; six tasks in total, from Task 0 to Task 5; and an actual total execution time of about 43 minutes.

These conditions constitute a measurement of the capability lower bound. They remove external web search as a strong auxiliary and measure the model’s real ability to complete tasks in a closed environment using its own knowledge, repository context, and local tools.

2. Task Design: From Metacognition to Real Engineering Work

The task set should cover different capability dimensions rather than repeat six similar tasks.

Task 0: A Metacognitive Task — Model Self-Description Trustworthiness

The first task directly asks the model: what model are you, what version are you, and when is your knowledge cutoff?

The answer itself is not the main test point. What matters is whether the source of the answer is clearly identified. In the hands-on test, the model explicitly stated three things:

1. The model identifier and host information came from the system prompt injected by the runtime.
2. The knowledge cutoff time was provided by the runtime environment.
3. “The closer a date is to the cutoff, the thinner the memory becomes,” and events after the cutoff are only fragmentarily remembered.

A conclusion that is important for engineering deployment is this: a language model’s self-description of its identity and knowledge boundaries is essentially “restating context,” not reliable introspection. Therefore, in scenarios involving compliance audits, data-export judgments, or license requirements, version information must be based on model fields in invocation logs and response metadata. The model’s natural-language self-report should not be used as evidence. This is more valuable than the fact that “it answered correctly.”

Tasks 1 to 5: Engineering-Oriented Tasks

The following five tasks should be designed to feel like real engineering work, ideally spanning these dimensions:

1. Defect localization and repair: provide reproduction steps and a stack trace; require root-cause identification rather than symptom bypass.
2. Multi-file refactoring: involve interface convergence, naming consistency, and dependency-direction adjustment; assess global consistency and minimal-change capability.
3. API design and unit testing: require contracts, boundary cases, and failure paths; assess test thinking.
4. Build and dependency troubleshooting: under no-network conditions, external resources cannot be fetched; assess recall and reasoning about common failure patterns.
5. Performance optimization: require measurable evidence and prohibit “optimize by feel.”

The scoring dimensions for each task should be fixed at five items, avoiding a simple “correct or not” judgment:

Dimension	What to assess
Correctness	Whether the task’s built-in assertions or tests pass
Runnability	Whether the code can run directly in the sandbox without manual completion
Engineering practices	Error handling, logging, naming, and dependency minimization
Context efficiency	Whether the task is completed with fewer file reads and turns, avoiding meaningless exploration
Latency	Wall-clock time per task, including tool-call overhead

3. Quantifying Evaluation: Turning Subjective Impressions into Regression-Ready Data

The output of an evaluation should be structured data, not a paragraph of conclusions. A minimal executable harness skeleton is shown below. This is illustrative code.

import json, subprocess, time, pathlib
CASES = ["case_0_identity", "case_1_bugfix", "case_2_refactor",
         "case_3_testgen", "case_4_build", "case_5_perf"]
TIMEOUT = 30 * 60  # per-task timeout
def run_case(agent_cmd: list[str], case: str) -> dict:
    t0 = time.time()
    try:
        proc = subprocess.run(agent_cmd + [f"{case}.md"],
                              capture_output=True, text=True, timeout=TIMEOUT)
        status, out = proc.returncode, proc.stdout
    except subprocess.TimeoutExpired:
        status, out = -1, "TIMEOUT"
    tests = subprocess.run(["pytest", "-q", f"tests/{case}"],
                           capture_output=True, text=True)
    return {
        "case": case,
        "exit": status,
        "tests_passed": tests.returncode == 0,
        "wall_seconds": round(time.time() - t0, 1),
        "tokens_in": out.count("\n"),  # token usage should ideally be reported by the runtime
        "tokens_out": 0,
    }
results = [run_case(["./agent", "--sandbox", "readonly+patch"], c) for c in CASES]
pathlib.Path("run_result.json").write_text(
    json.dumps(results, ensure_ascii=False, indent=2))

There are three key engineering details:

• Token accounting data must be reported by the runtime, not estimated from character counts. Modern agent sessions include system prompts, tool definitions, historical messages, and cache hits. Character-based counting can be severely distorted.
• Timeout is part of the capability metric. Whether a task converges within the 30-minute limit directly reflects the efficiency of context management and tool-calling strategy.
• Persist and archive results. Raw evaluation logs and patch artifacts should be retained long-term for regression comparison and dispute review.

4. Cost-Effectiveness Analysis: Pricing Each Usable Result

A model’s unit price alone does not determine cost-effectiveness. The real cost unit is “one mergeable correct result.” The calculation is as follows:

Per-task cost = (input tokens / 1e6) × input price + (output tokens / 1e6) × output price
                (cached portions are priced according to the agreed discount)
Effective cost = per-task cost / first-pass rate
True labor-adjusted cost = effective cost + manual rework minutes × labor cost per minute

Example. The values below are illustrative. If the flagship-tier per-task cost is 1.2 and the first-pass rate is 0.8, the effective cost is about 1.5. If the lightweight-tier per-task cost is 0.25 and the first-pass rate is 0.5, the effective cost is 0.5, but it requires about 15 minutes of manual rework. If the labor cost is 2 per minute, the true cost of the lightweight tier rises above 30.

In teams with high labor costs, “cheap model plus manual fallback” is often the most expensive option. This is also a common root cause of failed cost-reduction efforts.

Layered usage should also be considered. Simple completions, boilerplate code, comments, and documentation generation are suitable for lightweight tiers. Cross-file refactoring, complex defect localization, and architecture-level design are suitable for flagship tiers. Letting a lightweight model perform long-chain refactoring may appear to save on unit price, but the actual loss comes from rework and retry rounds.

5. Common Benchmarking Pitfalls

• Benchmark contamination: public question banks may already be in the training corpus and can systematically overestimate capability. Maintain a private task set derived from real issues in your own repository.
• Capability suppression caused by no network: the sample in this article ran entirely without network access. Conclusions must be limited to “capability lower bound in a closed environment” and should not be extrapolated to overall capability.
• Single-sample noise: language model outputs are stochastic. Each task should be sampled multiple times, and both mean and variance should be reported; otherwise, rankings are not trustworthy.
• Order effects: completing six tasks consecutively in the same session creates context accumulation, which may affect later tasks. Measure both “independent sessions” and “continuous sessions” separately.
• Scoring subjectivity: the engineering-practices dimension must include anchor examples, use double-blind scoring, and report inter-rater agreement.
• Safety: agents may modify files and execute commands. Always run them in an isolated sandbox, restrict network egress and filesystem write permissions, and prevent the evaluation environment itself from becoming a risk surface.

6. Implementation Recommendations

1. Build a private regression task set. Extract 50–200 real tasks from your own codebase and run quarterly regressions to form a trend curve rather than a single-point conclusion.
2. Use general cloud components to build the pipeline. Store task sets and logs in object storage, run sandboxes on a container platform, distribute tasks with a message queue, and continuously monitor success rate and cost with logging and alerting. The goal is to ensure that “every model upgrade has comparable data.”
3. Put cost governance into the invocation layer. Enable prompt caching, route tasks to different tiers based on complexity, trim irrelevant context, and run parallelizable subtasks concurrently. These measures often affect the bill more than switching models.
4. Establish a model-information audit mechanism. Metadata such as version number and knowledge cutoff should always be taken from response metadata and invocation logs; do not accept the model’s self-report.

Conclusion: The real value of a six-task timed evaluation is not the leaderboard. It is the accumulation of a reproducible methodology: fixed variables, structured scoring, cost accounting per usable result, and long-term regression using a private task set. Capability determines the upper bound; engineered evaluation and cost governance determine whether you can reliably reach that upper bound.