A Practical Multi-Model Coding Evaluation: The Six-Task Method, Metrics, and Cost Breakdown

1. Why Run a Six-Task Hands-On Test Instead of Trusting Leaderboards

Model release cadence has clearly accelerated across vendors, and selection decisions tend to fall into one of two extremes: reading leaderboard rankings only, or going entirely on feel. Leaderboards answer “who is stronger on average,” while what a team actually has to answer is “under my task distribution, my context sizes, and my cost constraints, who completes the work most reliably and most economically?” The two questions often produce different conclusions: a model that leads on general-purpose tasks can land as mediocre inside a real codebase once context-management strategy, tool-calling habits, and output-length control differ from what the benchmark assumed.

What is worth borrowing from this hands-on record is not merely the “six tasks” result, but the discipline around it: define the exam room first, then start the clock, and log elapsed time and the whole process as you go. In what follows, “multiple models,” “lightweight / mid-size / large models,” and “agentic coding tools” correspond to the different models being compared side by side in the hands-on run, and to the host environments that carry those models; all are used as neutral terms. What this article dissects is the method and the measurement conventions. Any specific brand or model version should be verified against its own official information.

2. Exam Conditions: Write “Fairness” as Reproducible Parameters

Whether an evaluation conclusion is persuasive depends first on whether the conditions are stated completely and without ambiguity. The opening declaration of the original record is worth copying, and it listed six key constraints:

• Model identifier: the exact version of the model under test.
• Runtime host: the agent runtime or coding tool environment. The same model can behave noticeably differently across hosts, because the system prompt, tool permissions, and context management are all decided by the host.
• Time window: start and end times precise to the minute, used to align version changes and traffic fluctuations.
• Network policy: the network was disabled for the entire run, and any external links given in the prompts were never visited. This isolates the question “without external retrieval, can the model complete the task using only its own knowledge and the local code?”
• Per-task time limit: 30 minutes maximum per task; exceeding it counts as not completed.
• Delivery record: a per-task summary of the answer and the time spent.

The checklist can be extended with a few more items:

Parameter	Recommended practice	Why
Decoding randomness	Fix temperature, random seed, and max output length	Reproducibility is a prerequisite for comparison
Retry policy	Record retries separately	If retry counts differ across models, total elapsed time becomes distorted
Working directory	Use a fresh copy per task; forbid cross-task reuse	Prevents “last task’s edits” from contaminating the next task’s starting point
Scoring method	Freeze the acceptance criteria before each task starts	Prevents “setting the standard after seeing the answer”
Task set source	Self-built tasks as the primary source, public tasks as a supplement	Lowers the risk of benchmark items leaking into training data
Result retention	Archive full conversations, tool calls, patches, and logs	Makes post-hoc review and regression possible

It is worth noting that the whole exam took roughly 43 minutes even though the per-task ceiling was 30 minutes. Difficulty was clearly uneven across tasks: some were answered in a minute or two, others ran close to the limit. If you only look at “total elapsed time” or “average time per task,” it is easy to underestimate the real cost of long-tail hard problems. The correct approach is to record duration per task, form a duration distribution, and then discuss results by difficulty tier.

3. Task 0: A Model’s Self-Description Cannot Be Taken as Fact

The first item, Task 0, was extremely plain: “What model are you? What version? When does your knowledge base end?” The model under test answered in three parts:

1. It reported its model name and identifier, plus the runtime environment it was in.
2. It stated its knowledge cutoff as June 2026.
3. It explicitly noted that this information came from the system prompt supplied by the runtime environment rather than from any “introspection,” and it admitted that the closer you get to the cutoff, the sparser the memory: events after spring 2026 existed only as scattered impressions.

That answer lands right on a common mistake in evaluations: treating a model’s description of itself as fact. Several points have to be stated clearly:

• Models have no reliable self-inspection ability. Statements about their own version, parameter count, or training-data cutoff are essentially restatements of related text in the context, or imitations of common phrasings in the training corpus. Change the system prompt and the same model can report a different version number or a different cutoff date.
• The knowledge cutoff is the point where reliability starts degrading, not a hard line. Content after the cutoff is entirely absent; content close to the cutoff is covered more sparsely and remembered more vaguely. Anything involving version numbers, pricing, policies, or API changes must not rely on model memory and has to be checked against official documentation.
• Time-sensitive questions should be scored separately. For example, asking a model to list news from “the last few months” measures whether those events happen to fall in a dense region of the training data, not whether the model can code or reason. It measures corpus coverage and should not be counted toward general capability scores.
• The evaluation itself must guard against injection. If the system prompt is polluted, a model may claim to be a different version and thereby contaminate the conclusions of the whole run. The safe practice is to take the model identifier returned by the gateway or configuration service as authoritative, and to record the model’s self-description only as a reference for “how it understands its own identity.”

Task 0 works more like a meta-calibration item: it reminds us that anything along the lines of “the model says it is X” should be labeled “to be verified” rather than “known.”

4. A Six-Dimension Task Framework: Make the Tasks Cover Real Engineering Ability

With only six tasks from Task 0 to Task 5, the sample size is limited, so each task should cover one independent capability dimension as far as possible and come with explicit acceptance criteria. The framework below is a reasonable reference:

Dimension	What it probes	How it is judged
1: Basic comprehension and modification	Reading requirements and existing code, completing a local change	Unit test pass rate
2: Cross-module reasoning	Locating call chains, identifying interface contracts and implicit dependencies	Integration test results
3: Tool use and self-checking	Whether the model proactively runs builds, tests, and static checks, and self-corrects from the errors	Tool call sequence, final build state
4: Failure recovery	When the first approach fails, whether it switches paths instead of repeatedly patching	Number of attempted paths, whether it converges
5: Constraint compliance	Hard constraints such as no network, do not modify designated files, performance budget	Assertion checks, diff audit
6: Communication and delivery	Whether the change description is accurate, whether uncertainties are flagged, whether the patch is clean	Manual review checklist

Three principles apply when designing the tasks:

1. Each task should primarily test one dimension, with the others treated as “incidental observations”; otherwise results cannot be attributed.
2. Acceptance must be executable: wherever tests, scripts, or diff comparisons can decide the outcome, do not rely on subjective impressions.
3. Leave room for trap items: for example, deliberately supply an API usage that looks reasonable but is actually outdated, and see whether the model copies it or points out the problem. This corresponds to the model’s ability to “not be overconfident,” which is highly valuable in real development.

5. Process Metrics and Outcome Metrics, Tracked Together

Looking only at “how many tasks were solved correctly” is far from enough. The real cost of a coding task hides in the process. The following metrics are worth recording at the same time:

Metric	Definition	Purpose
First-pass rate	Tasks that pass acceptance on the first submission ÷ total tasks	Measures the share that is “directly usable”
In-time completion rate	Tasks with a usable result inside the time limit ÷ total tasks	Reflects stability and convergence speed
Average effective time	Per-task durations averaged with difficulty weights	Reflects the real rhythm better than total duration
Tool call rounds	Count of actions such as builds, tests, and file reads	Too many signals poor localization efficiency; too few may mean no self-checking at all
Output redundancy	Output size ÷ the minimum change size required to pass	Detects over-modification and opportunistic refactoring
Manual rework	Lines changed or time spent during human review	Maps directly to hidden cost
Constraint violations	Count of violations such as network access or unauthorized edits	A safety floor; a single occurrence needs attention

Output redundancy and constraint violations are the two most easily overlooked. A model that passes all tests inside the limit but casually touches a dozen unrelated files may impose a review cost after merging that exceeds the time it saved.

6. Cost-Effectiveness: Count Three Kinds of Cost, Then Segment by Difficulty

“Cost-effectiveness” is easy to misjudge because many comparisons only count unit price. When comparing two models, at least three kinds of cost must be accounted for:

1. Model call cost: input and output are priced separately, and for long-context tasks the input share is often larger; retries and failed attempts also consume quota.
2. Human cost: engineer time spent on review, fixes, and missing tests, usually the largest hidden expense.
3. Rework and risk cost: production incidents from a bad merge, rollback and postmortem effort, and the security risk of unauthorized operations.

This yields three measurement definitions:

Average cost per task    = call cost ÷ total tasks
Cost per accepted delivery = total cost ÷ tasks that pass acceptance
True total cost of ownership = call cost + manual rework hours × labor rate + rework risk adjustment

Consider an example on the same set of tasks:

• Model A has a lower unit price, with call cost per task about one third of Model B’s, but its first-pass rate is only about half.
• Model B has a higher unit price, and when it fails, Model A is used as the fallback.

On the failed tasks, the total spend of “B with A as fallback” is roughly 1.4 times “B answering alone,” yet it is usually lower than “A plus a large amount of manual rework.” The example shows that cost-effectiveness is only comparable when “cost per accepted delivery” is calculated separately for each model on the same task set and under the same acceptance criteria. The numbers below are illustrative only; real selection decisions should be based on measurements against your own task set.

Model tier	Call cost per task (illustrative)	First-pass rate (illustrative)	Cost per accepted delivery (illustrative)
Lightweight model	$1	50%	$2.00
Large model	$3	90%	~$3.33

A more practical strategy is difficulty-based routing: let the lightweight model answer quickly and self-check first; if acceptance passes, deliver directly, and escalate to the large model only for tasks that fail. Calibrate the routing thresholds with a small-scale evaluation on a few hundred historical tasks before going live:

import math
PRICE_IN = 0.004    # Input price per 1k tokens, example value
PRICE_OUT = 0.02    # Output price per 1k tokens, example value
FAST_MAX_TOKENS = 120_000     # Context limit the lightweight model can handle, example value
FAST_PASS_RATE = 0.50         # First-pass rate of the lightweight model, from the small-scale evaluation
BIG_PRICE_FACTOR = 3.0        # Price multiple of the large model, from the small-scale evaluation
def route(task_tokens: int, risk_level: str) -> str:
    """Pick a model tier based on task size and risk level."""
    if task_tokens > FAST_MAX_TOKENS:
        return "large"          # Beyond the lightweight model's context capacity, escalate immediately
    if risk_level == "high":
        return "large"          # High-risk tasks do not rely on a fallback; pick the large model on the first try
    return "small"              # Everything else is answered by the lightweight model first
def expected_cost(task_tokens: int, risk_level: str,
                  fast_pass=FAST_PASS_RATE) -> float:
    """Estimate expected call cost per task: failed tasks are covered by the large model."""
    t = task_tokens / 1000
    c_small = t * PRICE_IN + t * PRICE_OUT
    c_large = c_small * BIG_PRICE_FACTOR
    if route(task_tokens, risk_level) == "large":
        return c_large
    return c_small + (1 - fast_pass) * c_large
if __name__ == "__main__":
    assert route(10_000, "high") == "large"       # High-risk task escalates immediately
    assert route(500_000, "low") == "large"       # Extra-long context escalates immediately
    assert route(10_000, "low") == "small"        # Normal task is answered by the lightweight model first
    # Example: for a ~100k-token task with a 50% first-pass rate,
    # routing costs about 0.83x what the large model alone would cost
    cost = expected_cost(100_000, "low")
    c_large_alone = (100_000 / 1000) * (PRICE_IN + PRICE_OUT) * BIG_PRICE_FACTOR
    assert math.isclose(cost / c_large_alone, 0.833, abs_tol=1e-3)
    report = [(f"Task {i + 1}", route(n, "low"), round(expected_cost(n, "low"), 5))
              for i, n in enumerate([8_000, 60_000, 150_000, 300_000])]
    for row in report:
        print(row)
    # Values depend on the prices and rates plugged in above and are illustrative only.

These thresholds are only a starting point. Whenever a new batch of tasks goes live, review how the routing rules actually fired and adjust the parameters accordingly.

7. A Reusable Evaluation Harness and Metric Calculation

Below is a skeleton for an evaluation harness. The core idea is “a fresh environment per task, a uniform time limit, uniform scoring, and every item written to disk.” The commands and judging interfaces are given as generic placeholders; replace them with whatever tools your team actually uses.

"""Time-boxed multi-model evaluation harness (example code; replace the placeholder interfaces with your team's tooling)"""
import hashlib
import json
import subprocess
import time
from dataclasses import dataclass, field, asdict
from pathlib import Path
TIME_LIMIT_SEC = 30 * 60   # Per-task limit of 30 minutes
ALLOW_NETWORK = False      # No network for the entire run
@dataclass
class RunConfig:
    model_id: str           # Model identifier comes from the config service, never from the model's self-description
    model_tier: str         # small / medium / large, for grouped statistics
    task_set: str           # Task set version; any change counts as a new task set
@dataclass
class Case:
    case_id: str
    prompt: str
    difficulty: str         # easy / medium / hard, for weighted statistics
    check_cmd: list         # Judging command; exit code 0 means pass
@dataclass
class Attempt:
    case_id: str
    passed: bool = False
    timed_out: bool = False
    network_violation: bool = False
    wall_sec: float = 0.0
    tokens_in: int = 0
    tokens_out: int = 0
    tool_calls: int = 0
    agent_exit_code: int = -1
    check_output: str = ""
def _digest(obj) -> str:
    """Digest the content relevant to judging so records stay reproducible and searchable."""
    raw = json.dumps(obj, ensure_ascii=False, sort_keys=True).encode()
    return hashlib.sha256(raw).hexdigest()[:12]
def run_case(cfg: RunConfig, case: Case, workdir: Path,
             runner_cmd: list) -> Attempt:
    """Run one task in a fresh working directory; timeouts and violations are recorded as failures."""
    workdir.mkdir(parents=True, exist_ok=True)
    rec = Attempt(case_id=case.case_id)
    t0 = time.monotonic()
    proc = subprocess.run(
        runner_cmd + ["--model", cfg.model_id, "--prompt", case.prompt],
        cwd=workdir, capture_output=True, text=True,
        timeout=TIME_LIMIT_SEC,
        # No network: drop the network namespace here; with a container sandbox, configure it at the container layer.
        # See your runtime's documentation for the exact flags.
        preexec_fn=lambda: __import__("resource").setrlimit(
            __import__("resource").RLIMIT_NOFILE, (256, 256)),
    )
    rec.wall_sec = round(time.monotonic() - t0, 1)
    rec.agent_exit_code = proc.returncode
    usage = json.loads(Path(workdir / "usage.json").read_text())
    rec.tokens_in = usage["input_tokens"]
    rec.tokens_out = usage["output_tokens"]
    rec.tool_calls = usage.get("tool_calls", 0)
    check = subprocess.run(case.check_cmd, cwd=workdir,
                           capture_output=True, text=True)
    rec.check_output = check.stdout[-2000:] + check.stderr[-2000:]
    rec.passed = check.returncode == 0 and proc.returncode == 0
    # Cheating detection: if the changed-file list contains the tests themselves, fail outright
    changed = set(Path(workdir / "changed_files.txt").read_text().split())
    if changed & {"tests", "test_cases"}:
        rec.passed = False
    return rec
def aggregate(records: list) -> dict:
    """Aggregate process and outcome metrics, with duration distribution bucketed by difficulty."""
    total = len(records)
    passed = [r for r in records if r.passed]
    by_diff = {}
    for diff in ("easy", "medium", "hard"):
        times = [r.wall_sec for r in records
                 if r.case_id.startswith(diff) and r.passed]
        if times:
            by_diff[diff] = {
                "n": len(times),
                "p50": sorted(times)[len(times) // 2],
                "max": max(times),
            }
    return {
        "task_set": _digest([r.case_id for r in records]),
        "pass_rate": round(len(passed) / total, 3),
        "avg_tool_calls": round(
            sum(r.tool_calls for r in records) / total, 1),
        "token_total": sum(r.tokens_in + r.tokens_out for r in records),
        "network_violations": sum(r.network_violation for r in records),
        "time_by_difficulty": by_diff,
    }
if __name__ == "__main__":
    cases = [
        Case("easy-01", "Add unit tests for utils/parser.py", "easy",
             ["bash", "run_tests.sh", "tests/test_parser.py"]),
        Case("medium-02", "Fix the race condition in the concurrent path", "medium",
             ["bash", "run_tests.sh", "tests/test_concurrency.py"]),
        Case("hard-03", "Complete a dependency migration with no network and add tests", "hard",
             ["bash", "run_checks.sh"]),
    ]
    runner = ["python", "agent_cli.py", "--no-network", "--json-usage"]
    cfg = RunConfig(model_id="model-under-test", model_tier="medium",
                    task_set="v1.0")
    root = Path("runs") / cfg.task_set / cfg.model_id
    records = []
    for case in cases:
        rec = run_case(cfg, case, root / case.case_id, runner)
        records.append(rec)
        (root / case.case_id / "result.json").write_text(
            json.dumps(asdict(rec), ensure_ascii=False, indent=2))
    print(json.dumps(aggregate(records), ensure_ascii=False, indent=2))

The accompanying judging script is worth maintaining separately as well. Its job is not to “reimplement the solution,” but to verify externally observable behavior and to check the hard constraints:

"""Example judging script: verify external behavior only, never bind to a specific implementation"""
import os
import subprocess
NETWORK_HOSTS = {"pypi.org", "registry.example.com"}  # Forbidden targets in a no-network environment
def run(cmd, cwd="."):
    return subprocess.run(cmd, cwd=cwd, capture_output=True, text=True)
def assert_build_ok():
    res = run(["python", "-m", "compileall", "src"])
    assert res.returncode == 0, f"Build failed:\n{res.stderr[-800:]}"
def assert_tests_pass(test_path="tests"):
    res = run(["python", "-m", "unittest", "discover", "-s", test_path])
    assert res.returncode == 0, f"Tests failed:\n{res.stderr[-800:]}"
def assert_no_network_access(workdir="."):
    for name in os.listdir(workdir):
        if name.endswith((".py", ".sh", ".yaml")):
            text = open(os.path.join(workdir, name), encoding="utf-8").read()
            for host in NETWORK_HOSTS:
                assert host not in text, f"Network access detected: {host}"
def main():
    assert_no_network_access()
    assert_build_ok()
    assert_tests_pass()
    print("CHECK PASSED")
if __name__ == "__main__":
    main()

Two engineering conventions make the records far more valuable:

• Archive results by content digest: hash “model identifier + task set version + judging result” and write it into the report. When a model is silently upgraded or the task set is changed, the digest changes with it, so historical results stay traceable.
• Any violation voids the whole round: if a model is found to break a hard constraint, such as accessing the network or deleting test files, that round’s results should not enter the comparison at all.

8. From a One-Off Test to a Pipeline: Keep the Evaluation Results Working

A one-off six-task evaluation is only a starting point for selection. The real value comes from turning it into a continuously running regression asset:

1. Freeze the task set: every task carries a version number, a judging script, and an expected duration range; changes to the task set go through a review process.
2. Wire it into CI: whenever the model or the host environment is upgraded, run the core task set again. If the pass rate drops beyond a set threshold, raise an alert and route it into the team’s on-call rotation through the monitoring and alerting stack.
3. Keep everything: archive full conversations, tool calls, patches, and judging logs to object storage, indexed by task set version and model identifier for easy retrieval.
4. Recalibrate continuously: periodically fold sampled results from real production tasks into the task set, so evaluation items do not drift away from business needs.
5. Enforce safety at the host layer: tool-call permissions should follow least privilege, and network access, writable file scope, and a command execution allowlist must be enforced in the host environment, not left to a single line of “no internet access” in the system prompt.

9. Conclusion

The value of this six-task evaluation is not that it produced an answer to “who is stronger,” but that it demonstrates a reusable evaluation paradigm:

• Define the exam room before starting the clock. Model identifier, runtime environment, time window, network policy, time limit, and acceptance criteria should all be fixed in advance.
• Stay skeptical about what models say about themselves. Version numbers, cutoff dates, and similar information should be treated as to-be-verified, and any factual claim has to be checked against an authoritative source.
• Explain outcome metrics with process metrics. Tool call rounds, duration distribution, and output redundancy often explain cost differences better than “how many tasks were solved.”
• Do the math on cost per accepted delivery. Compare call, human, and rework costs together, then route models by difficulty.

Once this method is baked into the pipeline as a regression asset, the team can face every model update with data rather than impressions.