Why Leaderboard Scores Alone Are Unreliable

Public leaderboards are valuable references, but using them directly for model selection decisions comes with several common pitfalls.

1. Scores are highly sensitive to evaluation conditions. For the same model, differences in the task subset, sampling temperature, maximum output length, retry count, or system prompt can shift scores by several percentage points. Two scores measured under different conditions cannot be compared directly.

2. Data may change or become impossible to verify. Leaderboard pages can be updated, and raw data may be modified or removed. Without preserved original records, there is no way to trace a conclusion back to its source later.

3. Ranking gaps may fall within the margin of error. When two models’ scores differ by a tiny margin and the number of test cases is limited, the gap may simply be random noise. Treating it as evidence that “one model is stronger” is not a sound conclusion.

4. Performance metrics are incomplete. Leaderboards typically report accuracy only, while engineering decisions also require latency, unit cost, stability, and failure modes. Optimizing a single dimension can easily lead to a solution that “scores well but is too expensive to use.”

The more reliable approach: re-evaluate candidate models on task samples from your own business under a fixed methodology, and retain the complete raw data.

Building a Reproducible Evaluation Methodology

Whether evaluation results can be reproduced depends on whether the methodology is fixed. Freeze your evaluation setup in four areas.

1. Fix the test set and stratify it by task. The test set should cover the main task types in your real business, such as code generation, bug localization and fixing, code review, requirements understanding, and documentation Q&A. Retain a reasonable number of cases per task type, labeled with difficulty and reference answers or scoring rules. Once the test set is finalized, do not add or remove cases arbitrarily; version it whenever updates are needed.

2. Fix the invocation parameters. Temperature, maximum output length, stop tokens, system prompts, and tool-calling configuration should all be written to a configuration file and saved alongside the results. If the model service supports a fixed random seed, enable it.

3. Record environment information. Each result should include the model identifier, reasoning level, invocation time, request region, and evaluation script version. Only then can you trace the origin of a conclusion six months later.

4. Save raw data, not just summaries. Raw requests, raw responses, scoring results, and scoring rationale should all be persisted. Summary tables can always be recomputed from raw data — not the other way around.

Below is a generic evaluation script framework. The model-calling and scoring functions are implemented by the user according to their own integration; the script itself does not depend on any specific vendor’s API.

import json
import time
from dataclasses import dataclass, asdict
@dataclass
class EvalRecord:
    case_id: str
    model_id: str
    reasoning_level: str
    passed: bool
    latency_ms: int
    input_tokens: int
    output_tokens: int
    cost_usd: float
def run_suite(cases, models, call_model, judge, price_table, out_path):
    """
    cases:       list of test cases, each with an id, a prompt, and reference
                 info needed for scoring
    models:      list of models to evaluate, each with an id and a list of
                 reasoning levels
    call_model:  user-implemented invocation function, returns text,
                 input_tokens, output_tokens
    judge:       user-implemented scoring function, returns True or False
    price_table: unit prices indexed by (model, level), in USD per million tokens
    out_path:    output path for raw results (JSON Lines format)
    """
    with open(out_path, "w", encoding="utf-8") as f:
        for model in models:
            for level in model["levels"]:
                for case in cases:
                    start = time.time()
                    resp = call_model(model["id"], level, case["prompt"])
                    latency_ms = int((time.time() - start) * 1000)
                    passed = judge(case, resp["text"])
                    price = price_table[(model["id"], level)]
                    cost = (
                        resp["input_tokens"] * price["input"]
                        + resp["output_tokens"] * price["output"]
                    ) / 1_000_000
                    record = EvalRecord(
                        case_id=case["id"],
                        model_id=model["id"],
                        reasoning_level=level,
                        passed=passed,
                        latency_ms=latency_ms,
                        input_tokens=resp["input_tokens"],
                        output_tokens=resp["output_tokens"],
                        cost_usd=round(cost, 6),
                    )
                    f.write(json.dumps(asdict(record), ensure_ascii=False) + "\n")

Each record is written as one line of JSON; subsequent statistics, recomputation, and auditing can all be performed directly against this file.

Reasoning Levels and Cost Conversion

Many models offer multiple reasoning-effort levels. Higher levels usually mean more internal reasoning tokens: accuracy may improve, but cost and latency grow as well. Comparing two models only at their highest levels therefore often leads to skewed conclusions.

A more meaningful metric is cost per passing case:

Cost per passing case = total evaluation cost ÷ number of passing cases

It answers the question: how much does each correct result cost?

Here is a set of fictional illustrative data, used only to demonstrate the calculation and not representing any real model:

Option	Pass rate	Total cost (USD)	Cost per passing case (USD)
Example A, high level	92%	360	0.078
Example B, medium level	89%	140	0.031
Example C, low level	84%	45	0.011

As the table shows, Example A has the highest pass rate, but its cost per passing case is roughly seven times that of Example C. If the business scenario tolerates some failure rate and human review is in place, a lower reasoning level may be the better choice.

A further step is to plot a quality-cost Pareto curve, with unit cost on the horizontal axis and pass rate on the vertical. Keep only the points that are “highest quality at equal cost” or “lowest cost at equal quality.” Any option below the curve can be eliminated.

Using Statistics to Determine Whether Differences Are Real

With limited evaluation samples, any gap between two scores may contain random variation. Determining whether a difference is real requires interval estimation.

For binomial proportions such as pass rates, the Wilson interval is more robust than a simple normal approximation when sample sizes are small. The following function computes the confidence interval given the number of passing cases and the total:

import math
def wilson_interval(passed: int, total: int, z: float = 1.96):
    """Returns the 95% Wilson confidence interval for the pass rate (z=1.96)."""
    if total == 0:
        return (0.0, 0.0)
    p = passed / total
    denom = 1 + z ** 2 / total
    center = (p + z ** 2 / (2 * total)) / denom
    margin = z * math.sqrt(p * (1 - p) / total + z ** 2 / (4 * total ** 2)) / denom
    return center - margin, center + margin

Usage guidelines:

1. If two options’ confidence intervals do not overlap at all, the difference can be considered reasonably reliable.
2. If the intervals overlap substantially, treat it as “not yet distinguishable” and increase the number of test cases or repeat the runs before drawing conclusions.
3. For non-deterministic outputs, run each case at least 3 to 5 times, record the pass/fail outcome for each run, and observe the range of variation.

How to Handle Unofficial Information

During model development and testing, screenshots of unknown origin, unverified test results, or data from temporarily taken-down pages often circulate. Such information can serve as a lead worth watching, but should not be used directly as the basis for technical decisions.

Recommended principles:

• Label credibility by tier. Official documentation and publicly verifiable data count as high credibility; single-source screenshots count as low credibility; rumors alone are excluded from decision-making.
• Replace external rumors with your own testing. As soon as a new version is available, run it against your own test set first, and base conclusions on internal data.
• Avoid drawing conclusions from a single result. A model’s performance on a “classic question” may stem from similar samples in its training data and does not represent overall capability. If you use such questions, prepare variants as well to check whether the model truly understands the problem.

Staged Rollout and Continuous Monitoring

Even after passing evaluation, avoid switching all traffic at once. A safer rollout path has four stages:

1. Offline evaluation. Complete all metric measurements on the fixed test set and produce an evaluation report.
2. Shadow traffic. Mirror real production requests to the new option, recording results only, with no user impact. Compare output quality and latency between old and new.
3. Small-scale canary. Switch a small fraction of real traffic to the new option with explicit rollback thresholds — for example, automatically reverting if the success rate drops beyond a defined proportion.
4. Full rollout. Once metrics are confirmed stable, gradually expand the proportion to 100%.

After launch, continuously monitor:

• Request success rate and error-type distribution
• Latency percentiles for first token and total response, such as P50 and P95
• Cost per task, tallied daily or weekly
• Human rejection or rework rates, reflecting real-world usability of outputs
• Sampled output-quality audits, cross-checked against offline evaluation conclusions

When production metrics diverge noticeably from offline evaluation results, first check whether the test set covers the real business distribution, rather than immediately blaming the model itself.

Selection Checklist

• Does the test set cover the main business tasks, with versioning?
• Are invocation parameters, reasoning levels, and environment information fully recorded?
• Is raw data saved, and can summary results be recomputed from it?
• Have comparison conclusions been validated with confidence intervals?
• Were quality, latency, and cost per passing case all compared?
• Has external information been tiered by credibility and kept out of direct decision-making?
• Does the rollout plan include shadow traffic, canary release, and rollback mechanisms?
• Do production monitoring metrics cover success rate, latency, cost, and human feedback?