LLM Benchmarking: From Leaderboard Guesswork to Reproducible Engineering

    Single-run LLM benchmarks rarely hold up: inconsistent version definitions and non-reproducible processes can swing scores by double digits, so leaderboard claims about which model is "stronger" often amount to mistaking noise for signal. This article explains why reasoning and agentic capabilities must be measured separately, breaks down the five main sources of evaluation noise, and argues that capability curves matter more than parameter scale. It then lays out a practical methodology for reproducible controlled experiments—five elements to fix, paired designs with significance testing, a minimal runnable evaluation skeleton, and additional design considerations for agentic scenarios—before turning to enterprise selection, private regression sets, tiered routing, and canary releases. The goal is to replace "who wins" with "how to measure the difference and decide whether it is worth the cost."

    1. From “Guesswork” to “Verifiable”: Why Single-Run Conclusions Are Untrustworthy

    Model versions iterate far faster than evaluation systems mature. After a new release, what circulates in the community is usually some variation of “a third-party tester ran one round, and the conclusion is that the new model is stronger at reasoning and agentic tasks.” This kind of information carries two risks:

    • Inconsistent version definitions: Differences in minor-version suffixes, quantization precision, and context window configuration can all significantly change measured performance, yet these are often omitted in discussion.
    • Non-reproducible evaluation process: Prompts, parsers, sampling parameters, tool sets, and grading criteria—any one of these changing can shift scores by a dozen or more percentage points.

    So the more sensible way to ask the question is not “can it beat” but: under the same task set, the same tool set, the same budget, and the same grading criteria, is the difference between the two models statistically significant, and is that difference worth the extra cost in my business scenario?

    2. Break Down “Capability” First: Reasoning and Agentic Measure Different Things

    2.1 Reasoning: The Accuracy Distribution of Single-Turn Output

    Reasoning evaluation typically focuses on accuracy under single-turn or multi-turn conditions where “enough information is provided and no external action is required.” Typical formats include mathematical proofs and computation, code generation and defect fixing, multi-hop knowledge Q&A, and constraint satisfaction and logic puzzles. Its core variable is test-time compute—allowing the model to generate longer chains of thought, or to sample multiple times and then vote, often lifts accuracy on hard problems more effectively than simply increasing parameters.

    This means two models may be close under “single-sample pass@1” yet pull clearly apart under “multi-sample + majority voting.” When evaluating, you must state which definition you are using.

    2.2 Agentic: The Success Rate of Long-Horizon Trajectories

    Agentic evaluation examines the quality of a model’s behavior under conditions with an environment, tools, and state. The measurement object shifts from “is the answer correct” to “is the whole trajectory correct”:

    • Planning quality: whether it can decompose a goal into executable subtasks and proactively explore when information is insufficient;
    • Tool use: whether call arguments are correct, whether it reads and understands the returned results, and whether it avoids ineffective repeated calls;
    • Error recovery: whether it can self-correct after tool errors, timeouts, or empty results rather than getting stuck in a loop;
    • Context management: whether key constraints are preserved in long tasks, and whether goal drift occurs;
    • Cost awareness: whether it converges within budget, and whether step count and token consumption stay controllable.

    A common mistake is inferring agentic performance from reasoning leaderboard results. The two correlate only weakly: a model that is strong at reasoning but has unstable tool-call formatting and poor recovery may achieve end-to-end task success rates well below expectations. Conversely, a model with mediocre reasoning but standardized tool calls and robust recovery strategies is often more practical in real workflows.

    Dimension Reasoning Agentic
    What is measured Correctness of single/multi-turn answers Result of the complete task trajectory
    Key variables Test-time compute, sampling strategy Tool set, environment feedback, budget cap
    Main metrics pass@1, pass@k, majority-vote accuracy Task success rate, steps, recovery rate, per-task cost
    Failure mode Wrong computation, hallucination Infinite loops, goal drift, ineffective calls
    Evaluation difficulty Medium (grading criteria relatively clear) High (environment must be reproducible)

    3. Why a Single Run Is Not Reproducible: Five Noise Sources

    1. Prompt and parser differences: Whether JSON output can be parsed stably, and whether code-block extraction includes language tags, directly determines whether a “formatting failure” is counted as a wrong answer or a correct result.
    2. Sampling parameters: Temperature, top-p, maximum generation length, and whether multiple resamples are taken all change the score range of the same model.
    3. Evaluation-set contamination: If public problems have largely entered the training corpus, scores reflect memorization rather than generalization. Mitigations include building private problem sets, generating problems dynamically, and rotating them periodically.
    4. Runtime environment differences: Sandbox timeout thresholds, dependency versions, and network availability can all make agentic evaluation results incomparable.
    5. Inconsistent metric definitions: pass@1, pass@k, majority voting, and average reward cannot be compared directly.

    The conclusion is straightforward: comparison without a unified definition is essentially treating noise as signal.

    4. More Reliable Than “Parameter Scale” Is the Capability Curve

    Scaling laws describe how training loss decreases as a power law with increases in parameter count, data volume, and compute. They explain a trend in the training phase and cannot directly infer the winning relationship on a specific downstream task. Likewise, parameter scale is only a profile of resource investment; what really determines the experience is the training data mix, post-training strategy (instruction tuning, preference alignment), the test-time compute budget, and systems-engineering optimization.

    So the evaluation dimensions should be three-dimensional: capability (accuracy/success rate)—cost (tokens and compute per task)—latency (time to first token, end-to-end time). A model that is 5 percentage points stronger in capability but 3x more expensive and 2x slower—whether it is “better” depends entirely on whether the business is sensitive to latency and cost.

    5. Evaluation Design: A Reproducible Controlled-Experiment Method

    5.1 Five Elements That Must Be Fixed

    • Fix the task set and version number, and archive the task set;
    • Fix sampling parameters and maximum budget (steps, tokens, wall-clock time);
    • Fix the tool set and tool description text;
    • Fix the grader, and manually spot-check the grader itself;
    • Fix the number of repetitions, and report confidence intervals rather than single-point values.

    5.2 Paired Design and Significance Testing

    Run candidate models separately on the same batch of problems, in the same tool environment, and with the same budget; compute per-problem differences, then apply a paired test to the distribution of differences (for example, a sign test or paired bootstrap). Only this way can you answer “is the improvement robust” rather than being led by single-problem fluctuations.

    5.3 A Minimal Runnable Evaluation Skeleton

    1. import math, random
    2. def wilson_interval(k: int, n: int, z: float = 1.96):
    3. """Wilson confidence interval for a binomial proportion; more robust than the normal approximation for small samples"""
    4. p = k / n
    5. d = 1 + z * z / n
    6. c = (p + z * z / (2 * n)) / d
    7. h = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    8. return round(max(0.0, c - h), 4), round(min(1.0, c + h), 4)
    9. def evaluate(model_call, cases, n_repeat: int = 5, temperature: float = 0.7):
    10. """model_call(prompt) -> str; cases is [{id, prompt, grader}]"""
    11. per_case = {}
    12. for case in cases:
    13. passed = 0
    14. for _ in range(n_repeat):
    15. raw = model_call(case["prompt"], temperature=temperature)
    16. if case["grader"](raw):
    17. passed += 1
    18. per_case[case["id"]] = passed / n_repeat
    19. k = sum(round(v * n_repeat) for v in per_case.values())
    20. n = len(cases) * n_repeat
    21. return {"pass_at_1": round(k / n, 4), "ci95": wilson_interval(k, n),
    22. "per_case": per_case}

    Using this skeleton, repeat the same experiment at least 3 times and report intervals; if two candidate models’ confidence intervals overlap substantially, it should be judged as “no significant difference yet” rather than claiming “already surpassed.”

    5.4 Additional Design for Agentic Scenarios

    • Environment reproducibility: fixed container images, locked dependencies, a clearly defined network policy, and fixed random seeds;
    • Error injection: return tool timeouts, invalid arguments, and empty results at a certain rate to test recovery;
    • Trajectory retention: fully save the input and output of every step to make failure attribution easier;
    • Multi-dimensional metrics: report task success rate, average steps, ineffective-call ratio, recovery success rate, and per-task cost all at once.

    6. From “Who Wins” to “How to Choose”: Enterprise Selection and Canary Process

    1. Layered evaluation by scenario: build separate problem sets and separate conclusions for short Q&A, long-form generation, structured extraction, code assistance, and multi-step process automation.
    2. Build a private regression set: distill real online failure samples (after desensitization) into regression cases; gate new releases through the regression suite first, to avoid “version upgrades causing experience regressions.”
    3. Tiered routing and cost control: route simple requests to lightweight models and escalate complex requests to stronger models, forming dynamic capability–cost scheduling; this pattern is a general industry engineering practice and does not depend on any specific vendor’s capability.
    4. Shadow traffic and canary release: route shadow traffic to the new model first for comparative statistics, then ramp up with a small ratio in canary, observe the success rate, latency, and cost curves, and only then move to full traffic.
    5. Carry the engineering foundation with general capabilities: the evaluation pipeline typically runs on a container platform, datasets and trajectories are stored in object storage, events and traces go through a logging service, and dashboards and anomaly triggers are handled by monitoring and alerting. These are standard capability categories on cloud platforms, and enterprises can build this structure themselves on Baidu AI Cloud. The platform does not participate in any conclusion; the right to make selection decisions always stays with the business.

    7. Conclusion

    The ebb and flow of model versions is the norm; a lead of a few weeks in the short term does not mean a capability has stably crossed a generation. Rewriting “guess who is stronger” into “measure differences under the same definition, judge significance, and evaluate business benefit” is what lets technical judgment serve real decisions. An evaluation system is not a one-time engineering effort but should become a long-term asset that evolves with the business: the task set keeps expanding, the grader is calibrated periodically, and the regression gate is baked into the release process. Do these things, and when a new version is released you no longer need to guess—you just run your evaluation pipeline.