LLM Capability Verification: Canary Testing and Reproducible Evaluation

    This article explains why many claims about large language model capabilities are difficult to verify and how engineering teams can build reproducible evaluation workflows. It covers common pitfalls, capability dimensions, evaluation set construction, fixed experiment configurations, statistical reporting with confidence intervals, canary testing considerations, common causes of non-reproducible results, a minimal evaluation script, and a team rollout checklist.

    1. Why Conclusions Often Fail to Hold

    Industry discussions about model capabilities often rely on a few types of material: official demos, screenshots shared by a small number of users, test samples circulating in the community, and media retellings. These materials share a common problem: small samples, opaque runtime environments, and missing version information. When a conclusion is supported by only a few examples and a second person cannot reproduce those examples, its credibility is limited.

    Reproducibility is a basic threshold for judging whether a technical conclusion holds. Engineering teams evaluating their own models face the same issue: a test case that passed yesterday fails today, changing the system prompt produces a different answer, and online feedback does not match offline metrics. The solution is not to collect more screenshots, but to establish an evaluation process that can be run repeatedly, keeps records, and supports statistical analysis.

    2. Common Pitfalls

    1. Using a single example in place of overall capability. One impressive response may be just one sample from many runs; a single result cannot represent the output distribution.
    2. Ignoring experimental conditions. Model version, decoding temperature, maximum output length, system prompt, and tool availability can all change the result.
    3. Contaminated evaluation sets. If test questions have already appeared in training or fine-tuning data, scores will be inflated and will not reflect generalization ability.
    4. Treating subjective impressions as conclusions. “Feels smarter” cannot be compared; it must be converted into quantifiable decision rules.
    5. Ignoring server-side changes. Behind the same API name, the model version, routing policy, and safety filtering rules may change over time.

    3. Building a Reproducible Evaluation Framework

    3.1 Define Capability Dimensions First

    Before evaluation, break “capability” into judgeable subdimensions, for example:

    • Facts and reasoning: whether a correct conclusion can be reached given the source material;
    • Code generation: the proportion of generated code that passes unit tests;
    • Instruction following: whether the output format, fields, and length meet requirements;
    • Long context: whether key information can be located and cited in long documents;
    • Safety and refusal: whether handling of violating requests conforms to the established policy.

    For tasks with determinable correctness, use rule-based judgment. For tasks that are hard to judge by rules, have multiple annotators score independently and compute inter-annotator agreement to confirm that the scoring standard itself is stable.

    3.2 Build Evaluation Sets

    The quality of the evaluation set determines the upper bound of the conclusions. The following practices are recommended:

    • Stratified sampling: stratify by difficulty, domain, and input length, avoiding samples that all come from the same type of question;
    • Keep a hidden set: use the public set for debugging and the hidden set only before release, to prevent targeted optimization;
    • Deduplication and contamination detection: compare the similarity between evaluation questions and the training corpus, and remove near-duplicate samples;
    • Version management: generate a new version number for every modification to the evaluation set and document the changes.

    3.3 Fix Experimental Conditions

    Every experiment should save a configuration snapshot that includes at least: model identifier and version, decoding parameters, random seed, hash of the system prompt, context length limit, available tool list, and execution date. Hash the snapshot contents; the resulting fingerprint can be used to determine whether two results are comparable. Only results with the same configuration fingerprint can be compared directly.

    3.4 Use Statistics Instead of Impressions

    Sample the same question multiple times and record the pass rate. The confidence interval of the pass rate can be estimated with the normal approximation:

    p ± 1.96 × √(p(1−p)/n)

    For small sample sizes, the normal approximation has larger error, so the Wilson interval is recommended. For example, with a sample size of 100 and a pass rate of 0.80, the standard error is about 0.04, and the 95% interval is about [0.72, 0.88]. When the intervals of two results overlap substantially, you cannot conclude that one side is stronger; you can only say that the difference is within the error range.

    4. Focus Areas in Canary Testing

    The goal of canary testing is to observe performance under real traffic within a controlled scope, not to produce a final ranking. Consider the following when designing it:

    • Traffic split: start with a small percentage, observe error rate, latency, and user feedback, and be able to roll back quickly when anomalies appear;
    • User segmentation: task distributions differ greatly across user groups, so metrics should be split and reviewed by group;
    • Sample retention: save the complete input, output, configuration snapshot, and timestamp. If someone reports a sample, the team should be able to reproduce it under the same configuration or explain why it cannot be reproduced;
    • Compliance boundaries: samples involving user data must be de-identified before retention and sharing.

    5. Common Causes of “Non-Reproducible” Results

    When a result that appeared before cannot be obtained again, there is usually more than one cause:

    1. Version iteration: the backend has been updated, and the old version is no longer served;
    2. Configuration differences: temperature, system prompt, or tool invocation settings differ from the original experiment;
    3. Sampling randomness: a single result contains chance, and repeated runs are needed to see the distribution;
    4. Filtering rule changes: safety policies or output filtering may change the content that is presented;
    5. Sample distortion: screenshots may be cropped or edited and lack full context.

    During troubleshooting, check the experiment snapshot item by item in order; do not jump to conclusions. Being able to state “under the same configuration, what is the pass rate over repeated runs” is more valuable than any speculation.

    6. Minimal Evaluation Script Example

    The example below uses only the standard library and shows how to fix the configuration, repeat sampling, and calculate the pass rate and confidence interval. model_call is an inference function wrapped by the caller and must be replaced with the actual service.

    1. import hashlib
    2. import json
    3. import math
    4. from dataclasses import dataclass
    5. @dataclass
    6. class EvalCase:
    7. case_id: str
    8. prompt: str
    9. must_contain: str
    10. def config_fingerprint(config: dict) -> str:
    11. raw = json.dumps(config, sort_keys=True, ensure_ascii=False)
    12. return hashlib.sha256(raw.encode("utf-8")).hexdigest()[:16]
    13. def wilson_interval(passed: int, total: int, z: float = 1.96):
    14. if total == 0:
    15. return 0.0, 0.0
    16. p = passed / total
    17. denom = 1 + z * z / total
    18. center = (p + z * z / (2 * total)) / denom
    19. half = z * math.sqrt(p * (1 - p) / total + z * z / (4 * total * total)) / denom
    20. return max(0.0, center - half), min(1.0, center + half)
    21. def run_eval(model_call, cases, config, repeats=5):
    22. fingerprint = config_fingerprint(config)
    23. records = []
    24. for case in cases:
    25. for i in range(repeats):
    26. output = model_call(case.prompt, **config)
    27. records.append({
    28. "case_id": case.case_id,
    29. "run": i,
    30. "passed": case.must_contain in output,
    31. "output": output,
    32. })
    33. passed = sum(r["passed"] for r in records)
    34. low, high = wilson_interval(passed, len(records))
    35. report = {
    36. "fingerprint": fingerprint,
    37. "config": config,
    38. "total": len(records),
    39. "passed": passed,
    40. "pass_rate": round(passed / len(records), 4),
    41. "ci95": [round(low, 4), round(high, 4)],
    42. }
    43. return report, records

    After running, append report and records to a log file. This makes the configuration, raw outputs, and statistical results of every experiment traceable, so later comparisons do not depend on memory.

    7. Team Rollout Checklist

    • Every experiment has a configuration fingerprint saved together with the report;
    • The evaluation set has a version number, and the hidden set has not been used for tuning;
    • Reports include sample size and confidence intervals, not just a single number;
    • External conclusions include reproduction conditions: version, configuration, and number of repeats;
    • Canary samples retain complete raw records for later review;
    • State only verified content; unverified rumors should not be presented as findings.