Why a Redacted Leaderboard Screenshot Is Hard to Verify

    A coding leaderboard screenshot with redacted model names and scores sparks an online debate over whether open-weight models can match closed-source ones on complex engineering tasks. As the image gets re-saved, doctored, and speculated on, this article dissects how much a single screenshot can really prove. It walks through the arithmetic behind percentage scores, the blind spots of pixel comparison and AI image recognition, what size-tiered model releases reveal, and four hidden variables—data contamination, evaluator flaws, sampling variance, and task difficulty—that make large benchmarks untrustworthy. It closes with a six-step evaluation workflow, model-routing and fallback patterns, and an action checklist for making every benchmark number recomputable.

    Introduction: A Screenshot Sparks a Debate

    Recently, the developer community lit up with a debate over whether open-weight models can handle complex engineering tasks. One developer argued that on complex engineering work, open-weight models have fallen behind closed-source frontier models, and shared traffic data from their own AI coding tool.

    During the discussion, someone posted a screenshot of a coding benchmark. The screenshot showed 16 records, three of which had their model-name prefixes and scores redacted, leaving only the size suffixes—ordered largest to smallest as Pro, Med, and Small. The poster named no one, but from his wording, netizens inferred he was probably quietly testing a certain vendor’s next-generation model.

    After that, the image went through several rounds of “remixing”:

    • Some estimated and reconstructed by eye: Pro scored 59.4%, 1.6 percentage points ahead of Opus 5.5 but still behind the leader.
    • Someone in the comments immediately posted the “real score”: 70.1%, exactly 0.1 percentage points above the leader.
    • Someone typeset “GLM-Name” themselves, applied a Gaussian blur, and compared it pixel-by-pixel with the original, claiming a residual of less than 1/255; others had AI recognize the text in the image and concluded the prefix was “GLM”.
    • In the later-circulating “clear version” of the leaderboard, the prefix became “GLM-Nano”, and the guesses in the comments shifted again to “GLM-Next”.

    The three suffixes got clearer and clearer, but the prefix kept multiplying.

    This article isn’t about the actual capabilities of these specific products. Instead, it uses this incident to unpack a more general question: after a leaderboard screenshot has been repeatedly re-saved, redacted, and speculated on, how strong a conclusion can it actually support? Along the way, it lays out a practical model evaluation methodology.

    1. Align the Math First: Percentages Aren’t Raw Scores

    The original leaderboard had 6 coding tasks with a maximum of 60 points; the leaderboard percentage is just the score converted to a 100-point scale. From this conversion, we can draw the following score reference table (for reference only):

    Model / Tier Leaderboard % Raw score (out of 60)
    Top closed-source model 70.0% 42.0
    Redacted version A (Pro) 59.4% 35.64
    Redacted version B 70.1% 42.06
    A closed-source flagship 57.8% 34.68
    Redacted version (Med) 57.1% 34.26
    Redacted version (Small) 54.0% 32.4
    A previous-gen model 45.8% 27.48

    Developers who have reproduced leaderboards will spot the problem at a glance:

    1. Insufficient interpretability. In a 6-task, 60-point system, one task is worth about 16.7 percentage points. Version A beats the flagship by only 1.6 percentage points, which is 0.96 raw points—less than one task.
    2. “70.1%” is an arithmetic outlier. 0.1 percentage points is 0.06 raw points. 42.06 points is neither a whole number nor expressible as 1/n of a point. If scoring is per-task, this score is arithmetically impossible for this leaderboard to produce. Even under per-rubric scoring, this number at most means “just barely above the leader.”
    3. Some netizens’ interpretations don’t hold up. “Version A is 1.6 percentage points above the flagship, which means it can already replace it for complex tasks” is a classic case of over-interpretation.

    The leaderboard percentage is a form of precision theater. Precise decimals create the illusion of comparability while masking the huge uncertainty behind small-sample, single-metric evaluation. Raw scores and percentages should be recorded separately in the configuration to avoid ambiguity from conversion and rounding. Concretely:

    1. # Example: record both raw score and conversion in the eval config
    2. benchmark:
    3. name: code_bench_v4
    4. num_tasks: 6
    5. max_raw_score: 60
    6. score_unit: task # task: whole-task scoring; rubric: per-item rubric scoring
    7. display: percentage # outward display format

    2. From Blurry to “Clear”: Blind Spots in Pixel Comparison and AI Recognition

    2.1 Information in a Blurry Image Is Irreversibly Lost

    Gaussian blur is a low-pass filter that erases high-frequency detail; yet the differences between letters like N and M, l and I are concentrated exactly in the high-frequency part. When comparing a reconstructed image to the original, there is really only one constraint: whether the reconstruction, after the same blur, matches the original pixel-for-pixel.

    Reporting only “residual less than 1/255” is far from enough. At minimum you must state three things: at which stage the comparison was done, who the candidate prefixes were, and whether the candidate set was unique.

    Watch out even more for “overfitting”: when you try enough candidates, you can always find one that looks convincing.

    2.2 Font Width: An Overlooked Hard Constraint

    Text rendering follows a plain geometric rule: with font size fixed, total text width accumulates from character widths, and narrow characters like i and l are markedly narrower than m and w. “GLM-Name”, “GLM-Nano”, and “GLM-Next” all have 8 characters, and the first 7 are identical—their widths and aspect ratios are likely nearly identical.

    This is exactly why “pixel-perfect match” struggles to tell them apart. Conversely, if a reconstructed name’s text-block width clearly doesn’t match the original, you can rule it out directly. Rather than asking “does it look similar?”, ask “how many possibilities still hold simultaneously?”

    2.3 A Screenshot Is Not Raw Data

    A screenshot is itself a lossy artifact: resampling, JPEG compression, re-screenshots, and platform transcoding all introduce distortion; visual estimation and AI recognition equally mistake guesses for observations. A more sensible evidence hierarchy is:

    Level Evidence form Credibility
    L0 Paraphrase, verbal mention Low, clue only
    L1 Redacted, partial, or second-hand screenshot Low, can only raise a hypothesis
    L2 Complete leaderboard page, original tasks, per-task scoring records Medium, recomputable and spot-checkable
    L3 Official technical report + reproducible eval script Fairly high, independently recomputable
    L4 Independent reproduction by multiple parties on their own task distributions High, usable as a decision basis

    In this case, everything circulating online is L1-level evidence, and “pixel verification” and “AI recognition” don’t change that level.

    When recognition results diverge, the right move is to lower the strength of the conclusion, not to pick whichever answer “looks most reasonable.” The SIFT method from fact-checking applies equally to leaderboard forensics:

    • S (Stop): When you see an astonishing conclusion, first ask “what am I looking at?” A screenshot is not raw data.
    • I (Investigate): Follow the trail to the original source, publish time, and context. As mentioned above, the poster was responding to a developer’s complaint about complex tasks before posting the leaderboard.
    • F (Find): Look for official announcements, technical reports, independent reproductions, and the leaderboard maintainer’s own statements.
    • T (Trace): Restore the original image, text, and context to confirm the screenshot wasn’t cropped or stitched.

    3. What “3 Redacted Slots” Reveal: The Meaning of Size Tiers

    Setting aside the specific products, what this leaderboard is really worth noting is the strategy of “multiple sizes in the same generation.” This has become common in recent model development: the large cup handles the capability ceiling, the medium cup balances capability and cost, and the small cup targets high concurrency, low latency, and on-device scenarios. But across generations you can’t look at a single overall score.

    From the original data:

    • The large cup (59.4%) is 13.6 percentage points above the previous-gen flagship (45.8%), i.e., 8.16 raw points.
    • The small cup (54.0%) is 8.2 percentage points above the previous gen, i.e., 4.92 raw points—more than “half a task.”

    The problem is:

    1. The sample is too small. 8.2 percentage points is less than one task in a 6-task system. Switch the task set or the scoring method and the ranking could flip.
    2. The spread is unnatural. In a min-max-normalized leaderboard, adjacent ranks usually differ by very little. If the small cup really is nearly 10 percentage points above the previous gen and no one had seen a corresponding clue on earlier leaderboards, then “this redacted image is the only source” is itself worth doubting.
    3. Different comparison baselines. A small model catching up to a large one with a bigger token budget or self-consistency voting isn’t rare, but that is a different matter from “the small model’s native capability is stronger.”

    The genuinely decision-relevant question is: under the target load, which size can meet the quality floor at acceptable latency and cost? Here is a demonstration with illustrative data.

    Suppose a team splits online requests into three tiers by complexity, with sample counts of 1000, 400, and 100. Sampling estimates for a new small cup (S2) and large cup (S1):

    1. import math
    2. def wilson_interval(k, n, z=1.96):
    3. """Wilson score interval: more robust than normal approximation on small samples"""
    4. p = k / n
    5. denom = 1 + z**2 / n
    6. center = (p + z**2 / (2 * n)) / denom
    7. half = z * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denom
    8. return center - half, center + half
    9. # Illustrative data: pass count/total on the sampled task set, plus average per-request cost (relative)
    10. layers = {
    11. "L1_Simple": {"n": 1000, "s1": (940, 1.0), "s2": (915, 0.2)},
    12. "L2_Medium": {"n": 400, "s1": (300, 1.0), "s2": (250, 0.2)},
    13. "L3_Complex": {"n": 100, "s1": (60, 1.0), "s2": (28, 0.2)},
    14. }
    15. rows = []
    16. for name, d in layers.items():
    17. lo, hi = wilson_interval(d["s1"][0], d["n"])
    18. rows.append((name, d["n"], lo, hi, d["s1"][1], d["s2"]))
    19. traffic = {"L1_Simple": 0.7, "L2_Medium": 0.25, "L3_Complex": 0.05}
    20. def route(threshold):
    21. cost = q = wsum = 0.0
    22. for name, _, lo, _, c1, s2 in rows:
    23. pick_big = lo < threshold # lower bound of large-cup pass rate below threshold -> upgrade to large cup
    24. c, s = (c1, s2) if pick_big else (1.0, s2)
    25. cost += traffic[name] * c
    26. q += traffic[name] * s
    27. wsum += traffic[name]
    28. return cost, q / wsum
    29. print(f"{'Layer':<8}{'N':>6}{'Large-cup 95% CI':>20}{'Large cost':>10}{'Small cost':>10}")
    30. for name, n, lo, hi, c1, s2 in rows:
    31. print(f"{name:<8}{n:>6}{f'{lo:.1%} ~ {hi:.1%}':>20}{c1:>10.2f}{s2[1]:>10.2f}")
    32. for t in (0.90, 0.95, 0.98):
    33. c, q = route(t)
    34. print(f"Quality floor {t:.0%}: avg cost {c:.3f}, weighted pass rate {q:.1%}")

    Illustrative output:

    1. Layer N Large-cup 95% CI Large cost Small cost
    2. L1_Simple 1000 92.5% ~ 95.3% 1.00 0.20
    3. L2_Medium 400 70.1% ~ 79.5% 1.00 0.20
    4. L3_Complex 100 52.7% ~ 67.1% 1.00 0.20
    5. Quality floor 90%: avg cost 0.320, weighted pass rate 91.2%
    6. Quality floor 95%: avg cost 0.785, weighted pass rate 95.2%
    7. Quality floor 98%: avg cost 1.000, weighted pass rate 95.2%

    Two takeaways:

    • Facing the same “is the large cup stronger?” debate, “looking only at the mean” and “looking at the lower bound of the confidence interval” lead to different routing strategies.
    • In the illustrative data, raising the floor from 90% to 95% pushes average cost from 0.320 up to 0.785. The reason: the medium tier’s CI lower bound is only 70.1%, below the floor, so 75% of that traffic gets routed to the large model.

    The conclusion depends on the task distribution and decision rule, not on a single ranking on a leaderboard.

    4. Large Samples ≠ Trustworthy: Four Hidden Variables in Evaluation

    4.1 Data Contamination

    Once public leaderboard tasks spread widely, they may enter later models’ training data, and evaluation degrades from “testing capability” to “testing memory.” Countermeasures:

    • Build a private task set;
    • Continuously inject new tasks and retire those with high leakage risk;
    • Use perturbed tasks to detect signs of memorization.

    For example, insert a side-effect-free line into a function body:

    1. def add_inventory(items, delta):
    2. _ = 0 * len(items) # side-effect-free perturbation; if the output changes with it, watch for memorized answers
    3. return [it + delta for it in items]

    4.2 Evaluator Flaws

    • On metrics like “functional correctness,” incomplete unit tests overestimate capability; defects in the benchmark itself likewise cause misjudgment.
    • Scoring models for generative tasks have known biases in position, length, and self-preference; randomly swap answer order and, where possible, use multiple scoring models.
    • “Verifiable criteria” beat “subjective scoring”: prefer unit tests, static checks, compilation, and run results.

    4.3 Sampling Variance

    Single-run randomness shows up directly in the scores. Countermeasures:

    • Fix sampling parameters like temperature, and run each sample at least 3 times;
    • Report mean, variance, and confidence interval together;
    • Note context length, reasoning mode, and tool use when evaluating.

    4.4 Task Difficulty and Sampling

    Different difficulty distributions can change the ranking. For instance, the simulation below shows that when task difficulty mismatches a model’s strengths, a 6-task leaderboard can reach the opposite conclusion.

    1. import numpy as np
    2. rng = np.random.default_rng(42)
    3. tasks = ["T1", "T2", "T3", "T4", "T5", "T6"] # 2 reasoning tasks + 4 coding tasks
    4. binary = {
    5. "A_code_leaning": [0, 0, 1, 1, 1, 1],
    6. "B_reasoning_leaning": [1, 1, 0, 0, 0, 0],
    7. }
    8. print("Case 1: original 6 tasks (binary scoring)")
    9. for name, x in binary.items():
    10. print(f" {name}: {np.mean(x):.1%}")
    11. def bootstrap_ci(x, n_boot=20000, alpha=0.05):
    12. """Paired bootstrap: resample tasks with replacement to get CI of the mean difference"""
    13. x = np.asarray(x, dtype=float)
    14. diffs = x - y
    15. idx = rng.integers(0, len(x), size=(n_boot, len(x)))
    16. stats = diffs[idx].mean(axis=1)
    17. return np.quantile(stats, [alpha / 2, 1 - alpha / 2])
    18. for name, x in binary.items():
    19. lo, hi = bootstrap_ci(x)
    20. print(f" {name}: mean diff 95% CI [{lo:.1%}, {hi:.1%}], interval crosses 0, can't distinguish better/worse")
    21. ax = np.array([0.65, 0.45] + [0.9] * 4) # pass rates in steps of 0.05
    22. bx = np.array([0.30, 0.70] + [0.3] * 4)
    23. print("\nCase 2: switch to 10 tasks (add 4 coding tasks), rubric partial credit")
    24. for name, x in [("A", ax), ("B", bx)]:
    25. lo, hi = bootstrap_ci(x)
    26. print(f" {name}: {x.mean():.1%}, 95% CI [{lo:.1%}, {hi:.1%}]")
    27. print(" Mean diff 95% CI [%.1f%%, %.1f%%]" % tuple(bootstrap_ci(ax)))

    Illustrative output:

    1. Case 1: original 6 tasks (binary scoring)
    2. A_code_leaning: 66.7%
    3. B_reasoning_leaning: 33.3%
    4. A_code_leaning: mean diff 95% CI [0.0%, 100.0%], interval crosses 0, can't distinguish better/worse
    5. B_reasoning_leaning: mean diff 95% CI [0.0%, 100.0%], interval crosses 0, can't distinguish better/worse
    6. Case 2: switch to 10 tasks (add 4 coding tasks), rubric partial credit
    7. A: 76.5%, 95% CI [56.5%, 89.4%]
    8. B: 42.0%, 95% CI [31.8%, 53.3%]
    9. Mean diff 95% CI [20.8%, 49.2%]

    Interpreting the results:

    • Case 1: Under binary scoring, the two models’ scores on the same tasks are exactly complementary. Paired bootstrap shows the 95% CI of the mean difference is 0% to 100%, so no ranking conclusion holds.
    • Case 2: Switch to 10 tasks with partial credit and the interval narrows markedly, and only then does the difference become credible.

    Two rules of thumb follow:

    • Small leaderboards are for “elimination rounds,” not “ranking rounds.”
    • Changes in tasks and scoring are enough to reverse a leaderboard’s conclusion.

    5. From Leaderboard to Decision: An Actionable Evaluation Workflow

    Many teams rely solely on leaderboards and online screenshots when deciding whether to adopt a new model. I suggest switching to a six-step loop:

    1. Build the task set
      • Sources: desensitized samples of real online requests, historical tickets, real commits from code repositories;
      • Scale: at least 100 per complexity tier;
      • Coverage: single-file completion, cross-file edits, bug fixing, test generation, code explanation.
    2. Define criteria
      • Coding tasks: run unit tests in an isolated container platform and check static-scan results;
      • Generative tasks: rule checks and human review in parallel;
      • Version the criteria and task set together.
    3. Unify run conditions
      • Record the model, decoding parameters, context length, and tool use;
      • Run each sample at least 3 times;
      • Write results to object storage and logs to a logging service for later traceability.
    4. Statistical analysis
      • Use paired-bootstrap confidence intervals for overall conclusions;
      • When comparing the same pair of models multiple times, apply multiple-comparison correction. For example, the Benjamini-Hochberg (BH) method first sorts by p-value ascending, finds the largest k satisfying p(k) ≤ k·α/m, and declares the first k hypotheses significant;
      • The example code below demonstrates the full analysis chain from raw run records.
    1. from collections import defaultdict
    2. import numpy as np
    3. from scipy.stats import binomtest
    4. rng = np.random.default_rng(7)
    5. # Illustrative data: task -> model -> score per run (0~1)
    6. runs = {
    7. "T1": {"A": [1, 1, 0.9], "B": [0.7, 0.8, 0.7]},
    8. "T2": {"A": [0.5, 0.6, 0.5], "B": [0.9, 0.8, 0.9]},
    9. "T3": {"A": [1, 1, 1], "B": [0.6, 0.7, 0.6]},
    10. "T4": {"A": [0.8, 0.9, 0.8], "B": [0.8, 0.7, 0.8]},
    11. }
    12. # 1) Variance across repeated runs on the same task: distinguish "unstable model" from "hard task"
    13. print("Task\tModel\tMean\tStdDev")
    14. for task, by_model in runs.items():
    15. for model, xs in by_model.items():
    16. print(f"{task}\t{model}\t{np.mean(xs):.2f}\t{np.std(xs, ddof=1):.2f}")
    17. # 2) Paired bootstrap: resample tasks with replacement to get CI of the A-B mean difference
    18. def paired_boot_diff(runs, model_x="A", model_y="B", n_boot=20000, alpha=0.05):
    19. tasks = sorted(runs)
    20. per_task = np.array([[np.mean(runs[t][model_x]), np.mean(runs[t][model_y])]
    21. for t in tasks])
    22. diffs = per_task[:, 0] - per_task[:, 1]
    23. idx = rng.integers(0, len(tasks), size=(n_boot, len(tasks)))
    24. stats = diffs[idx].mean(axis=1)
    25. return np.quantile(stats, [alpha / 2, 1 - alpha / 2])
    26. lo, hi = paired_boot_diff(runs)
    27. print(f"A-B mean diff 95% CI: [{lo:.1%}, {hi:.1%}]")
    28. # 3) Multiple-comparison correction (BH) against baseline, using T1, T2 as examples
    29. pvals = [binomtest(3, 3, 0.5).pvalue, binomtest(0, 3, 0.5).pvalue]
    30. m, alpha = len(pvals), 0.05
    31. order = sorted(range(m), key=lambda i: pvals[i])
    32. max_k, flags = 0, [False] * m
    33. for rank, i in enumerate(order, start=1):
    34. if pvals[i] <= rank * alpha / m:
    35. max_k = rank
    36. for rank, i in enumerate(order[:max_k], start=1):
    37. flags[i] = True
    38. print(f"Tasks significant after BH correction: {sum(flags)}/{m}")

    Illustrative output:

    1. Task Model Mean StdDev
    2. T1 A 0.97 0.06
    3. T1 B 0.73 0.06
    4. T2 A 0.53 0.06
    5. T2 B 0.87 0.10
    6. T3 A 1.00 0.00
    7. T3 B 0.63 0.06
    8. T4 A 0.83 0.06
    9. T4 B 0.77 0.06
    10. A-B mean diff 95% CI: [-6.2%, 34.4%]
    11. Tasks significant after BH correction: 1/2

    This code illustrates three engineering details:

    • Decompose variance first: the dispersion of repeated runs of the same model reflects stability; performance differences across tasks of varying difficulty form the basis for comparison.
    • Distinguish two dimensions: bootstrap targets uncertainty from “task sampling”; when comparing multiple models, single-run variance and the number of repeats affect statistical power and require a separate power analysis.
    • Interval crossing 0 means no significant difference: with 4 tasks, A and B’s mean difference is about 14 percentage points, but the interval still crosses 0; after BH correction, only one of the two tests is significant.
    1. Blind review
      • Hide the model source, shuffle order randomly, and have reviewers score against a checklist;
      • Follow a Likert scale: 5 levels per dimension, e.g., functional correctness as “1 completely wrong, 2 partly correct, 3 usable but with obvious defects, 4 basically usable, 5 fully meets expectations”;
      • Aggregate with weighting at the end.
    2. Canary and continuous regression
      • Ramp traffic in 1% → 10% → 50% steps;
      • Use monitoring and alerting to track adoption rate, rework rate, latency, and token consumption;
      • Auto-trigger a regression after every model change.

    If code execution or externally displayed generated content is involved, also handle isolation and content-safety controls. The sandbox should restrict network and filesystem permissions; generated content should pass through a safety filter; treat model outputs under the “untrusted input” assumption to prevent prompt injection.

    This workflow doesn’t depend on any specific vendor—whether carried by a general cloud platform or by Baidu AI Cloud’s general capabilities, the decision basis comes from the team’s own data.

    6. Toolchain Adaptation: The Engineering Meaning of “Ship the Full or Lite Version First”

    The community discussion around when a new model becomes available is worth analyzing too. According to a paraphrase, when a coding-tool developer talked about capacity expansion, he mentioned that a certain new model series was next. When asked whether to prioritize adapting the “full” version or the “lite” version, he answered with a single word: “full.”

    It’s worth noting that the first half of this exchange was about service capacity expansion and mentioned no version number. Interpreting “X is up next” as “the new model is about to open up” is likewise L0-level speculation. The 2025 International AI Safety Report, discussing “critical capabilities,” likewise notes that whether a model “has” a capability and whether it can be “accessed” are two independent questions.

    Still, the question has real engineering significance:

    • Prioritizing the full version signals that the tool treats complex tasks as a core scenario;
    • Prioritizing the lite version puts more weight on throughput and cost.

    For consumers, the answer isn’t to passively wait:

    • Insert an abstraction layer between application and model;
    • Switch models via configuration;
    • Build in fallback and circuit breaking to avoid a single model’s jitter causing an outage.
    1. import time
    2. from dataclasses import dataclass, field
    3. @dataclass
    4. class ModelRoute:
    5. primary: str # primary model id, injected by config, not hard-coded
    6. fallback: str # fallback model id
    7. timeout_s: float = 30.0
    8. max_retries: int = 1
    9. cooldown_s: float = 60.0 # cooldown after the primary model fails repeatedly
    10. _open_until: float = field(default=0.0, init=False)
    11. def pick(self) -> str:
    12. return self.primary if time.monotonic() >= self._open_until else self.fallback
    13. def invoke(self, client, payload: dict) -> dict:
    14. last_err = None
    15. for model in (self.pick(), self.fallback):
    16. for attempt in range(self.max_retries + 1):
    17. try:
    18. return client.chat(model=model, timeout=self.timeout_s, **payload)
    19. except Exception as err: # Example: a real implementation should distinguish retryable from non-retryable errors by type
    20. last_err = err
    21. if model == self.primary:
    22. self._open_until = time.monotonic() + self.cooldown_s
    23. if attempt < self.max_retries:
    24. time.sleep(0.5 * 2 ** attempt) # exponential backoff
    25. raise RuntimeError("Both primary and fallback model calls failed") from last_err

    Example config:

    1. routes:
    2. code_chat:
    3. primary: "platform_a_flagship" # placeholder id, filled in by the integrator per their platform
    4. fallback: "platform_a_lite"
    5. timeout_s: 30
    6. max_retries: 1
    7. commit_message:
    8. primary: "platform_b_small"
    9. fallback: "platform_a_lite"
    10. timeout_s: 10

    Two notes:

    1. Model ids should be “placeholders.” Declare only the required capabilities in code (e.g., long context, function calling, code explanation), and keep model ids and credentials in configuration. When a model is delayed, renamed, or retired, business code doesn’t need to change.
    2. This is only a fallback. If evaluation shows a model lags clearly on key tasks, fix the routing strategy rather than letting it play the fallback role long-term.

    7. Conclusion: Make Every Number Recomputable

    Back to the screenshot at the start:

    • Three size suffixes;
    • Two mutually contradictory overall scores;
    • At least three prefix guesses;
    • One over-interpreted conversation.

    This is the typical state of L1-level evidence: it can raise a hypothesis but isn’t enough to support a conclusion.

    A more robust approach is to build your own “evidence chain”:

    1. Original data is traceable;
    2. Statistical conclusions have intervals;
    3. The evaluation workflow is reproducible;
    4. The routing strategy can fall back.

    With these four in place, “is the new model worth adopting?” no longer depends on whose screenshot is sharper, but on the recomputation results on your own task set.

    Action checklist:

    • Build a tiered task set, at least 100 per tier;
    • Fix the eval config and scoring method, run 3+ times, and record variance;
    • Attach a confidence interval to every comparison conclusion; no rankings on small samples;
    • Apply multiple-comparison correction for multi-model comparisons;
    • Complete model abstraction, configuration, and fallback in the application layer;
    • Track online performance with monitoring and alerting, and auto-regress after every model change.