LLM Selection: From Public Leaderboards to a Self-Built Evaluation System

Recently, discussions in the model-serving community have heated up around a new generation of models. New entries have appeared in cloud model catalogs that are still in early testing, and some people have inferred release windows from them. Another detail that keeps coming up is a test image drawn in vector graphics: a bicycle riding on a beach, with richer detail than the previous generation. For developers, though, there are two more worthwhile technical questions:

1. Can the score differences on leaderboards support production selection decisions?
2. Beyond watching catalog entries, how should a team quantify the real gains brought by a “new generation”?

This article does not evaluate the capabilities of any specific product. It only discusses reusable methods and provides evaluation code you can put to work directly.

1. First, clarify: what catalog entries and benchmark screenshots tell us — and what they don’t

An entry appearing in a model catalog usually means the model has entered a release or validation stage. Making trend judgments from screenshots and timelines is a form of intelligence analysis; the method itself isn’t wrong, but how the conclusion is applied matters:

• It can only answer “when the model might appear,” not “is it usable and better for my business.” Release cadence depends on the vendor’s engineering progress and is highly uncertain.
• A catalog entry does not contain all the information your business needs, such as context length, concurrency quota, rate-limiting policy, pricing, data-retention policy, and service-level objectives. For these, the official documentation and the service agreement remain the source of truth.
• Third-party benchmark screenshots have inherent limitations: the sampling parameters are unknown, the prompt is unknown, and you only see the selected samples, which introduces obvious survivorship bias.

If you’re planning engineering investment ahead of a release window, a reasonable approach is to design around capability goals rather than bind to a version number. For example, encapsulate a capability requirement in a unified model-invocation layer, keeping the interface, rate limiting, retries, timeouts, and logging consistent, and letting only the “model identifier” be a configurable item. That way, when a new version ships, you only need to complete evaluation and change the config to switch during a canary rollout. If you hard-code vendor-specific parameters — reasoning effort, thinking budget, and similar switches — the switching cost grows linearly with the number of integration points.

2. The inherent limitations of public benchmarks

The gap between leaderboard scores and real business performance comes mainly from four sources:

1. Data contamination: The questions and their variants may already be in the training data, so the score reflects memorization rather than generalization.
2. Saturation of discrimination: On the same capability dimension, the structural-restoration gap among top models is already small, and only detail-level dimensions can separate them. As with certain fixed test questions where “the structure is right, the pose is right,” a new version just refines lighting and texture.
3. Single-sample sampling and prompt sensitivity: The same model can shift noticeably with a different prompt or sampling temperature, so a single sample carries no statistical significance.
4. Construct validity: A benchmark measures “can it do this kind of task,” while the business cares about “how well does it do on my data distribution.” The two are not equivalent.

One more point is often overlooked: once a category of questions becomes a public test, it enters training corpora and vendors’ optimization lists, and scores keep climbing. But real business distributions have a very long tail, and the parts the score does not cover won’t be reflected. So the correct use is to treat public benchmarks as an initial screening tool: they’re enough to eliminate clearly unqualified candidates, but not enough to support a final decision.

3. A three-layer evaluation system

Layer	Purpose	Sample size	Cost	Trigger
Public benchmark quick screen	Rule out clearly unqualified candidates	Thousands of questions	Low	When a candidate model first appears
Self-built business eval set	Quantify business fit, drive selection	Hundreds to thousands	Medium	On every model or prompt change
Online canary & A/B	Validate real gains and stability	Real traffic	High	On model version release, before peak events

Each layer has its own job: the public benchmark answers “is it worth the time to evaluate,” the self-built set answers “which one,” and online canary answers “is it actually better.” Skip the second layer and you’ll keep oscillating between “higher leaderboard” and “worse business performance”; skip the third and you won’t catch long-tail regressions.

4. The self-built evaluation set: four key actions

1. Collection and cleaning: Sample from online logs, stratified by business scenario; after anonymization, keep the input, the expected output, and the basis for judgment. During cleaning, remove duplicate samples, otherwise high-frequency requests will dominate the scores.
2. Coverage design: Use a matrix to guarantee coverage, e.g., “task type × difficulty × input form × language.” Prepare at least 20–30 items per cell so that sub-scores have basic credibility.
3. Judging strategy: Prioritize objective questions that can be verified programmatically, then rubric-based model judging, and finally human spot-checks to calibrate the first two. Report the agreement between model judging and human scoring, e.g., Cohen’s kappa, and revise the rubric when it falls below 0.6.
4. Version control: Put the eval set, prompt templates, and scorers all under version control, and for each evaluation run record the model snapshot, sampling parameters, and raw artifacts.

4.1 A reproducible scorer for structured graphics generation

Take vector graphics generation as an example. You can first use a deterministic checker to score the “structural layer,” including syntactic validity, whether key parts are present, and whether positions are roughly correct, then use a rubric for the “detail layer.” The code below is illustrative only; the coordinate parsing is a simplified implementation, and in real use you should lean on a mature path-parsing library.

"""Deterministic scorer for structured graphics generation (illustrative implementation)."""
import math
import re
import statistics
import xml.etree.ElementTree as ET
from dataclasses import dataclass, field
from typing import Callable
CANVAS = (1000, 1000)  # Canvas size; adjust to your actual output
@dataclass
class Part:
    name: str
    box_ref: tuple  # reference bounding box (x, y, w, h)
    weight: float = 1.0
@dataclass
class Judge:
    """Judgment result for a single sample, for per-item traceability."""
    item_id: str
    syntax_ok: bool = False
    error: str = ""
    part_scores: dict = field(default_factory=dict)
    @property
    def score(self) -> float:
        # Invalid syntax scores 0 directly, so "parses but draws wrong"
        # is not conflated with "complete failure"
        return 0.0 if not self.syntax_ok else sum(self.part_scores.values())
def parse_shapes(text: str) -> dict:
    root = ET.fromstring(text)
    shapes = {}
    for el in root.iter():
        tag = el.tag.split("}")[-1]
        if tag in {"path", "rect", "circle", "ellipse", "polygon", "line", "polyline"}:
            shapes.setdefault(el.get("id") or el.get("class") or tag, []).append(el)
    return shapes
_NUM = re.compile(r"-?\d+(?:\.\d+)?")
def bbox_of(el) -> tuple | None:
    """Rough bounding box: rect/circle use explicit attributes, path uses the numeric coordinate range."""
    if el.tag.endswith("rect"):
        x, y = float(el.get("x", 0)), float(el.get("y", 0))
        return x, y, float(el.get("width", 0)), float(el.get("height", 0))
    if el.tag.endswith("circle"):
        cx, cy, r = (float(el.get(k, 0)) for k in ("cx", "cy", "r"))
        return cx - r, cy - r, 2 * r, 2 * r
    if el.tag.endswith("path"):
        nums = [float(n) for n in _NUM.findall(el.get("d", ""))]
        xs, ys = nums[0::2], nums[1::2]
        if not xs or not ys:
            return None
        return min(xs), min(ys), max(xs) - min(xs), max(ys) - min(ys)
    return None
def iou(a: tuple, b: tuple) -> float:
    ax, ay, aw, ah = a
    bx, by, bw, bh = b
    ix = max(0.0, min(ax + aw, bx + bw) - max(ax, bx))
    iy = max(0.0, min(ay + ah, by + bh) - max(ay, by))
    inter, union = ix * iy, aw * ah + bw * bh - ix * iy
    return inter / union if union > 0 else 0.0
def evaluate(text: str, parts: list[Part], item_id: str = "demo") -> Judge:
    judge = Judge(item_id=item_id)
    try:
        shapes = parse_shapes(text)
    except ET.ParseError as exc:
        judge.error = f"XML parsing failed: {exc}"
        return judge
    judge.syntax_ok = True
    for p in parts:
        boxes = [bbox_of(el) for el in shapes.get(p.name, [])]
        best = max((iou(p.box_ref, b) for b in boxes if b), default=0.0)
        judge.part_scores[p.name] = round(p.weight * best, 4)
    return judge

The “detail layer” can use a fixed rubric, scoring each dimension 0–2 and computing agreement with human scores:

# rubric.yaml: takes 1-2 minutes to score per sample; suitable for both
# human spot-checks and model-based judging
dimensions:
  - name: Structural correctness   # all key parts present, sensible hierarchy
  - name: Geometric precision      # part positions and proportions match the reference
  - name: Detail richness          # lighting, material, and environmental details
  - name: Visual plausibility      # no floating, clipping, or proportion imbalance
  - name: Editability              # clear grouping and naming, easy to modify further
scoring: {1: clearly missing, 2: basically satisfied, 3: complete and natural}

Quantitative scores must serve decisions and should at least answer four questions:

1. Is the gap stable: Do the means and confidence intervals separate across multiple samplings and prompt templates?
2. Is the gap significant: Make paired comparisons on the same set of inputs, e.g., paired bootstrap or the Wilcoxon signed-rank test, with multiple-comparison correction when comparing more than two models.
3. Can the gain reach business metrics: e.g., tool-call success rate, form-field extraction accuracy, number of manual edits per task, end-to-end latency, and per-unit cost.
4. Is the cost acceptable: Whether changes in inference time, artifact size, and per-call cost stay within budget.

The following is an example of estimating a confidence interval for the mean using bootstrap:

import random
def bootstrap_ci(scores: list[float], n_boot: int = 5000, alpha: float = 0.05,
                 seed: int = 42) -> tuple[float, float]:
    """Return the (1-alpha) confidence interval of the mean distribution."""
    rng = random.Random(seed)  # fixed seed for reproducibility
    n = len(scores)
    means = [
        sum(rng.choices(scores, k=n)) / n
        for _ in range(n_boot)
    ]
    means.sort()
    lo = means[int(alpha / 2 * n_boot)]
    hi = means[int((1 - alpha / 2) * n_boot) - 1]
    return round(lo, 4), round(hi, 4)

5. Engineering implementation: regression evaluation and canary release

• Regression gate: Run the eval set on every prompt or model-config change, and block the release if any core metric drops below baseline beyond a threshold, avoiding “fix one scenario, regress three.”
• Failure-sample feedback: Samples judged as failures automatically enter a candidate pool, and after human review are added to the eval set, forming a closed loop of “evaluate - discover - fix - solidify.”
• Raw artifact retention: Store the generated graphics, code, and intermediate results in object storage with tags, so it’s easy to review afterward whether scoring was fair; connect eval run logs to a logging service and monitoring/alerting so you’re alerted promptly when metrics drift.
• Dual-track scoring: Use objective metrics for continuous regression and human spot-checks to calibrate the rubric; when the two disagree, fix the rubric first rather than lowering the threshold.

6. Conclusion

Catalog entries and benchmark screenshots can provide signals, but they can’t replace evaluation. The suggested path: first do a quick screen with public benchmarks, then build a self-built eval set with a few hundred to a few thousand real business samples, use “deterministic checks + rubric + human calibration” to get reproducible scores, use paired tests to confirm whether differences are significant, and finally validate business gains with canary release and small-traffic A/B. When the evaluation process itself is reproducible and comparable, frequent changes in model versions are no longer a disturbance but a cadence you can actively exploit.