LLM Selection Pitfalls: From Benchmark Obsession to Reproducible Evaluation

    Choosing an LLM from leaderboard rankings often fails to match real-world results. This article explains why aggregate scores lose discriminative power once top models saturate, the concrete technical reasons behind perceived model "degradation," and the metrics that actually matter for agent workloads. It then walks through building a reproducible, business-aligned evaluation set, routing tasks across models by capability, and a practical deployment checklist. Use it to make model selection decisions grounded in evidence rather than rankings.

    A problem almost every team hits

    Most teams go through the same experience during model selection: they pick a model by leaderboard rank, only to find that real-world results badly miss expectations. The same model name performs well in the morning and worse in the afternoon. The same question, re-run three times, yields three different conclusions. Two models tied on a benchmark turn out to be one usable and one unusable in real workloads.

    None of this can be dismissed as simply “unreliable subjective perception.” It reflects a combination of evaluation methodology and engineering implementation problems. What follows is organized around five questions: why scores lose meaning → why performance degrades → which metrics to look at → how to build an evaluation → how to divide work and route.

    1. Once aggregate scores saturate, discriminative power collapses

    The capability distribution of leading large models has already converged tightly. Across multiple third-party composite evaluations, the top-ranked models often differ by only 2–3 points overall (for example, the 60-point tier versus the 63-point tier). Meanwhile, the variance from single-round scoring, sampling of question sets, and scorer jitter is frequently on the same order as that gap.

    The root issue is how scores are aggregated. A composite index is essentially a fixed-weight weighted sum:

    1. S = w1·reasoning + w2·code + w3·knowledge + w4·instruction_following + w5·tool_use + ...

    This scalar discards three critical pieces of information:

    • Dimension weights have nothing to do with your business. For a team building a customer-support Q&A product, the weights of “world knowledge and refusal rate” in their utility function far outweigh “algorithm contest problems.”
    • Distribution shape matters more than the mean. A model with a mean of 62 and high variance, versus one with a mean of 61 and very low variance, are not remotely comparable in usability inside an automated pipeline.
    • Measurement error is ignored. If a 1-point overall difference is not backed by a confidence interval, it cannot serve as a basis for a decision.

    The conclusion: once scores enter the saturation zone, stop “looking at total-score rankings.” Instead, look at a per-dimension radar chart + per-dimension variance + pass rate on your own business dataset.

    2. “Degradation” isn’t mysticism: four sources of broken reproducibility

    The feeling that “a certain model got dumber” usually has traceable technical causes, not just perception bias.

    1) Server-side dynamic routing and tiered scheduling. To control cost and latency, providers often introduce a task classifier that automatically downgrades “simple requests” to a smaller or cheaper model instance. This is a reasonable cost strategy on the server side, but for the client it is a silent capability change: you call the same model name, but a different set of weights may actually execute. The symptom is that simple tasks seem fine and complex ones stay okay, or that after a version update, debugging and refactoring tasks drop off a cliff in quality.

    2) Quantization and batching strategies drift with load. Dynamic batching, KV Cache reuse policies, and the choice of different hardware backends all introduce subtle numerical differences that get amplified over long reasoning chains.

    3) Decoding parameters and context-management policies change. Adjustments to temperature, top-p, repetition penalty, and context truncation or compression strategies often affect pass@1 more than the model’s intrinsic capability does.

    4) Effective interaction budget lost to safety guardrails. Over-refusal, frequent insertion of safety prompts, and extra restrictions on tool calls reduce “theoretically available turns” to fewer “actually available turns.” A false refusal is cheap in a human conversation but can directly fail an entire batch of tasks in an unattended automated pipeline.

    So any evaluation must first establish a reproducible experiment record; otherwise the results cannot be attributed:

    1. EVAL_RECORD = {
    2. "model_id": "...", # exact version/snapshot ID, not a friendly name
    3. "snapshot_ts": "2026-08-27T10:00:00Z",
    4. "region": "...", # different regions/clusters may behave differently
    5. "sampling": {"temperature": 0.0, "top_p": 1.0, "max_tokens": 8192},
    6. "system_prompt_hash": "sha256:...",
    7. "tool_schema_hash": "sha256:...",
    8. "seed": 42, # if supported
    9. "harness_version": "eval-v1.3",
    10. "n_repeat": 5,
    11. }

    Of these, n_repeat (how many times the same question is repeated) is the most easily overlooked and the most valuable. Without repeat sampling, you are measuring a single lucky draw, not capability.

    3. For agent scenarios, stability is the first metric

    Single-attempt pass rate and multi-sample pass rate must be viewed separately:

    • pass@1: the probability of success on a single attempt, directly corresponding to “getting it right the first time” in real workloads.
    • pass@k: the probability of at least one success across k independent attempts, reflecting “whether the capability exists within the model’s distribution.”

    When pass@k is clearly higher than pass@1, the meaning is unambiguous: the model has the knowledge and reasoning ability to solve the task, but its output distribution is unstable and lacks self-verification. Such a model, paired with a framework of “multi-sample + result verification + best-of selection,” often delivers better cost-effectiveness than switching to a model with a higher pass@1 that costs far more.

    Multi-step execution calls for even greater caution about error accumulation. Let the per-step success rate be p and the task require n serial steps; the probability of getting the whole chain right in one pass is roughly:

    1. P_success ≈ p^n
    2. p = 0.97, n = 20 → P ≈ 0.54
    3. p = 0.99, n = 20 → P ≈ 0.82

    A 2-percentage-point gap per step turns into a nearly 30-percentage-point gap in success rate over a 20-step task. This explains why models that differ by “just 1 point on a benchmark” can diverge enormously in long automated workflows. The engineering answer is to put a verifiable intermediate state at every step (compile, unit tests, type checks, schema validation, assertions), converting invisible quality fluctuation into visible failure signals and allowing local retries rather than rerunning the entire chain.

    4. Efficiency isn’t TPS: end-to-end task time is the cost unit

    Raw token throughput (TPS) is a misleading metric. What developers actually care about is “how long a task took from dispatch to acceptance, and how many effective tokens it consumed.”

    In scenarios with tool-calling capability, end-to-end time is made up of three parts:

    1. T_task = T_plan + max(T_tool of the serial dependency chain) + T_verify + T_retry

    The key insight: if edit and retrieval tool calls can be orchestrated in parallel on the server side, end-to-end time will be significantly lower than the sum of pure decoding times. As a result, a model with modest TPS but well-managed tool-call scheduling can feel far better in real IDE/terminal scenarios than a higher-TPS model whose tool calls are badly serialized.

    When evaluating, record these directly observable quantities rather than looking only at TPS:

    Metric Definition Why it matters
    Time to first token Request to first token The sensitive item for interactive experience
    End-to-end task time Dispatch to acceptance The real cost unit
    Effective token rate Tokens reaching the final artifact / total tokens consumed Measures rework and wasted output
    Tool call success rate Valid arguments and correctly executed The first failure source in agent scenarios
    Mean retry count Average retries per task Directly maps to the bill
    Refusal rate Share of unnecessary refusals A quantified signal of over-tight guardrails

    5. Build your own evaluation set: turn “feel” into auditable data

    Leaderboards cannot replace your own evaluation set, because your task distribution has nothing to do with the leaderboard’s. Build it in three layers:

    Layer 1: Deterministic checks (the largest share). Tasks with a unique or machine-judgeable answer, such as whether code compiles and passes unit tests, whether JSON conforms to a schema, whether SQL returns the expected result set, and whether data is consistent after a migration script runs. These scorers have zero cost and zero jitter, and should serve as the primary regression baseline.

    1. def score_deterministic(task, output):
    2. checks = []
    3. checks.append(("schema", validate_schema(output, task.schema)))
    4. checks.append(("compile", run("tsc --noEmit", task.workdir)))
    5. checks.append(("unit", run_pytest(task.workdir)))
    6. checks.append(("lint", run_lint(task.workdir)))
    7. return {"score": sum(c for _, c in checks) / len(checks), "detail": checks}

    Layer 2: Rule-based rubric. For tasks without a unique answer, such as code review or option comparison, write out checkable criteria in advance (whether it covers edge cases, whether it introduces unnecessary dependencies, whether it preserves backward compatibility, whether error handling is complete). Rules-first scoring is stable and comparable across models.

    Layer 3: Model-as-judge (the smallest share, and mandatory bias control). When using a strong model as a judge, do at least three things: hide the model’s source and order; score the same output in both presentation orders (a two-way pass) to cancel positional bias; and sample 10% for human review to calibrate the judge’s agreement with humans.

    The supporting engineering requirements are:

    1. Version the task set. Manage the evaluation set with the same rigor as code: changes go through review, and historical results are bound to the evaluation-set version number.
    2. Regression dashboard. Run the full set on every model version, prompt, and scaffolding change, outputting per-dimension trends rather than a single point score.
    3. Interleaved sampling. On the same batch of tasks, alternate calls to different candidate models across time windows to offset server-load-driven time bias—this is the only effective way to counter “the whole thing got dumber during a certain period.”
    4. Confidence intervals. Repeat the same model n times and report the median and interquartile range, not a single run.

    6. Treat capabilities as a resource pool: multi-model work division and routing

    Since different models hold different relative positions on knowledge depth, instruction following, tool use, and latency/cost, the optimal solution is usually not “pick one all-rounder” but rather divide work by stage:

    • Planning and decomposition: requires strong world knowledge and global reasoning; low call volume and latency-insensitive—suitable for the most expensive reasoning-oriented model.
    • Execution and editing: high-frequency, short-context, and pattern-based—suitable for a cost-effective model, paired with strict schema constraints.
    • Review and acceptance: requires an independent perspective—be sure to use a different model from the execution stage, otherwise shared-origin bias makes the review meaningless.
    • Downgrade principle: any automatic downgrade must be explicitly observable (annotate the actual executing model and the reason in response metadata or logs). Silent downgrades are forbidden—they are the root of all the attribution problems above.

    On cloud platforms such as Baidu AI Cloud, you can typically organize calls across models in different tiers and cost levels, turning the division above into a configurable routing policy, and build separate dashboards for each tier’s quality metrics, latency, and retry rate through log services and monitoring alerts.

    The routing logic can be simplified to:

    1. def route(task):
    2. if task.needs_deep_reasoning: # planning, architecture, root-cause analysis
    3. return STRONG_MODEL
    4. if task.verifiable and task.high_volume: # batched edits with a clear verifier
    5. return CHEAP_MODEL_WITH(schema=task.schema, retry=2)
    6. if task.stage == "review":
    7. return REVIEWER_MODEL != task.exec_model # reviewer must differ from executor
    8. return STRONG_MODEL

    Make the decision along a three-dimensional Pareto frontier of “quality—latency—cost”: first eliminate candidates that exceed budget, then filter the remainder by latency constraints, and only then compare quality. Most selection mistakes come from reversing this order.

    7. Deployment checklist

    • Do you record the exact model snapshot ID, sampling parameters, and evaluation-set version?
    • Did you repeat sampling ≥3 times per question and report variance?
    • Do you track both pass@1 and pass@k?
    • Do deterministic verifiers cover your main task types?
    • Is end-to-end task time included in your metrics, rather than only TPS?
    • Are tool call success rate and retry count monitored separately?
    • Is there any silent downgrade? Is every downgrade observable and traceable?
    • Does the review stage use a different model from the execution stage?
    • Did you use interleaved sampling to rule out time-based bias?

    Conclusion

    Evaluation scores degrading from a “decision basis” to a “reference signal” is an inevitable outcome at this stage, and a sign that evaluation capability must move up into engineering. Reliable selection does not come from leaderboard rank, but from three things: a reproducible experiment record, an evaluation set aligned with your business distribution, and a scheduling architecture that divides work by capability. Get these three right, and a model version change becomes a matter of parameter tuning—not a from-scratch gamble.