LLM Evaluation Methodology: From Capability Breakdown to Practical Model Selection

    Evaluating large language models well means turning vague impressions into signals you can act on. This article lays out a repeatable methodology: why "we've used it for a while and it seems fine" fails as a decision input, how to decompose capabilities into a risk-ordered evaluation checklist, how to build layered and contamination-resistant evaluation sets, and how to combine three automated scoring routes—verifiable rules, model judges, and blind human review. It also covers folding latency and cost into the decision, wiring up a continuous evaluation pipeline with regression, avoiding common pitfalls, and running a two-stage selection matrix that eliminates on hard gates before applying weighted scoring. Readers will get concrete patterns, including pitfalls to avoid and a working example of a main evaluation loop.

    1. Where Evaluation Starts: Turning “We’ve Used It for a While, It Seems Fine” into a Decision-Ready Signal

    In real development settings, whether a model is usable often comes down to a plain, hands-on impression: everyday tasks have been running for a while, you’ve burned through a fair amount of quota, work hasn’t noticeably stalled, and outputs are mostly deliverable. This kind of judgment isn’t worthless—it captures three important things: long-term stability, cost per unit of work, and the real distribution of tasks. The problem is that it isn’t comparable, isn’t reproducible, and is hard to explain to a team.

    To engineer it, you have to sidestep three typical traps:

    • Leaderboard distortion: Public benchmarks can saturate and may be contaminated by training data, and general-purpose capability is not equivalent to your business tasks.
    • Opaque dimensions: Strong performance on common-sense Q&A does not mean a model is equally reliable at long-document citation, structured output, or tool calling.
    • Overlooked cost structure: For the same apparent quality, differences in input/output length distribution and retry rates can make the actual bill differ by several times.

    So the first principle of evaluation is this: the evaluation unit is the “task,” not the “model.” A minimal evaluation unit should contain five elements—the input, the expected output or acceptance rules, the scoring criteria, the weight, and the acceptable failure boundary. Only when business tasks are translated into this kind of executable unit does any downstream automation become meaningful.

    2. Breaking Down Capability Dimensions: How to Design the Evaluation Table

    An evaluation table isn’t a list of capabilities; it’s a troubleshooting checklist ordered by business risk. Cover at least the following dimensions:

    1. Instruction following and format constraints: field completeness, enum-value validity, JSON parseability, and how conflicting constraints are prioritized. These cause the most insidious failures in engineering pipelines, because the model “gets the content right but the format wrong” and downstream parsing fails outright.
    2. Reasoning and multi-hop ability: the focus isn’t whether the answer is right, but whether the intermediate steps can be verified. For tasks that decompose into usable steps, bring the intermediate conclusions into the scoring.
    3. Long-context handling: don’t just test “does it fit”—test the length-decay curve: recall differences when key information sits at the beginning, middle, or end, plus consistency of cross-paragraph citation.
    4. Code and tool calling: judge by executable results, such as unit-test pass rate, parameter-selection accuracy, average number of calls, and the ability to self-correct after errors.
    5. Stability and reproducibility: sample repeatedly at the lowest-randomness settings and observe output consistency; also test boundary inputs, very long inputs, and empty inputs to see whether the model crashes or produces runaway output.
    6. Engineering metrics: time to first token, tokens generated per second, tail latencies, rate-limiting behavior, and timeout and retry behavior.
    7. Cost metrics: distribution of input/output token consumption, prefix-cache hit rate, and cost per successful task.

    3. Building the Evaluation Set: Data Sources, Stratification, and Contamination Defense

    The quality ceiling of your evaluation set directly caps the quality of your conclusions.

    Three sources of data, typically ranked by value as follows:

    • Sampled, desensitized production logs: closest to the real distribution—sample primarily from mid- and long-tail requests rather than only high-frequency simple ones.
    • Expert-constructed hard cases and edge cases: added to cover historical incident reviews, such as format-parsing failures, cross-document contradictions, and very long tables.
    • Synthetic data augmentation: used to fill low-frequency scenarios, but it must be deduplicated semantically to avoid near-duplicate items diluting difficulty.

    Stratification and quotas: stratify on two dimensions—“scenario × difficulty”—set a fixed quota per stratum, and lock it. Without fixed quotas, as the business evolves the share of easy items keeps rising and the overall evaluation score is inflated.

    Contamination defense: keep the private set off external services, rotate items periodically, hold out a “time-isolated reserve set” that never participates in tuning, and periodically insert new items as sentinels during evaluation. A sudden jump in sentinel scores usually means data leakage, not a genuine jump in capability.

    Labeling and statistics: use a dual track of “gold answers + scoring rubric,” have two people label independently and compute inter-annotator agreement, and send disagreements to arbitration. Conclusions must come with sample size and interval estimates—a 3-point difference over a few dozen items usually has no statistical meaning. When comparing models, paired comparison (directly comparing two models’ outputs on the same item) is far more sensitive than comparing two independent means.

    4. Three Technical Routes for Automated Scoring

    Route 1: Verifiable rules (preferred)
    Anything with an objective acceptance criterion should be judged programmatically: unit tests passing, schema validation, SQL result-set comparison, numeric tolerance comparison. Its value lies in zero subjectivity, zero marginal cost, and long-term regression safety.

    Route 2: Model judge
    Suited to open-ended outputs. It carries three built-in biases—position bias, length bias, and self-preference bias. The mitigations are engineering work, not prayer over a prompt:

    1. def pairwise_judge(judge, candidate, reference, rubric):
    2. """Pairwise comparison with position swap to suppress position bias."""
    3. first = judge.compare(candidate, reference, rubric) # candidate first
    4. second = judge.compare(reference, candidate, rubric) # order swapped
    5. if first.winner == second.winner:
    6. return first.winner
    7. return "tie" # On conflicting verdicts, record a tie—don't force a winner.

    Three points matter here: the rubric and prompt templates must be version-pinned; when the verdicts disagree, record a tie, otherwise random noise gets systematically counted toward one side; and calibrate the judge periodically against a human-labeled set, watching whether its agreement rate with human judgment degrades.

    Route 3: Blind human review
    Used for sampling confirmation before a final decision. It must be de-identified, in randomized order, and scored against the rubric—not given an overall “good/bad” impression.

    Use the three routes together. Below is a sketch of the main evaluation loop:

    1. def run_eval(cases, client, judge=None):
    2. # client is a unified inference adapter, decoupled from the specific
    3. # service implementation so providers can be swapped out easily.
    4. report = []
    5. for case in cases:
    6. outputs = [
    7. client.generate(case.prompt, temperature=0.0, max_tokens=case.max_tokens)
    8. for _ in range(case.repeats)
    9. ]
    10. verdict = case.verify(outputs) # rule check takes priority
    11. if verdict is None and judge is not None:
    12. verdict = judge.compare(outputs[0], case.reference, case.rubric)
    13. report.append({
    14. "id": case.id, "scene": case.scene, "difficulty": case.difficulty,
    15. "score": verdict.score, "fail_type": verdict.fail_type,
    16. "tokens": case.usage,
    17. })
    18. return aggregate(report) # Aggregate by scene/difficulty, with confidence
    19. # intervals and failure-type distribution.

    Note that temperature=0 does not mean fully deterministic: floating-point operation order, batch-processing differences, and backend implementation differences all introduce jitter. So repeats should be at least 3, and you should draw conclusions from a consistency rate rather than a single run.

    5. Folding Latency and Cost into the Decision

    An evaluation that compares quality but not cost is incomplete from an engineering standpoint.

    • Look at latency in layers: time to first token drives interactive feel, tokens per second drive the long-read experience, and P95/P99 drive capacity planning and safety margin. Reporting only the mean hides the long tail.
    • Look at cost through token structure: account for input and output separately; for long-context scenarios, focus on prefix-cache hit rate, because reusable system prompts and fixed documents can significantly reduce repeated billing.
    • The most important metric is cost per successful task: total cost divided by the number of tasks that pass rule checks. For an illustrative derivation: with an average of 3,000 input tokens and 400 output tokens per task, if the pass rate rises from 70% to 85% at unchanged unit prices, cost per successful task drops by roughly 18%. Cost savings from quality gains are often more worthwhile than switching to a cheaper model.
    • Sampling parameters are themselves cost switches: the max output length cap, retry count, and failure-fallback strategy all shift the cost curve significantly and should be treated as evaluation variables, not fixed configuration.

    6. The Evaluation Engineering Pipeline and Continuous Regression

    For evaluation to run over the long term, you need a stable engineering pipeline:

    • Sample retention: collect requests and results through a logging service, desensitize and archive them to object storage, set retention periods and access auditing, and make data-compliance boundaries explicit.
    • Batch scheduling: run evaluation jobs on a container platform with elastic scaling sized to the evaluation set, so they don’t compete with online business for resources.
    • Request peak shaving: buffer batch evaluation requests through a message queue and control concurrency to keep test traffic from hitting online quotas.
    • Observability: monitoring and alerting should cover availability, error-code distribution, timeout rate, human negative-review rate, and refusal rate, with anomalies automatically triggering regression.
    • Version triple binding: bind the model version, prompt version, and evaluation-set version into a single baseline snapshot. This is the key to regression triage—only when all three are recorded can metric drift be attributed to a specific variable.
    • Canary and rollback: first run shadow traffic for offline comparison, then ramp up in layers by traffic percentage, with automatic rollback when metrics breach thresholds.

    The object storage, message queues, container platforms, logging services, and monitoring and alerting capabilities this pipeline requires all fall into general cloud-service categories, and product lines such as Baidu AI Cloud also offer corresponding services to support this kind of evaluation engineering.

    7. A Checklist of Common Pitfalls

    1. Treating public leaderboards as the sole basis: use leaderboards for initial screening only; decisions must rest on a private evaluation set.
    2. Drawing strong conclusions from too small a sample: score comparisons without confidence intervals are essentially noise comparisons.
    3. Looking only at the mean, not the long tail and failure modes: failures must be classified into types such as format errors, factual errors, refusals, out-of-bounds behavior, and timeouts, because the fix paths for each type are completely different.
    4. Confusing model differences with scaffolding differences: run ablation experiments—fix the engineering wrapper and swap only the model, then fix the model and swap only the wrapper. Many “the model got worse” conclusions are actually caused by prompt or tooling-layer changes.
    5. Ignoring desensitization and compliance: real logs must be desensitized before entering the evaluation set, and sample access needs auditing.
    6. One-off evaluation: models get updated and business distributions drift, so evaluation must be continuous.

    8. The Selection Decision Matrix: Eliminate First, Then Weight

    A two-stage decision is recommended. The first stage eliminates candidates with hard gates: format-parsing failure rate, P95 latency, cost per successful task, and data-compliance requirements—failing any one means immediate elimination. The second stage applies weighted scoring to the candidates that clear the gates, with weights set by the nature of the business. For example, quality-oriented tasks put accuracy and stability first, real-time interactive tasks raise the weight on latency, and batch offline tasks raise the weight on cost per unit. The weight table itself is illustrative; what matters is making the weights explicit and putting them up for review, rather than hiding them inside someone’s intuition.

    9. Conclusion

    “We’ve used it for a while, burned through a lot of quota, but we can keep going”—if this sentence can be translated into a set of metrics, it becomes a valuable signal; if it can’t, it’s just an impression. Evaluating large language models is not a one-off scoring activity but a continuously running engineering pipeline: the evaluation set must be updated continuously, the scoring methods must be calibrated continuously, and cost and latency must be compared on the same screen as quality. Build this pipeline, and model selection turns from a gamble into a reproducible decision process—and only then does the team gain the ability to deliver reliably in an environment of frequent version iteration.