LLM Evaluation and Selection: A Practical Methodology for Choosing the Right Model

1. Start with a correction: there is no single strongest model

One of the most common questions in technical communities is: “Which mainstream large language models exist, and which one has the strongest overall capability?”

That question contains a hidden assumption: that model capability is one-dimensional and can be globally ranked. In reality, model capability is closer to a multi-dimensional vector, while most business requirements only use a few dimensions of that vector.

A model that leads by a few points on long legal-document extraction may be eliminated in a real-time short-text rewriting scenario because its first-token latency is too high. A model that performs excellently on code completion may fail to enter production because its instruction-following stability is insufficient for multi-turn tool orchestration.

Therefore, the better questions are:

• Which capability profile does my task belong to?
• On the relevant capability dimension, is the gap between candidate models large enough to justify their disadvantages in cost, latency, or data compliance?
• Can that gap be reduced through engineering methods such as prompt engineering, retrieval augmentation, fine-tuning, or post-processing checks?

The third point is especially important. Many so-called “model capability gaps” narrow significantly after engineering support is added. Reducing model selection to a pure model-capability question often leads to over-provisioning.

2. Decomposable dimensions of LLM capability

2.1 Language understanding and generation quality

This is a foundational dimension. It covers semantic understanding, ambiguity resolution, discourse coherence, and style control. It determines whether output “reads like a human wrote it,” but it usually has lower weight in structured tasks.

2.2 Reasoning and multi-step planning

This includes mathematical and logical deduction, multi-hop question answering, constraint satisfaction, and counterfactual reasoning. The difficulty is that the correctness of a reasoning chain is hard to judge with string matching. Evaluation usually has to rely on answer correctness or introduce process-level scoring.

2.3 Code capability

Code capability should be divided into at least four layers. Evaluating them together is meaningless:

1. Syntax-level completion: local code completion with little or no long-context requirement.
2. Single-file understanding and modification: defect localization, unit-test generation, comments, and documentation.
3. Cross-file or cross-repository refactoring: building symbol references and call relationships over long contexts.
4. Autonomous debugging loops: the model can run commands, read errors, modify code, and re-verify.

The deeper the layer, the higher the requirements for long-context handling, tool-call reliability, and self-correction. Differences between models also become more obvious at deeper levels.

2.4 Long-context handling and information retrieval

Context window size is a hard threshold for task feasibility. Context windows visible in the industry have expanded from thousands of tokens to the million-token range. However, window length is not the same as effective utilization length.

Both academic and engineering practice repeatedly show that when key information is located in the middle of a long context, retrieval and citation accuracy drops significantly. Other costs also need to be measured: reasoning degradation caused by diluted attention, as well as linear growth in prefill latency and memory overhead.

Long-context capability should be evaluated with “needle in a haystack” style probe tests, not by simply stuffing a large passage into a prompt and checking whether the model can repeat it.

2.5 Knowledge freshness and RAG compatibility

Knowledge inside model parameters has a training cutoff. For time-sensitive scenarios, the correct solution is usually not to switch models, but to add retrieval-augmented generation (RAG): an external knowledge base provides facts, while the model focuses on understanding and expression.

In this case, the more important evaluation target is the model’s faithfulness: whether it answers based on the provided material, and whether it clearly refuses to answer when the material is insufficient instead of fabricating information.

2.6 Tool calling and structured output

This is the dividing line between “chatting” and building a production system. Key evaluation points include:

• Tool-selection accuracy and parameter-filling accuracy.
• Completion rate for serial or parallel multi-tool orchestration.
• Legal rate and first-pass rate for strict structured output, such as JSON Schema-constrained responses.
• Refusal behavior when no suitable tool exists, rather than forcing an invalid tool call.

In production systems, an interface with a 99% legal-rate structured output is often more valuable than a model with better prose but unstable formatting.

2.7 Engineering attributes

These attributes are often overlooked, but they are frequently decisive: first-token latency and throughput, concurrency quotas and rate-limiting policies, context caching and batch-processing support, stability and degradation behavior under throttling, logging and monitoring observability, deployment models that support private deployment or data residency, and token-based cost structures.

3. Why general leaderboards are not enough for selection

• Benchmark contamination: public benchmark questions and solutions often enter training corpora, inflating scores and disconnecting them from real-world distributions.
• Distribution shift: leaderboard questions are often academic, while business data contains abbreviations, typos, domain jargon, oversized tables, and semi-structured text.
• Prompt sensitivity: the same model can show score fluctuations larger than the differences between models when the instruction wording changes.
• Single-sample variance: generative tasks are stochastic. A single evaluation is insufficient; multiple samples and confidence intervals are needed.
• Capability coupling: a composite leaderboard score mixes dimensions you care about with dimensions you do not. The weights are not aligned with your business needs.

The conclusion is direct: leaderboards are useful for initial screening, mainly to eliminate obviously unsuitable candidates. Final decisions must rely on an internally built evaluation set.

4. Build a layered evaluation system

It is recommended to build the evaluation system in three layers, with increasing cost and increasing realism at each layer.

Layer 1: Offline golden-set evaluation

Sample 200–1,000 cases from real business traffic and manually label reference answers or scoring points. Cover high-frequency scenarios and known failure cases. Once established, include the set in CI so that model changes or prompt changes trigger regression tests.

Layer 2: End-to-end task evaluation

For agent-style applications, evaluate task completion rate rather than answer similarity. Define an expected terminal state, such as a file being correctly modified or an order-status field being correctly written, and verify it with assertions in a test environment.

Layer 3: Online canary testing and human rating

Roll out by traffic percentage. Collect explicit feedback such as thumbs up or thumbs down, and implicit feedback such as whether the output was rewritten, copied, or followed by a session interruption. Use periodic human sampling to calibrate automated metrics.

Metric selection

• Classification tasks: accuracy, precision, and recall.
• Extraction tasks: field-level F1.
• Generation tasks: prefer an LLM-as-judge model with a scoring rubric, but apply bias control. Use a unified prompt template, randomize answer order to remove position bias, require the judge to output scoring rationale before giving a score, and periodically check agreement with human ratings.

5. Illustrative evaluation framework

from dataclasses import dataclass, field
@dataclass
class Case:
    cid: str
    category: str            # code / reasoning / long_ctx / tool_use / structured
    prompt: str
    reference: str | None = None
    rubric: str | None = None    # Rubric used when a judge model is needed
    weight: float = 1.0
@dataclass
class Result:
    category: str
    score: float
    n: int
    p50_latency_ms: float = 0.0
def run_eval(client, cases: list[Case], k: int = 3, judge=None) -> list[Result]:
    """Run k samples per case, keep the best result (pass@k), and collect latency percentiles."""
    buckets: dict[str, list[float]] = {}
    latencies: dict[str, list[float]] = {}
    for c in cases:
        best = 0.0
        for _ in range(k):
            out = client.chat(c.prompt)                  # Use a unified entry point to hide vendor differences
            latencies.setdefault(c.category, []).append(out.latency_ms)
            if c.reference is not None:
                s = exact_or_f1(c.reference, out.text)   # Prefer deterministic metrics when possible
            else:
                s = judge.score(c.prompt, out.text, c.rubric)
            best = max(best, s)
        buckets.setdefault(c.category, []).append(best)
    results = []
    for cat, vals in buckets.items():
        lat = sorted(latencies[cat])
        results.append(Result(
            category=cat,
            score=sum(v * w for v, w in zip(vals, vals)) / len(vals),
            n=len(vals),
            p50_latency_ms=lat[len(lat) // 2],
        ))
    return results

A matching tiered-routing strategy can reduce cost by an order of magnitude without sacrificing quality on critical scenarios:

def route(task) -> str:
    if task.token_count > 64_000:            # Ultra-long context -> flagship model
        return "flagship"
    if task.requires_multi_tool_or_proof:    # Multi-step planning / rigorous reasoning -> flagship model
        return "flagship"
    if task.is_extractive_or_formatting:     # Extraction, rewriting, formatting -> lightweight model
        return "lightweight"
    return "lightweight"

6. Turn evaluation results into decisions: weighted matrix and cost model

Do not output “which model is best.” Output a weighted scoring matrix instead.

Composite score = Σ (dimension_score_i × business_weight_i) − λ × normalized_cost − μ × normalized_latency_penalty

Weights must be confirmed by the business side. In risk-control or compliance review scenarios, weight “faithfulness” and “refusal rate” most heavily. In customer-support assistance, weight first-token latency and concurrent throughput more heavily. In code-assistant scenarios, weight cross-file refactoring capability more heavily.

The cost side should use a reproducible formula rather than intuition:

Daily cost ≈ [(input tokens × input unit price) + (output tokens × output unit price)]
             × request volume
             × (1 − cache hit rate × cache discount)

The three most easily missed items are cache hit rate, batch-processing discount, and retry amplification factor. A retry amplification factor of 1.1–1.3 is a practical planning estimate. Similarly, capacity planning should request quota based on peak QPS, not average QPS.

7. Four hard constraints for engineering deployment

1. Interface abstraction layer: route all model calls through a unified gateway to hide provider differences. When switching models or running canary deployments, business code should require zero changes. This is the lowest-cost insurance against vendor lock-in.

2. Observability: log request IDs, model versions, prompt versions, token usage, latency percentiles, structured-output legal rate, refusal rate, and human feedback. Connect all of this to logging, monitoring, and alerting. Selection without instrumentation is blind selection.

3. Reliability design: implement timeouts, exponential backoff retries, idempotency keys, rate limiting, circuit breaking, and fallback paths to a lightweight model or rule engine. Generative APIs can fail, so the system must tolerate failures.

4. Data compliance: for scenarios involving personal information or business secrets, prefer solutions that support private or dedicated deployment, do not use customer data for training, and provide complete audit logs. Desensitize and minimize input data before sending it, and route output through content-safety review.

8. A practical methodology summary

Treat model selection as continuous engineering, not a one-time procurement decision:

• Use leaderboards for initial screening and an internal golden set for decision-making.
• Break the business problem into capability dimensions first, then look for models that fit those dimensions, rather than starting from a model and reverse-engineering use cases.
• If a gap can be reduced with RAG, prompt engineering, fine-tuning, or post-processing checks, do not solve it by buying a more expensive model.
• Use tiered routing and caching to reduce cost, and concentrate budget on high-value long-tail scenarios.
• Establish routine regression evaluation. Model capabilities change every quarter, so selection conclusions have an expiration date.

In Baidu AI Cloud’s model-service ecosystem, this “capability matrix + internal evaluation + tiered routing” methodology also applies: first connect multiple model capabilities through a unified gateway, such as those provided by the Qianfan platform; then compare them using an internal golden set; and finally lock in selection decisions according to business weights and include them in CI regression. Tools will keep evolving, but the transferability of the methodology is far higher than the lead window of any single model.