LLM Selection Guide: Five-Dimension Evaluation and Engineering Implementation

1. Ask the Right Question First: From “Which Model Is Strongest?” to “The Best Configuration Under Current Constraints”

Almost every head-to-head comparison of general-purpose LLMs lands on the same kind of conclusion: model X is best at math, model Y writes best, model Z has the longest context. The problem with these conclusions is not that they are wrong — it is that they are not reusable. They compress a multi-dimensional, dynamic, usage-dependent engineering problem into a ranking that cannot guide any action.

Three facts determine the correct posture for model selection:

1. Capability is a vector, not a scalar. Multi-step reasoning, long-document understanding, code generation, style control, and handling time-sensitive information live in different capability subspaces with weak correlation between them. A model that leads on competition math problems is not therefore more reliable at extracting fields from business contracts.
2. System capability ≠ model capability. The same model can differ by an order of magnitude in final performance across three configurations: whether external tools are permitted, whether retrieval augmentation is enabled, and whether output validation is performed. The unit of evaluation should be the full chain of “model + prompt + tools + post-processing,” not the model in isolation.
3. The model side is non-stationary. Version cycles are measured in months, so any specific score expires quickly. The evaluation system therefore has to be engineered, repeatable, and re-runnable with a single command — not a one-off research report.

The correct question is: within my business distribution, cost ceiling, P95 latency target, and compliance boundary, which combination of “model + orchestration strategy” reliably meets the bar? Framing it this way also demotes leaderboard scores to an input rather than a conclusion.

2. Breaking Down and Validating Five Capability Dimensions

2.1 Reasoning and Math: Distinguishing Four Different Difficulty Levels

“Strong at math reasoning” is a heavily conflated claim. In practice it covers at least four kinds of task:

Sub-capability	Typical tasks	Key influencing factor
Single-step computation	Arithmetic, unit conversion	Whether a computation tool is connected
Multi-step chain reasoning	Word problems, logic puzzles	Chain-of-thought length, intermediate step verification
Abstract pattern recognition	Number sequences, shape/symbol rules	Coverage of the model’s pretraining distribution
Formal reasoning	Proofs, constraint satisfaction	Symbolic capability and a closed verification loop

The most important engineering lesson here: do not conflate “can the model compute” with “can the system compute correctly.” Once the model is allowed to call code execution or a calculation tool, most arithmetic errors collapse; only when tools are disallowed do the differences between models get amplified. Evaluations must therefore pin down “tool calls allowed” and “multi-sample voting enabled” as explicit conditions. Otherwise two evaluation runs are not comparable.

A practical technique is self-consistency: sample the same question multiple times, take the mode of the answer distribution, and treat the distribution entropy as a confidence signal. When answers are scattered, routing the request to a stronger model or triggering a tool call is more cost-effective than upgrading the model unconditionally.

2.2 Long Context and Document Processing: Window Size Is Not Effective Length

The selling point of ultra-long context windows is easy to misread. The window limit only decides whether content fits; what decides quality is the effective context — the model’s ability to recall information from the middle sections of a long input. The widely observed “lost in the middle” effect means that dumping 20 contracts into a 1M-token window at once often performs worse than doing structured extraction first and then reasoning section by section.

Design the evaluation in three progressive layers:

• Layer 1: Single-needle retrieval. Bury a unique fact in a long text, vary its relative position (beginning / middle / end), and plot recall rate against position. This curve directly determines how your prompt template should organize critical instructions.
• Layer 2: Multi-hop QA. Answering requires combining information across 2–3 positions, testing association rather than memorization.
• Layer 3: Global synthesis. Full-document summarization, cross-document conflict detection, metric aggregation. These tasks are extremely sensitive to omission rate and should be checked item by item against a rule-based checklist rather than self-assessed by the model.

The engineering conclusion is equally clear: for most long-document scenarios the right architecture is layered processing, not an ever-longer window. Use a parsing service to convert PDFs and scans into structured text, chunk by section, embed and retrieve, extract block by block, then merge and summarize. This pipeline typically costs one to two orders of magnitude less per unit than feeding the full long context, and the results are auditable.

2.3 Code and Engineering Loops: Look at the Closed Loop, Not First-Pass Success

Code capability should be evaluated at task granularity, because difficulty varies enormously across subtasks:

• Single-file completion and unit test generation (easiest to saturate)
• Cross-file refactoring and dependency inference (tests repository-level understanding)
• Locating compile/runtime errors (tests the ability to use feedback)
• Natural language to query language, scripts, and configuration (sensitive to format constraints)

The key insight: a low first-pass generation rate is normal; the engineering value comes from the repair loop. Evaluation metrics should therefore expand from “first-pass success rate” into a trio: pass rate, average number of repair rounds needed to pass, and cost per task. Structurally feeding compiler errors, failing test output, and static analysis warnings back to the model usually yields more improvement than swapping the model.

One more advantage: code tasks have verifiable outputs, which is their biggest edge over other task types. Whether code compiles, whether tests pass, whether cyclomatic complexity exceeds thresholds — all can be judged by a machine. These hard signals should enter your evaluation set ahead of human review.

2.4 Language Generation Quality and Controllability: Replace Absolute Scores with Pairwise Comparison

Writing tasks (long-form drafting, style rewriting, structural reorganization, instruction following) have no objective ground truth, so scoring is the least stable of all. Three practices make it workable:

1. Score by dimension. Split into at least: factual consistency, structural completeness, instruction adherence, fluency, and redundancy. A total score hides problems; per-dimension scores locate the gap.
2. Pairwise comparison (A/B). Have reviewers pick the better of two outputs instead of assigning absolute scores. Humans are markedly more consistent in relative judgments than in absolute ones.
3. Randomize order. Reviewers have position bias (favoring the first or second output), so presentation order must be swapped at random and the magnitude of that bias must be measured.

If you use a model as the judge, calibrate its preferences against a small set of human annotations and publish its agreement rate with human scores. Otherwise “model self-evaluation” easily degenerates into a preference for its own style, which contaminates the entire evaluation set.

2.5 Freshness and External Knowledge: An Architecture Problem, Not a Model Problem

Any model that relies on parametric memory has a knowledge cutoff, so requirements like “real-time market data, latest policy, today’s sentiment” should be solved architecturally, not by swapping models:

• Retrieval-augmented generation (RAG): inject the most recently timestamped corpus fragments into the context and require source citations in the output.
• Tool calling: connect to external interfaces such as search, database queries, and computation services.
• Citation consistency checking: extract the key assertions in an answer and trace each one back to a retrieved fragment; flag anything that cannot be traced as low confidence.

On the evaluation side, freshness-sensitive questions require an expiring evaluation set: record the date the question was asked and the correct answer at that time, and re-verify periodically. Otherwise an evaluation set will start treating expired answers as ground truth six months later.

3. Building a Reusable Evaluation System

The goal of an evaluation system is not to produce a leaderboard. It is to turn “change model / change configuration” into a regression-testable pipeline operation.

Evaluation set construction principles

• Drawn from the real distribution. Sample in a stratified way from sanitized production logs, covering high-frequency intents, long-tail intents, and known failure cases. Synthetic data is only a supplement.
• Just large enough. 30–50 cases per intent are enough to surface the main differences; when differences are not significant, a larger sample usually just measures the noise more precisely.
• Rules first. Anything that can be decided with a regex, JSON Schema validation, a unit test, or a numeric tolerance should never be handed to a model for scoring.

Reference evaluation framework

from dataclasses import dataclass, field
from typing import Callable, Optional
@dataclass
class Case:
    case_id: str
    intent: str                 # Business intent label, used for stratified statistics
    inputs: dict
    expected: Optional[dict] = None
    tags: list[str] = field(default_factory=list)  # e.g. "needs_tools", "long_context"
@dataclass
class RunConfig:
    model_id: str
    temperature: float = 0.0
    max_tokens: int = 2048
    allow_tools: bool = True
    context_budget: int = 32_000   # Above this, switch to chunking/retrieval
@dataclass
class CaseResult:
    case_id: str
    passed: bool
    latency_ms: int
    prompt_tokens: int
    completion_tokens: int
    retries: int
    failure_mode: Optional[str] = None   # format error / factual error / timeout / truncation / refusal
def evaluate(cases, cfg: RunConfig, client, judge: Callable) -> dict:
    results = []
    for c in cases:
        for attempt in range(3):                      # Bounded retries so tail latency does not skew conclusions
            r = client.chat(
                model=cfg.model_id,
                messages=build_messages(c, cfg),      # Decides chunking vs. retrieval based on context_budget
                temperature=cfg.temperature,
                max_tokens=cfg.max_tokens,
                tools=get_tools(c) if cfg.allow_tools else None,
            )
            ok, mode = judge(c, r)                    # Rule-based judgment first, model judge as fallback
            if ok:
                break
        results.append(CaseResult(c.case_id, ok, r.latency_ms,
                                  r.prompt_tokens, r.completion_tokens,
                                  attempt, mode))
    agg = {}
    for intent in {c.intent for c in cases}:
        sub = [x for x in results if next(c for c in cases if c.case_id == x.case_id).intent == intent]
        agg[intent] = {
            "pass_rate": sum(x.passed for x in sub) / len(sub),
            "p95_latency_ms": percentile([x.latency_ms for x in sub], 95),
            "avg_tokens": sum(x.prompt_tokens + x.completion_tokens for x in sub) / len(sub),
            "top_failure": Counter(x.failure_mode for x in sub if not x.passed).most_common(3),
        }
    return {"config": cfg.__dict__, "by_intent": agg}

Core metrics to track: per-intent pass rate, P95 latency, average token consumption, and failure mode distribution. The last one is the most valuable — it tells you whether to fix the prompt, connect a tool, or genuinely switch models.

4. Multi-Model Abstraction Layer and Routing Strategy

Single-model lock-in is the most common architectural debt today. Once the evaluation system is in place, the correct way to converge is to introduce a provider-agnostic uniform abstraction that treats the model as a replaceable runtime dependency.

from typing import Protocol
class LLMBackend(Protocol):
    name: str
    max_context: int
    capabilities: set[str]          # {"reasoning", "long_context", "code", "writing", "tools"}
    def chat(self, messages: list[dict], **kw) -> "ChatResponse": ...
class Router:
    def __init__(self, backends: dict[str, LLMBackend], classifier):
        self.backends, self.classifier = backends, classifier
    def pick(self, intent: str, need_context: int, budget_tier: str) -> LLMBackend:
        # Hard constraints first: context length and required capabilities; eliminate anything that fails
        cands = [b for b in self.backends.values()
                 if b.max_context >= need_context
                 and intent in b.capabilities]
        if not cands:
            raise RuntimeError("no backend satisfies hard constraints")
        # Soft policy: select by cost tier / quality tier, always keeping a fallback
        ordered = sorted(cands, key=lambda b: self.score(b, budget_tier), reverse=True)
        return ordered[0]
    def run(self, intent: str, messages: list[dict], need_context: int, budget_tier: str):
        primary = self.pick(intent, need_context, budget_tier)
        try:
            return primary.chat(messages)
        except (TimeoutError, RateLimited) as e:
            metrics.incr("route.fallback", tags={"from": primary.name})
            for alt in self.backends.values():
                if alt.name != primary.name and alt.max_context >= need_context:
                    return alt.chat(messages)      # Degrade: preserve availability, accept some quality loss
            raise

Routing tiers can be arranged from lowest to highest cost:

• Zero-cost tier: exact-match cache, rule templates, semantic cache (reusing historical answers for similar questions, with a similarity threshold and expiry).
• Low-cost tier: small-parameter model plus retrieval, handling structured work such as classification, extraction, format conversion, and intent recognition.
• Mid-cost tier: mainstream general-purpose model, handling most dialogue and writing.
• High-cost tier: strong reasoning configuration (tools enabled, multi-sample voting), triggered only on branches marked as “hard cases” by the evaluation.

The clearest gains come from pairing this with confidence-based routing: let the low-cost tier answer first, judge confidence via rule validation or a lightweight model score, and escalate only low-confidence samples to the high-cost tier. In most business distributions, samples needing escalation are a minority, so total cost is significantly below using the high-cost configuration everywhere, while quality stays near the ceiling.

5. Non-Functional Constraints You Cannot Ignore

Technical selection is often vetoed on these five points, so build them into the evaluation sheet up front:

• Data compliance and data boundaries. Can business data leave the country, is it allowed to be used for training, are encryption and audit trails required. Healthcare, finance, and government scenarios often settle this immediately: dedicated deployment is the only option.
• Deployment form. Public cloud API, dedicated instance, and fully private deployment differ enormously in unit cost and operational burden; private deployment sacrifices model iteration speed and that tradeoff must be made explicitly.
• Portability. Can the abstraction layer switch backends by changing configuration alone, or do proprietary in-house data structures create migration cost?
• Cost and capacity planning. Extrapolate from the real token distribution, distinguish input and output token pricing (input is typically much higher than output), and assess how cache hit rate affects the bill.
• Observability. Every call needs a trace, token accounting, latency percentiles, and failure reason labels, wired into logging and alerting. Without these, no optimization can be verified.

6. Implementation Checklist

1. Define 3–5 high-frequency intents and collect 50 real sanitized samples for each.
2. For each intent, fix the scoring method (rules / unit tests / pairwise comparison) and the pass threshold.
3. Pin the evaluation conditions (tools allowed or not, context budget, number of samples, temperature).
4. Run the same evaluation against 2–3 candidate backends and report per-intent pass rate, P95 latency, unit cost, and failure modes.
5. Wire in the uniform abstraction layer and implement confidence routing plus fallback retry.
6. After launch, regress weekly, re-run the full evaluation on version changes, and attach the evaluation report to the change record.
7. Host the pipeline on general-purpose cloud capabilities: model serving, object storage (corpora and artifacts), a message queue (async batch processing), a container platform (scheduling inference and evaluation jobs), and logging plus monitoring and alerting for observability. Baidu AI Cloud provides this category of general cloud products and can serve as a candidate foundation for the pipeline described above.

7. Three Common Pitfalls

• Substituting public leaderboards for your own evaluation. Leaderboard distributions do not match your business distribution, and they cannot reflect differences in tool and prompt configuration.
• Treating the context window as the solution. Ultra-long windows cost you on both price and middle-section recall; retrieval plus chunking plus hierarchical aggregation is the more stable default architecture.
• Single-model lock-in. A system without an abstraction layer turns every model iteration into an architectural rework; multi-backend plus routing actually lowers long-term total cost in most scenarios.

8. Conclusion

The point of LLM selection is not to memorize the ranking from a given head-to-head comparison. It is to build a mechanism that keeps producing trustworthy conclusions: decompose capability into verifiable sub-dimensions, replace intuition with an in-house evaluation set, use an abstraction layer and routing to keep the decision at runtime, and use observability to turn every adjustment into a verifiable experiment. Once that mechanism exists, “which model should we use” stops being a debate and becomes a report that finishes on its own every week.