LLM Selection Methodology: A Six-Dimension Evaluation and Production Engineering Guide

1. Start with the right question: “Which model is better?” is not a ranking problem

In engineering practice, “Which model is the strongest?” is not a useful question, because “strong” must be tied to a specific task. The answerable question is: “Within my business scenario, which type of model can pass my acceptance cases under a given latency and cost budget?”

Public evaluation leaderboards have three structural limitations that must be addressed before model selection:

1. Task distribution mismatch. The distribution of tasks in a leaderboard almost never matches real business traffic. A leaderboard dominated by general knowledge or academic reasoning questions has limited predictive value for a mixed scenario such as enterprise internal knowledge Q&A, coding assistance, and long-contract summarization.
2. Average scores hide the long tail. Business usability is often determined by the worst 5% of requests. Two models with the same average score can behave very differently in production: the one with higher variance will create more rework and more complaints.
3. Snapshot timing versus rapid iteration. Leading models and their inference services are continuously updated. The conclusion “Model A was better six months ago” usually has a validity period of only one quarter.

The correct approach is to treat model selection as a continuously regression-tested engineering asset: fixed task set, fixed scoring rules, fixed baselines, and a full rerun whenever a candidate model or prompt changes.

2. First step: translate the business scenario into a capability weight table

The first deliverable of model selection is not a model name. It is a capability weight table. For three typical scenarios, the weight differences are significant:

Capability dimension	Key measurable metric	Code assistant	Enterprise knowledge Q&A	Long-document processing
Reasoning and factual accuracy	Single-pass accuracy, hallucination rate, reasonable refusal rate	Medium	High	Medium-high
Code capability	Compilation pass rate, unit test pass rate, manual edit volume	High	Low	Low
Effective long context	Long-context needle retrieval accuracy, multi-hop recall	Medium	High	High
Instruction following / structured output	Schema compliance rate, tool-call correctness	High	Medium	Medium
Engineering performance	TTFT, P95 latency, error rate	High for completion	High for chat	Medium, can be asynchronous
Cost and operations	Cost per task, controllability	High	High	Medium

The value of this table is that it turns “debate” into “allocation”. If one model leads by eight points on long-document capability but fails the latency requirement, whether that tradeoff is acceptable becomes a weight-based decision rather than a subjective discussion.

3. Deep dive into the six dimensions

3.1 Reasoning and knowledge accuracy

Two types of questions must be distinguished.

Parameterized-knowledge questions — facts, definitions, common knowledge — test whether the model’s memory is reliable. The main risk is confident fabrication.

Multi-step reasoning questions — conditional deduction, code semantic analysis, numerical computation — test chain stability. The main risk is drift at some intermediate step.

Evaluation points:

• Hallucination rate and refusal rate should be examined together. A high refusal rate can reduce hallucinations, but it destroys usability. A business-acceptable refusal ceiling must be defined.
• For knowledge-based tasks, the correct approach is to externalize answers into the retrieval layer and use the model for understanding and summarization. Knowledge stored in model parameters will eventually become outdated and cannot be easily cited.
• A private question set of 100 to 300 questions is usually enough to produce statistically meaningful differentiation, as long as the questions come from real, anonymized traffic rather than being adapted from public benchmarks.

3.2 Code capability: test at least five layers

Code capability is often treated as a single score, but it should be split into at least five layers:

1. Snippet generation: complete a function body from comments and signatures. This tests syntax and basic correctness.
2. Low-latency completion: used inside an IDE, usually with a latency budget of hundreds of milliseconds. First-pass acceptance matters more than maximum quality, because users may ignore stale suggestions while typing.
3. Repository-level understanding: locate definitions and call sites across files and modules. This tests retrieval support, not just context window size.
4. Defect fixing and test generation: requires the model to understand failing test cases, locate the root cause, and produce a runnable patch. This tests multi-step tool orchestration.
5. Refactoring and migration: requires large-scale consistency changes and diff-level precision.

Recommended metric combination: compilation pass rate, unit test pass rate, accepted patch ratio, and manual edit volume per 1,000 lines of code.

3.3 Long context: context window size is not effective context

The advertised context length and the usable context length can differ significantly. Effective context should be defined as the maximum token length at which the model’s accuracy does not drop beyond a business tolerance threshold, such as five points, compared with short-context performance.

Three phenomena must be tested:

• Needle retrieval: insert factual points into a long document and test whether the model can locate and cite them accurately.
• Position bias: place the same fact at the beginning, middle, and end of the context. Accuracy should not differ significantly. Middle-of-context degradation is common.
• Multi-hop questions and distractors: if similar but incorrect information exists in the document, does the model get misled? For long-document summarization, information coverage and redundancy should also be measured.

Engineering implication: long context is a cost amplifier. Doubling the context length usually increases input tokens and latency at the same time. A more reasonable architecture is “retrieval + reranking + precise snippet injection + long-context fallback”, rather than stuffing all documents into one request.

3.4 Instruction following and structured output

This is often underestimated, but it has the largest impact on system integration. When model output is consumed by downstream programs, any format deviation is a failure.

• Schema compliance rate: frequency of missing fields, type errors, enum violations, and extra fields. It is recommended to enforce JSON mode or structured output constraints at the calling side, and add secondary validation and bounded retries at the gateway layer.
• Tool/function-calling correctness: single-tool selection accuracy, parameter extraction completeness, whether user intervention is required during multi-turn orchestration, and whether infinite loops occur.
• Constraint following: maximum length, language, tone, template structure, forbidden topics. Negative test cases, where the model is required not to do something, are more discriminative than positive cases.
• Reproducibility: even with fixed temperature and random seed, many services still cannot guarantee token-by-token identical outputs. Evaluation should sample multiple times and report both mean and variance.

3.5 Engineering performance and stability

Chat experience is determined by first-token latency. Batch throughput is determined by total duration. These should be measured separately.

• TTFT, time to first token: directly affects interactive feel, especially for streaming output.
• TPOT, time per output token: affects the reading experience of long answers and whether the response appears stalled.
• P50/P95/P99 total duration: averages are misleading. The long tail is the basis for capacity planning.
• Error rate and error-code distribution: distinguish retryable errors, such as rate limiting and timeouts, from non-retryable errors, such as content policy violations and invalid parameters. This determines retry and fallback strategies.
• Quota and rate limits: concurrency limits, token quotas per minute, and burst capacity should be measured through load testing rather than assumed from documentation.

3.6 Cost and operations

Cost should be calculated as cost per task, not unit price alone.

# Python-like pseudocode. Price unit: currency per 1,000 tokens.
def estimate_cost(daily_calls, in_tok, out_tok,
                  price_in, price_out, cache_ratio=0.0, cached_discount=0.8):
    hit = in_tok * cache_ratio          # Tokens covered by prompt cache
    miss = in_tok - hit
    per_call = (miss * price_in
                + hit * price_in * (1 - cached_discount)
                + out_tok * price_out) / 1000
    return per_call, per_call * daily_calls * 30

Cost-reduction measures, ordered by cost-effectiveness:

1. Compress prompts and remove redundant content.
2. Reuse stable prefixes through prompt caching.
3. Route simple tasks to smaller models.
4. Trim retrieved context.
5. Batch asynchronous tasks.
6. Use distilled or fine-tuned smaller models for high-frequency fixed tasks.
7. Consider more expensive models only after the above options are exhausted.

4. Building an in-house evaluation system: a practical evaluation framework

4.1 Task set construction

Sample production logs in a stratified manner, covering high-frequency, long-tail, and high-risk cases. Anonymize the data and freeze a version. For each question, label the reference answer, scoring points, and allowed variants. The task set must be version-controlled; otherwise regression testing is meaningless.

4.2 Evaluation harness

# Cross-model evaluation using the same task set, prompts, and scoring rules.
def run_eval(cases, client, grader, repeats=3, temperature=0.2):
    report = []
    for case in cases:
        scores, lats, costs = [], [], 0.0
        for _ in range(repeats):
            t0 = now()
            r = client.chat(case.messages, temperature=temperature,
                            max_tokens=case.max_tokens, stream=False)
            lats.append(now() - t0)
            costs += r.usage.cost
            scores.append(grader(case, r.text))   # Returns a score from 0 to 1
        report.append({
            "id": case.id,
            "score": mean(scores),        # Average quality
            "score_std": stdev(scores),   # Output stability
            "p95_latency": percentile(lats, 95),
            "cost": costs / repeats,
            "samples": keep_worst(scores),  # Keep worst samples for root-cause analysis
        })
    return report

4.3 Scoring rules

Rule-based scoring should be preferred whenever possible, for example: schema validation pass rate, unit test pass rate, key entity hit rate, and whether citations are traceable. These scores can be automated and regression-tested.

Subjective quality, such as wording and logical organization, can be evaluated through dual blind review with consistency checks, or by using a stable strong model as a reviewer with periodic manual calibration.

The final output should be a decision table containing dimension scores, weighted total score, and whether cost and latency constraints are satisfied.

5. Production implementation: abstraction layer, routing, and fallback

5.1 Unified model abstraction layer

Treat models as replaceable implementations. Business code should depend only on a unified interface. This is the only reliable way to preserve bargaining power and enable fast switching.

5.2 Tiered routing and fallback

def ask(task, messages, budget_ms=3000):
    tier = router.pick(task)   # Based on task type, context length, and historical quality statistics
    for model in [tier.primary, tier.fallback_small, tier.backup]:
        try:
            with timeout(budget_ms):
                r = model.chat(messages, temperature=task.temperature,
                               max_tokens=task.max_tokens, stream=task.stream)
            if validate(r, task.schema):     # Dual validation: schema and business rules
                metrics.record(model.id, latency=elapsed(), cost=r.usage.cost)
                return r
        except (ModelError, TimeoutError) as e:
            metrics.record(model.id, error=str(e))
            continue
    return safe_default(task)   # Fallback: cached answer / rule engine / human ticket

Related operational requirements:

• End-to-end instrumentation for prompt version, model version, token usage, latency, and scores.
• Quality dashboards segmented by model.
• Monthly cost allocation.
• Fault drills: actively disconnect the primary model and verify whether the fallback path actually works.

If an enterprise’s evaluation and inference pipelines already run on Baidu AI Cloud, prompt versions, task sets, and scoring results can be stored in object storage for long-term archival. Message queues can carry asynchronous batch evaluation and inference workloads. Logging services and monitoring alerts can form an observability loop for token consumption, latency percentiles, and error rates. These are model-agnostic infrastructure capabilities and are not tied to a specific model.

6. Non-functional constraints: often overlooked, but frequently decisive

• Data compliance and residency: whether training or inference data leaves the environment, whether it is used to improve services, log retention period, and whether data retention can be disabled.
• Deployment form: public cloud APIs, dedicated instances, and private deployment have very different cost curves. High-sensitivity scenarios often exclude candidates during compliance review.
• Version locking and change management: server-side model upgrades can silently change behavior. Version numbers must be locked, the same regression set must be run before and after changes, and prompt versions should be included in release records together with model versions.
• Portability: the more deeply prompt techniques depend on proprietary features, the higher the switching cost. Periodically rerun prompts on candidate models to assess migration cost.

7. Seven common pitfalls

1. Making decisions directly from public leaderboards without using a private task set.
2. Looking only at average scores without considering variance and P95 latency.
3. Treating context window size as the only indicator of long-document capability.
4. Optimizing total duration while ignoring first-token latency, which damages interactive experience.
5. Tuning prompts only on a single model, leading to de facto vendor lock-in.
6. Selecting a model once and never running regression tests, causing quality degradation to go unnoticed after model updates.
7. Comparing only unit price while ignoring retries, cache misses, and cost amplification from very long contexts.

8. Two-week rollout plan and action checklist

• Days 1–2: inventory scenarios, produce a capability weight table and hard constraints, such as compliance, latency, and cost ceilings.
• Days 3–5: build a stratified task set of 150 to 300 questions from historical traffic, and complete anonymization and labeling.
• Days 6–8: implement a unified abstraction layer and an evaluation harness. Prioritize rule-based scoring.
• Days 9–10: evaluate three to four candidate models and produce a decision table.
• Days 11–12: integrate routing and fallback, conduct fault drills, and run load tests.
• Days 13–14: perform a gray release, establish quality and cost dashboards, and set up a monthly regression mechanism.

9. Conclusion

Model capabilities are still changing rapidly. Any conclusion that “a certain model is best in a certain year” has a very short half-life.

Sustainable competitiveness comes from three things:

1. A capability weight table aligned with the business.
2. A private evaluation set based on real traffic.
3. An engineering architecture that makes models pluggable.

Once these three are in place, switching models changes from a high-risk decision into a routine release.