How to Scientifically Evaluate and Select Large Language Models
Choosing a large language model (LLM) for production rarely goes as expected: the model everyone assumes will win often loses ground, while an unassuming candidate pulls ahead. This article explains why LLM selection has no absolute "best" answer and lays out a reproducible evaluation framework built on three layers: unified access, multi-dimensional evaluation, and decision/regression. It covers configuration for fair comparison, building evaluation sets from real business data, scoring across dimensions such as factuality, reasoning, instruction following, code generation, long-text handling, and safety, and turning results into decisions through statistical testing, blind evaluation, weighted ranking, and cost accounting. A deployment checklist and a continuous regression loop help teams keep selection decisions data-driven as models keep evolving.
1. Background: Why Selection Results Rarely Match Expectations
In real selection work, the following scenario is common:
- The team forms a prior based on past impressions and assumes a certain model should rank near the top.
- Several candidate models are wired into the same gateway and tested against one shared set of questions.
- The results defy expectations: a favored model loses points on dimensions such as factuality, while a less prominent model takes a clear lead—by a margin far beyond random fluctuation.
Such a reversal usually means the test was not wrong. Instead, it confirms a selection principle: there is no absolute “best” model; there is only the one that best matches a specific scenario, task distribution, and set of constraints. Three factors drive this:
- Subjective impressions are biased. Demos, a few impressive answers, and marketing claims all shape how we judge a model’s overall capability.
- Public leaderboards have limitations. Questions may already be “seen” by a model, so results cannot separate genuine reasoning from memorization. A leaderboard’s task distribution and scoring rules may also differ from your own business.
- Single outputs are stochastic. Sampling parameters, phrasing, and question order can all change results, so a handful of questions cannot represent overall capability.
Selection therefore needs an evaluation process with unified access, multi-dimensional assessment, and reproducible results. Such a process does not depend on a preference for any brand; the data drives the conclusion.
2. Overall Approach: Three Design Principles
A trustworthy model evaluation process should satisfy three principles:
- Unified: Candidate models face the same evaluation set, prompt template, and sampling parameters. Changing only the model variable keeps comparisons fair.
- Reproducible: Record the model version, evaluation set version, run parameters, and scoring rules. Anyone rerunning under the same conditions should get approximately the same results.
- Close to the business: Questions come from real usage scenarios, and metrics map directly to business goals rather than chasing a good-looking “overall score.”
On top of these, the evaluation splits into three layers, each supporting the one above it:
| Layer | Core content | Problem it solves |
|---|---|---|
| Unified access layer | Unified interface, fixed parameters, key and retry management | Eliminates vendor differences and keeps test conditions consistent |
| Multi-dimensional evaluation layer | Evaluation set construction, dimension design, metric calculation | Answers “how each model actually performs across capability dimensions” |
| Decision and regression layer | Weighted ranking, cost and latency assessment, regression testing | Turns evaluation results into selection decisions and ongoing protection |
3. Unified Access: Removing Differences for a Fair Comparison
The fairness of an evaluation starts with consistency in how models are called. Interfaces from different vendors often differ in authentication, request and response structure, supported parameters, and error codes. Integrating each one directly can lead to missing parameters, inconsistent timeout settings, or parsing failures that ultimately distort comparisons. A common approach is to introduce a unified access layer that presents a consistent interface upward and adapts to each vendor’s differences downward.
3.1 Metadata Configuration: Describing Differences as Data
Use a single configuration file to describe the gateway address, candidate model list, and unified evaluation parameters:
gateway:base_url: ${LLM_GATEWAY_BASE_URL} # Unified access endpointapi_key: ${LLM_GATEWAY_API_KEY} # Read from environment variables or a secrets manager; never commit to the repotimeout_seconds: 60models:- {id: "baseline", label: "Baseline model", supports_tools: true}- {id: "candidate-a", label: "Candidate A", supports_tools: false}- {id: "candidate-b", label: "Candidate B", supports_tools: true}eval:prompt_template: "prompts/eval.txt"temperature: 0.0top_p: 1.0max_tokens: 1024max_retries: 3
When integrating, use the model identifiers returned by the platform or gateway you use. Inject keys only through environment variables or a secrets manager to avoid hardcoding and accidental commits.
3.2 Unified Call Entry and Retries
The unified call entry fills in the request body, sets timeouts, handles transient errors, and performs bounded retries. The example below is illustrative and meant to clarify the structure; parameter names and field meanings follow the general interface convention:
import jsonimport osimport timedef complete(gateway, model_id, messages, params, cfg):"""Unified call entry: hide vendor differences, fix test conditions, and record latency"""payload = {"model": model_id,"messages": messages,"temperature": cfg["eval"]["temperature"],"top_p": cfg["eval"]["top_p"],"max_tokens": cfg["eval"]["max_tokens"],}# Pass this parameter only when a candidate declares support, to avoid unfair comparisonif cfg["models_by_id"][model_id].get("supports_tools"):payload["tools"] = params.get("tools", [])for attempt in range(cfg["eval"]["max_retries"] + 1):started = time.perf_counter()try:resp = gateway.chat_completion(payload,headers={"Authorization": f"Bearer {os.environ['LLM_GATEWAY_API_KEY']}"},timeout=cfg["gateway"]["timeout_seconds"],)return {"text": resp["choices"][0]["message"].get("content", ""),"usage": resp.get("usage", {}),"latency_ms": round((time.perf_counter() - started) * 1000),"attempts": attempt + 1,}except TimeoutError:# Retry only on transient errors such as timeouts; auth and parameter errors are raised immediately and alertedif attempt == cfg["eval"]["max_retries"]:raisetime.sleep(2 ** attempt) # Exponential backoff
3.3 Prompt Management
- Same template: All candidate models share one prompt template; only the task content changes.
- Fixed temperature and top_p: For example, set temperature=0.0 and top_p=1.0 uniformly to reduce fluctuation from randomness. When a model genuinely does not support a parameter, record it and note it in the report rather than making an implicit change.
- Output and time allowances: Reserve sufficient output length and response time for thinking-oriented and instruction-following models separately, so truncation or timeouts do not skew conclusions.
4. Building the Evaluation Set: Keeping Questions Close to the Business
The quality of the evaluation set directly determines whether conclusions are worth referencing.
- Real sources: Prefer drawing questions from desensitized production logs, support tickets, and historical Q&A, and set the proportion of each task type according to real traffic distribution.
- Avoid contamination: Use your own data and internally rewritten questions to reduce the chance that a model has “seen the answer.” Public test sets are better as a reference for longitudinal trends and should not be the sole basis.
- Matched scale: Prepare at least a hundred questions per task category, each annotated with scoring rules; add more questions for critical dimensions.
- Standard fields: Each evaluation record should consistently contain the following fields, shared by calling, scoring, and review:
{"id": "fact-0231","category": "factual","input": "Answer the question based on the following material...","reference": "Reference answer or the corresponding passage from authoritative sources","rubric": ["Contains no content that contradicts the material", "States uncertainty explicitly when it cannot be confirmed"],"context_tokens": 6800}
5. Multi-Dimensional Evaluation: Beyond the “Overall Score”
Capability assessment is best split into multiple dimensions, scored separately and then weighted by actual business needs.
5.1 Factuality
Factuality is the core baseline for high-value scenarios and a key focus of selection: one plausible-looking wrong answer can directly mislead a decision. Assess two abilities together:
- What it knows. Ask factual questions closed-book and check answers against authoritative sources.
- What it doesn’t know. Include questions whose answers do not exist or are not covered by the material, and see whether the model honestly states uncertainty instead of fabricating.
When possible, add open-book scenarios that answer from provided material to check whether responses stay faithful to the source. This is exactly why some models shine in demos yet lose points in factuality tests.
An illustrative scoring rule:
def score_factual(case, text):"""Illustrative factual scoring: correctness, hallucination, refusal, and over-refusal"""lower = text.lower()if "uncertain" in lower or "not provided in the material" in lower:# Refusing when an answer exists is over-refusal; stating uncertainty when no answer exists should scorereturn 0.0 if case.get("has_answer") else 1.0# Score 1 only when the answer matches the key facts in the reference and contains no conflicting conclusionmatched = sum(k in text for k in case.get("key_facts", []))conflict = any(c in text for c in case.get("contradictions", []))if matched >= max(1, len(case.get("key_facts", [])) // 2) and not conflict:return 1.0return 0.0 # A fabricated answer is judged as a hallucination
| Metric | Meaning | Healthy signal |
|---|---|---|
| Factual accuracy | Proportion of correct answers | Higher is better |
| Hallucination rate | Proportion of fabricated, incorrect information | Closer to 0 is better |
| Correct refusal rate | Proportion of no-answer questions where uncertainty is honestly stated | Higher is better |
| Over-refusal rate | Proportion of answerable questions that are wrongly refused | Lower is better |
5.2 Reasoning and Calculation
- Multi-step reasoning: Questions involving conditional reasoning, causal analysis, and multi-hop reasoning, checking chain completeness.
- Math and calculation: Requests for intermediate steps make it easier to locate where an error occurs.
- Logical consistency: In multi-turn follow-ups, check whether the model contradicts itself and whether it can spot and correct errors promptly.
5.3 Instruction Following
- Explicit constraints: Output format, word limits, forbidden words, required elements, and so on.
- Implicit constraints: Role setting, tone and style, intended audience, and so on.
- Scoring method: Use programs to automatically validate quantifiable constraints (e.g., whether the output is valid JSON, whether word count is within range); leave subjective constraints to human or rule-model judgment.
5.4 Code Generation and Understanding
- Executable verification: Use a sandbox built on a container platform and measure pass rate with unit tests.
- Engineering capability: Assess defect localization, code explanation, and cross-file modification.
- Security: Check for common issues such as injection vulnerabilities and hardcoded keys.
5.5 Long-Text and Document Processing
- Localization: Retrieving specified information within a long document.
- Integration: Aggregating and comparing across sections.
- Compression: Producing summaries that omit no key information.
- Cost coupling: Long-context capability directly affects cost (see 6.3), so record context length during evaluation.
5.6 Structured Output and Tool Calling
In enterprise applications, model output is often consumed by downstream systems. Evaluation should focus on:
- Format compliance: Whether the output conforms to the agreed data structure.
- Field completeness: Whether required fields are present and values fall within enum ranges.
- Tool-call correctness: Whether the right tool is chosen, parameters are correct, and the model can self-correct after an error return.
5.7 Language and Domain Adaptation
- Language-specific tasks: Such as Chinese formal document writing, professional terminology translation, and legal and regulatory comprehension.
- Domain adaptation: Use your own questions—such as explanations of internal terminology and business consultation Q&A—to assess fit.
- Reasonableness: A model strong in general capability is not necessarily as strong in a vertical domain.
5.8 Safety and Compliance
- Risk identification: Whether the model appropriately refuses requests involving personal privacy or illegal/non-compliant content.
- Prompt injection: Whether instructions embedded in user input can be resisted.
- Alignment strength: Safety policies vary in strictness across models; choose based on the business risk level.
5.9 Engineering and Cost Metrics
- Time to first token and generation speed: Separately measure the wait users perceive under streaming responses and the steady-state output speed.
- Latency stability: Record P50, P95, and P99 percentile latency, focusing on tail behavior rather than the average alone.
- Success rate and rate limiting: Track the retry ratio and rate-limit hit rate.
- Context limit: Record the model’s effective context length to judge whether it meets long-document needs.
For easy comparison, register each dimension’s metrics and collection method uniformly:
| Dimension | Key metrics | Collection method |
|---|---|---|
| Factuality | Factual accuracy, hallucination rate, refusal accuracy | Rule checks + human spot checks |
| Reasoning and calculation | Reasoning accuracy, step completeness | Automated checks + human scoring |
| Instruction following | Constraint satisfaction rate | Mainly program checks |
| Code generation | Unit test pass rate | Sandbox execution |
| Long-text processing | Key-information recall | Rule checks + human scoring |
| Structured output | Format compliance, field completeness | Program checks |
| Language and domain adaptation | Business question pass rate | Human scoring |
| Safety and compliance | Risk-request refusal rate | Rule checks + human review |
| Engineering and cost | P95 latency, cost per task | Call log statistics |
6. Making Conclusions Trustworthy: Statistics, Blind Evaluation, and Cost Accounting
6.1 Repeated Sampling and Statistical Judgment
A single call cannot support a ranking. Recommended:
- Sample each question multiple times, take the average, and observe score variance.
- Compute a confidence interval using the same paired comparison method to judge whether the gap between two models is significant.
- When the gap falls within the confidence interval, treat the models as “comparable in performance” rather than declaring a winner.
6.2 Double-Blind Comparison and Position Rotation
- Anonymized display: When scoring manually, hide model names and label them only “Plan A / Plan B” to avoid brand bias.
- Position rotation: Randomly swap the left/right order to eliminate positional preference.
- Two-person review: Key questions are scored independently by two people; disagreements go to a third for adjudication.
6.3 Total Cost Estimation
Model inference is usually billed separately for input and output tokens. The average cost of a single task is:
Cost per task = avg input tokens / 1000 × input unit price+ avg output tokens / 1000 × output unit price
Because tokenization can differ across models, base the calculation on the usage actually returned by the call. Converting cost to the granularity of “completing one real business request” is more meaningful than comparing unit prices alone.
If you build your own inference environment, also factor in resource usage, operations effort, and elastic scaling capability. Model services can also use object storage to store evaluation data and reports, and logging services with monitoring and alerting to continuously track call quality.
6.4 Weighted Ranking and Short-Board Constraints
The composite score should not be a simple average. Weight each dimension by business importance—for example, a support scenario with high factuality requirements:
| Dimension | Weight |
|---|---|
| Factuality | 30% |
| Instruction following | 15% |
| Reasoning and calculation | 15% |
| Language and domain adaptation | 15% |
| Long-text processing | 10% |
| Safety and compliance | 10% |
| Code generation | 5% |
Also recommend a “short-board veto” rule: if a model is significantly below the other candidates on the highest-weight dimension, use it cautiously even if its composite score leads. A model that is balanced and strong across all dimensions—its advantage is usually no accident.
7. Deployment Checklist and Continuous Regression
After evaluation, check process completeness against this checklist:
- All candidate models are integrated through the unified access layer, with consistent, recorded parameters;
- The evaluation set comes from real scenarios and is kept isolated from public test sets;
- Every dimension has clear scoring rules and enough samples;
- Key conclusions pass repeated sampling and significance judgment;
- Manual scoring uses blind evaluation and position rotation;
- Latency distribution, success rate, and cost per task are included in the decision;
- Evaluation scripts, data, and reports are all archived.
Models iterate quickly, so a one-time evaluation cannot guarantee long-term suitability. Recommended: wire the evaluation pipeline into continuous integration. Automatically trigger regression tests when onboarding a new candidate model, adjusting prompts, or when online performance anomalies appear; online quality issues found through monitoring and alerting can flow back to supplement the evaluation set, forming a loop of “evaluate → deploy → monitor → re-evaluate.” Platforms such as Baidu AI Cloud typically provide general-purpose cloud capabilities like container platforms, logging services, and monitoring/alerting, which can serve as the underlying support for this evaluation and operations system.
8. Conclusion
When an evaluation ranking reverses subjective expectations, it highlights the value of a unified evaluation process: it moves selection from “gut feeling and marketing” to “letting data speak.” Unified access ensures fairness, multi-dimensional metrics ensure completeness, statistics and blind evaluation ensure trustworthiness, and cost and latency ensure it is deployable. Fix this process in place and run it continuously, and the team can quickly reach well-grounded conclusions for any new model while keeping the quality of online services under control.
