How to Scientifically Evaluate and Select Large Language Models

    Choosing a large language model (LLM) for production rarely goes as expected: the model everyone assumes will win often loses ground, while an unassuming candidate pulls ahead. This article explains why LLM selection has no absolute "best" answer and lays out a reproducible evaluation framework built on three layers: unified access, multi-dimensional evaluation, and decision/regression. It covers configuration for fair comparison, building evaluation sets from real business data, scoring across dimensions such as factuality, reasoning, instruction following, code generation, long-text handling, and safety, and turning results into decisions through statistical testing, blind evaluation, weighted ranking, and cost accounting. A deployment checklist and a continuous regression loop help teams keep selection decisions data-driven as models keep evolving.

    1. Background: Why Selection Results Rarely Match Expectations

    In real selection work, the following scenario is common:

    1. The team forms a prior based on past impressions and assumes a certain model should rank near the top.
    2. Several candidate models are wired into the same gateway and tested against one shared set of questions.
    3. The results defy expectations: a favored model loses points on dimensions such as factuality, while a less prominent model takes a clear lead—by a margin far beyond random fluctuation.

    Such a reversal usually means the test was not wrong. Instead, it confirms a selection principle: there is no absolute “best” model; there is only the one that best matches a specific scenario, task distribution, and set of constraints. Three factors drive this:

    • Subjective impressions are biased. Demos, a few impressive answers, and marketing claims all shape how we judge a model’s overall capability.
    • Public leaderboards have limitations. Questions may already be “seen” by a model, so results cannot separate genuine reasoning from memorization. A leaderboard’s task distribution and scoring rules may also differ from your own business.
    • Single outputs are stochastic. Sampling parameters, phrasing, and question order can all change results, so a handful of questions cannot represent overall capability.

    Selection therefore needs an evaluation process with unified access, multi-dimensional assessment, and reproducible results. Such a process does not depend on a preference for any brand; the data drives the conclusion.

    2. Overall Approach: Three Design Principles

    A trustworthy model evaluation process should satisfy three principles:

    • Unified: Candidate models face the same evaluation set, prompt template, and sampling parameters. Changing only the model variable keeps comparisons fair.
    • Reproducible: Record the model version, evaluation set version, run parameters, and scoring rules. Anyone rerunning under the same conditions should get approximately the same results.
    • Close to the business: Questions come from real usage scenarios, and metrics map directly to business goals rather than chasing a good-looking “overall score.”

    On top of these, the evaluation splits into three layers, each supporting the one above it:

    Layer Core content Problem it solves
    Unified access layer Unified interface, fixed parameters, key and retry management Eliminates vendor differences and keeps test conditions consistent
    Multi-dimensional evaluation layer Evaluation set construction, dimension design, metric calculation Answers “how each model actually performs across capability dimensions”
    Decision and regression layer Weighted ranking, cost and latency assessment, regression testing Turns evaluation results into selection decisions and ongoing protection

    3. Unified Access: Removing Differences for a Fair Comparison

    The fairness of an evaluation starts with consistency in how models are called. Interfaces from different vendors often differ in authentication, request and response structure, supported parameters, and error codes. Integrating each one directly can lead to missing parameters, inconsistent timeout settings, or parsing failures that ultimately distort comparisons. A common approach is to introduce a unified access layer that presents a consistent interface upward and adapts to each vendor’s differences downward.

    3.1 Metadata Configuration: Describing Differences as Data

    Use a single configuration file to describe the gateway address, candidate model list, and unified evaluation parameters:

    1. gateway:
    2. base_url: ${LLM_GATEWAY_BASE_URL} # Unified access endpoint
    3. api_key: ${LLM_GATEWAY_API_KEY} # Read from environment variables or a secrets manager; never commit to the repo
    4. timeout_seconds: 60
    5. models:
    6. - {id: "baseline", label: "Baseline model", supports_tools: true}
    7. - {id: "candidate-a", label: "Candidate A", supports_tools: false}
    8. - {id: "candidate-b", label: "Candidate B", supports_tools: true}
    9. eval:
    10. prompt_template: "prompts/eval.txt"
    11. temperature: 0.0
    12. top_p: 1.0
    13. max_tokens: 1024
    14. max_retries: 3

    When integrating, use the model identifiers returned by the platform or gateway you use. Inject keys only through environment variables or a secrets manager to avoid hardcoding and accidental commits.

    3.2 Unified Call Entry and Retries

    The unified call entry fills in the request body, sets timeouts, handles transient errors, and performs bounded retries. The example below is illustrative and meant to clarify the structure; parameter names and field meanings follow the general interface convention:

    1. import json
    2. import os
    3. import time
    4. def complete(gateway, model_id, messages, params, cfg):
    5. """Unified call entry: hide vendor differences, fix test conditions, and record latency"""
    6. payload = {
    7. "model": model_id,
    8. "messages": messages,
    9. "temperature": cfg["eval"]["temperature"],
    10. "top_p": cfg["eval"]["top_p"],
    11. "max_tokens": cfg["eval"]["max_tokens"],
    12. }
    13. # Pass this parameter only when a candidate declares support, to avoid unfair comparison
    14. if cfg["models_by_id"][model_id].get("supports_tools"):
    15. payload["tools"] = params.get("tools", [])
    16. for attempt in range(cfg["eval"]["max_retries"] + 1):
    17. started = time.perf_counter()
    18. try:
    19. resp = gateway.chat_completion(
    20. payload,
    21. headers={"Authorization": f"Bearer {os.environ['LLM_GATEWAY_API_KEY']}"},
    22. timeout=cfg["gateway"]["timeout_seconds"],
    23. )
    24. return {
    25. "text": resp["choices"][0]["message"].get("content", ""),
    26. "usage": resp.get("usage", {}),
    27. "latency_ms": round((time.perf_counter() - started) * 1000),
    28. "attempts": attempt + 1,
    29. }
    30. except TimeoutError:
    31. # Retry only on transient errors such as timeouts; auth and parameter errors are raised immediately and alerted
    32. if attempt == cfg["eval"]["max_retries"]:
    33. raise
    34. time.sleep(2 ** attempt) # Exponential backoff

    3.3 Prompt Management

    • Same template: All candidate models share one prompt template; only the task content changes.
    • Fixed temperature and top_p: For example, set temperature=0.0 and top_p=1.0 uniformly to reduce fluctuation from randomness. When a model genuinely does not support a parameter, record it and note it in the report rather than making an implicit change.
    • Output and time allowances: Reserve sufficient output length and response time for thinking-oriented and instruction-following models separately, so truncation or timeouts do not skew conclusions.

    4. Building the Evaluation Set: Keeping Questions Close to the Business

    The quality of the evaluation set directly determines whether conclusions are worth referencing.

    • Real sources: Prefer drawing questions from desensitized production logs, support tickets, and historical Q&A, and set the proportion of each task type according to real traffic distribution.
    • Avoid contamination: Use your own data and internally rewritten questions to reduce the chance that a model has “seen the answer.” Public test sets are better as a reference for longitudinal trends and should not be the sole basis.
    • Matched scale: Prepare at least a hundred questions per task category, each annotated with scoring rules; add more questions for critical dimensions.
    • Standard fields: Each evaluation record should consistently contain the following fields, shared by calling, scoring, and review:
    1. {
    2. "id": "fact-0231",
    3. "category": "factual",
    4. "input": "Answer the question based on the following material...",
    5. "reference": "Reference answer or the corresponding passage from authoritative sources",
    6. "rubric": ["Contains no content that contradicts the material", "States uncertainty explicitly when it cannot be confirmed"],
    7. "context_tokens": 6800
    8. }

    5. Multi-Dimensional Evaluation: Beyond the “Overall Score”

    Capability assessment is best split into multiple dimensions, scored separately and then weighted by actual business needs.

    5.1 Factuality

    Factuality is the core baseline for high-value scenarios and a key focus of selection: one plausible-looking wrong answer can directly mislead a decision. Assess two abilities together:

    1. What it knows. Ask factual questions closed-book and check answers against authoritative sources.
    2. What it doesn’t know. Include questions whose answers do not exist or are not covered by the material, and see whether the model honestly states uncertainty instead of fabricating.

    When possible, add open-book scenarios that answer from provided material to check whether responses stay faithful to the source. This is exactly why some models shine in demos yet lose points in factuality tests.

    An illustrative scoring rule:

    1. def score_factual(case, text):
    2. """Illustrative factual scoring: correctness, hallucination, refusal, and over-refusal"""
    3. lower = text.lower()
    4. if "uncertain" in lower or "not provided in the material" in lower:
    5. # Refusing when an answer exists is over-refusal; stating uncertainty when no answer exists should score
    6. return 0.0 if case.get("has_answer") else 1.0
    7. # Score 1 only when the answer matches the key facts in the reference and contains no conflicting conclusion
    8. matched = sum(k in text for k in case.get("key_facts", []))
    9. conflict = any(c in text for c in case.get("contradictions", []))
    10. if matched >= max(1, len(case.get("key_facts", [])) // 2) and not conflict:
    11. return 1.0
    12. return 0.0 # A fabricated answer is judged as a hallucination
    Metric Meaning Healthy signal
    Factual accuracy Proportion of correct answers Higher is better
    Hallucination rate Proportion of fabricated, incorrect information Closer to 0 is better
    Correct refusal rate Proportion of no-answer questions where uncertainty is honestly stated Higher is better
    Over-refusal rate Proportion of answerable questions that are wrongly refused Lower is better

    5.2 Reasoning and Calculation

    • Multi-step reasoning: Questions involving conditional reasoning, causal analysis, and multi-hop reasoning, checking chain completeness.
    • Math and calculation: Requests for intermediate steps make it easier to locate where an error occurs.
    • Logical consistency: In multi-turn follow-ups, check whether the model contradicts itself and whether it can spot and correct errors promptly.

    5.3 Instruction Following

    • Explicit constraints: Output format, word limits, forbidden words, required elements, and so on.
    • Implicit constraints: Role setting, tone and style, intended audience, and so on.
    • Scoring method: Use programs to automatically validate quantifiable constraints (e.g., whether the output is valid JSON, whether word count is within range); leave subjective constraints to human or rule-model judgment.

    5.4 Code Generation and Understanding

    • Executable verification: Use a sandbox built on a container platform and measure pass rate with unit tests.
    • Engineering capability: Assess defect localization, code explanation, and cross-file modification.
    • Security: Check for common issues such as injection vulnerabilities and hardcoded keys.

    5.5 Long-Text and Document Processing

    • Localization: Retrieving specified information within a long document.
    • Integration: Aggregating and comparing across sections.
    • Compression: Producing summaries that omit no key information.
    • Cost coupling: Long-context capability directly affects cost (see 6.3), so record context length during evaluation.

    5.6 Structured Output and Tool Calling

    In enterprise applications, model output is often consumed by downstream systems. Evaluation should focus on:

    • Format compliance: Whether the output conforms to the agreed data structure.
    • Field completeness: Whether required fields are present and values fall within enum ranges.
    • Tool-call correctness: Whether the right tool is chosen, parameters are correct, and the model can self-correct after an error return.

    5.7 Language and Domain Adaptation

    • Language-specific tasks: Such as Chinese formal document writing, professional terminology translation, and legal and regulatory comprehension.
    • Domain adaptation: Use your own questions—such as explanations of internal terminology and business consultation Q&A—to assess fit.
    • Reasonableness: A model strong in general capability is not necessarily as strong in a vertical domain.

    5.8 Safety and Compliance

    • Risk identification: Whether the model appropriately refuses requests involving personal privacy or illegal/non-compliant content.
    • Prompt injection: Whether instructions embedded in user input can be resisted.
    • Alignment strength: Safety policies vary in strictness across models; choose based on the business risk level.

    5.9 Engineering and Cost Metrics

    • Time to first token and generation speed: Separately measure the wait users perceive under streaming responses and the steady-state output speed.
    • Latency stability: Record P50, P95, and P99 percentile latency, focusing on tail behavior rather than the average alone.
    • Success rate and rate limiting: Track the retry ratio and rate-limit hit rate.
    • Context limit: Record the model’s effective context length to judge whether it meets long-document needs.

    For easy comparison, register each dimension’s metrics and collection method uniformly:

    Dimension Key metrics Collection method
    Factuality Factual accuracy, hallucination rate, refusal accuracy Rule checks + human spot checks
    Reasoning and calculation Reasoning accuracy, step completeness Automated checks + human scoring
    Instruction following Constraint satisfaction rate Mainly program checks
    Code generation Unit test pass rate Sandbox execution
    Long-text processing Key-information recall Rule checks + human scoring
    Structured output Format compliance, field completeness Program checks
    Language and domain adaptation Business question pass rate Human scoring
    Safety and compliance Risk-request refusal rate Rule checks + human review
    Engineering and cost P95 latency, cost per task Call log statistics

    6. Making Conclusions Trustworthy: Statistics, Blind Evaluation, and Cost Accounting

    6.1 Repeated Sampling and Statistical Judgment

    A single call cannot support a ranking. Recommended:

    1. Sample each question multiple times, take the average, and observe score variance.
    2. Compute a confidence interval using the same paired comparison method to judge whether the gap between two models is significant.
    3. When the gap falls within the confidence interval, treat the models as “comparable in performance” rather than declaring a winner.

    6.2 Double-Blind Comparison and Position Rotation

    • Anonymized display: When scoring manually, hide model names and label them only “Plan A / Plan B” to avoid brand bias.
    • Position rotation: Randomly swap the left/right order to eliminate positional preference.
    • Two-person review: Key questions are scored independently by two people; disagreements go to a third for adjudication.

    6.3 Total Cost Estimation

    Model inference is usually billed separately for input and output tokens. The average cost of a single task is:

    1. Cost per task = avg input tokens / 1000 × input unit price
    2. + avg output tokens / 1000 × output unit price

    Because tokenization can differ across models, base the calculation on the usage actually returned by the call. Converting cost to the granularity of “completing one real business request” is more meaningful than comparing unit prices alone.

    If you build your own inference environment, also factor in resource usage, operations effort, and elastic scaling capability. Model services can also use object storage to store evaluation data and reports, and logging services with monitoring and alerting to continuously track call quality.

    6.4 Weighted Ranking and Short-Board Constraints

    The composite score should not be a simple average. Weight each dimension by business importance—for example, a support scenario with high factuality requirements:

    Dimension Weight
    Factuality 30%
    Instruction following 15%
    Reasoning and calculation 15%
    Language and domain adaptation 15%
    Long-text processing 10%
    Safety and compliance 10%
    Code generation 5%

    Also recommend a “short-board veto” rule: if a model is significantly below the other candidates on the highest-weight dimension, use it cautiously even if its composite score leads. A model that is balanced and strong across all dimensions—its advantage is usually no accident.

    7. Deployment Checklist and Continuous Regression

    After evaluation, check process completeness against this checklist:

    • All candidate models are integrated through the unified access layer, with consistent, recorded parameters;
    • The evaluation set comes from real scenarios and is kept isolated from public test sets;
    • Every dimension has clear scoring rules and enough samples;
    • Key conclusions pass repeated sampling and significance judgment;
    • Manual scoring uses blind evaluation and position rotation;
    • Latency distribution, success rate, and cost per task are included in the decision;
    • Evaluation scripts, data, and reports are all archived.

    Models iterate quickly, so a one-time evaluation cannot guarantee long-term suitability. Recommended: wire the evaluation pipeline into continuous integration. Automatically trigger regression tests when onboarding a new candidate model, adjusting prompts, or when online performance anomalies appear; online quality issues found through monitoring and alerting can flow back to supplement the evaluation set, forming a loop of “evaluate → deploy → monitor → re-evaluate.” Platforms such as Baidu AI Cloud typically provide general-purpose cloud capabilities like container platforms, logging services, and monitoring/alerting, which can serve as the underlying support for this evaluation and operations system.

    8. Conclusion

    When an evaluation ranking reverses subjective expectations, it highlights the value of a unified evaluation process: it moves selection from “gut feeling and marketing” to “letting data speak.” Unified access ensures fairness, multi-dimensional metrics ensure completeness, statistics and blind evaluation ensure trustworthiness, and cost and latency ensure it is deployable. Fix this process in place and run it continuously, and the team can quickly reach well-grounded conclusions for any new model while keeping the quality of online services under control.