Evaluating Large Language Models: From Subjective Impressions to Reproducible Benchmarks

    This article explains how to evaluate large language models in a reproducible, data-driven way instead of relying on subjective impressions. It shows how to turn vague claims such as "between two model generations" into testable hypotheses, break the idea of a "stronger" model down into observable capability dimensions, and build reproducible evaluation sets. It then combines three types of graders—rule-based scoring, model-as-judge, and human review—describes a minimal engineering framework, and offers guidance on statistical interpretation (controlling randomness, paired comparisons, confidence intervals, distributions, and the quality-latency-cost tradeoff). Finally, it lays out a four-stage rollout process from offline evaluation to full launch and a checklist of common pitfalls, so that model selection can be backed by data and continuously re-validated as models evolve.

    1. Introduction: Subjective Evaluation Has Value, but It Is Not the End Point

    When developers work with large language models in practice, they often make informal side-by-side comparisons: one general-purpose model generation “sits between two generations of a specialized model,” or a general-purpose model generation “is at least as good as a certain flagship model.” These judgments come from extensive real-world interaction and capture details that a single metric cannot—response speed, style of expression, error frequency, and so on—so they do have reference value. But using them directly as the basis for a technical decision runs into three problems:

    • Not reproducible: Evaluators summarize conclusions from memory, without a consistent set of questions, prompts, and parameters, so others cannot reproduce the results.
    • Not transferable: Evaluators differ in usage scenarios, questioning habits, and weighting preferences. The same model may behave very differently on “writing business code” versus “reading long documents,” and one person’s experience does not necessarily apply to another team.
    • Not sustainable: Models iterate frequently, so a single experience quickly goes stale and cannot answer engineering questions like “did the new version regress?”

    A more professional approach is to turn subjective impressions into testable hypotheses, then use an evaluation framework to confirm or refute them with data. For example, “reasoning ability falls between two generations” can be turned into a concrete question: on the same reasoning task set, does this version’s mean score and confidence interval fall between the two comparison models? “At least flagship-level” can be turned into: on the task set for the target business scenario, can this version simultaneously meet preset thresholds for quality, latency, and cost?

    This article addresses exactly that question and lays out a complete method, from metric definition and evaluation-set construction to scoring design, statistical analysis, and the rollout process.

    2. First, Break Down “Strong”: Evaluation Dimensions for LLMs

    “The model is stronger” is a vague claim that must be broken down into observable capability dimensions before it can be put into practice. The table below shows a common breakdown used in engineering:

    Capability dimension Typical focus Common evaluation methods
    Factual accuracy Whether facts are correct, whether it “confidently makes things up” Reference-answer comparison, fact-checking, manual spot checks
    Reasoning ability Multi-step derivation, logical judgment, mathematical computation Answer comparison, unit-test-style verification
    Instruction following Format, length, role, and taboo constraints Rule checks (regex, pattern matching)
    Structured output Whether it can reliably return data in an agreed structure Schema validation, field type checks
    Long-context usage Whether it can locate and integrate key information in very long input Multi-hop retrieval QA, long-document summarization
    Tool calling Whether arguments are correct, whether it can complete a multi-turn call loop Argument comparison, task completion rate
    Coding ability Whether code runs, whether it passes test cases Unit test execution results
    Multilingual and generation style Cross-language consistency, tone and readability Model review, human preference comparison
    Engineering performance and cost Time to first token, throughput, stability, billing model Load testing, production monitoring, cost accounting

    Two points deserve special emphasis here:

    1. Long-context ability should be judged by “effective utilization,” not by “how large the window is.” A common approach is “needle-in-a-haystack” testing—planting several key pieces of information in long text and testing whether the model can retrieve them accurately. A more meaningful test is “multi-hop reasoning,” where the answer requires connecting scattered pieces of information. Some models perform excellently on short inputs but show a clear drop in key-information recall as input grows. This “length decay” can only be surfaced with dedicated test cases.
    2. Speed, stability, and cost are also part of model capability. When users call a model “good to use,” that often includes fast responses, stable output, and few failed calls. A model with slightly higher quality but a significantly longer average response time and frequent rate limiting during peak hours may not be the better choice for a product serving end users.

    3. Building Reproducible Evaluation Sets

    Whether evaluation conclusions can be trusted depends first on the quality of the evaluation set.

    1. Sources: close to real business

    • Production samples: Sample from real business requests, covering typical usage, boundary inputs, and failure cases. Before use, data must be desensitized—remove personal information, keys, and other sensitive fields.
    • Historical problems: Turn production complaints, human fallbacks, and bad cases into fixed test cases; these often expose a model’s weaknesses most effectively.
    • Hand-crafted cases: Design specifically for rule constraints, extreme lengths, adversarial inputs, and similar scenarios to ensure coverage of the evaluation set.

    2. Scale: stratified by difficulty
    The total number of cases does not need to be large. 100–300 carefully selected questions often carry more information than thousands of randomly collected ones. It is recommended to stratify by difficulty—for example, roughly a 3:5:2 split across easy, medium, and hard—and to report each tier’s score separately. Looking only at the overall mean can easily mask a model’s clear weaknesses on hard tasks.

    3. Annotation: define “what counts as correct”
    Each case should include the input, a reference answer or scoring points (the rubric), hard-check rules, and metadata (difficulty, tags, weight). Scoring points should be as concrete as possible. Rather than writing “the answer should be accurate and clear,” break them into judgable items:

    • Does it contain all the key conclusions?
    • Are the data and dates accurate?
    • Does it point out risks the source text did not cover?
    • Does the format meet the requirements?

    4. Isolation: version control
    The evaluation set should be kept in a code repository and version-controlled alongside prompt templates and model configurations. Each evaluation run should record the corresponding version number to avoid directly comparing results from different times and different versions. If you reuse a public evaluation set, you also need to watch for data contamination: once test questions may have appeared in a model’s training data, the resulting scores can no longer faithfully reflect the model’s performance on new tasks. Using your own cases effectively reduces this risk.

    4. Designing a Scoring Mechanism

    Scoring is at the core of evaluation. In engineering practice, three types of graders are usually combined:

    (1) Rule-based scoring: objective and automatable

    Suitable for scenarios with a clear correct answer, such as:

    • Exact or containment matching: closed tasks like information extraction and classification;
    • Unit tests: run the model’s generated code in a sandbox and use the pass rate as the score. A sandbox here means an execution environment isolated from the main workflow, used to prevent malicious or dangerous operations in the code from affecting the host system;
    • Schema validation: output counts as passing only if it can be parsed into the agreed structure with correct field types. Example below:
    1. import json
    2. import re
    3. from typing import Optional, TypedDict
    4. class ExtractResult(TypedDict):
    5. amount: float
    6. currency: str
    7. due_date: str # normalized to YYYY-MM-DD
    8. def grade_structured(output: str, expected: dict) -> float:
    9. """Rule-based scoring for structured extraction tasks: 0 or 1"""
    10. try:
    11. data: ExtractResult = json.loads(output)
    12. except json.JSONDecodeError:
    13. return 0.0 # output cannot be parsed, fail immediately
    14. if not isinstance(data.get("amount"), (int, float)):
    15. return 0.0 # amount must be numeric; the string "100" does not pass
    16. if data.get("currency") not in {"CNY", "USD", "EUR"}:
    17. return 0.0
    18. if re.fullmatch(r"\d{4}-\d{2}-\d{2}", data.get("due_date", "")) is None:
    19. return 0.0
    20. return 1.0 if data == expected else 0.0
    21. def grade_code(output: str, tests: list[str], runner) -> float:
    22. """Code generation scoring: save the generated code to a temp file, then let the runner execute all tests"""
    23. path = runner.save_temp(output, suffix=".py")
    24. passed = sum(1 for t in tests if runner.run([t, str(path)]).returncode == 0)
    25. return passed / len(tests) # range 0.0 ~ 1.0

    For code generation tasks, a more common metric is the k-of-n pass rate—the proportion of k independent samples for the same problem in which at least one passes all tests. This metric better reflects whether a model “can write correct code” rather than “happened to get it right once.”

    (2) Model-as-judge: suitable for open-ended tasks

    For tasks like summarization, Q&A, and copywriting that have no single correct answer, you can have another large model score based on the rubric. A prompt template example is below:

    1. JUDGE_PROMPT = """You are a strict, objective review expert. Based on the given
    2. [Scoring Criteria], evaluate the quality of the [Candidate Answer] relative to the
    3. [Reference Answer], and give an integer score from 1 to 5.
    4. [Question] {question}
    5. [Reference Answer] {reference}
    6. [Scoring Criteria]
    7. {rubric}
    8. [Candidate Answer] {candidate}
    9. Review requirements:
    10. 1. Judge only on the question, reference answer, and scoring criteria; do not introduce external assumptions;
    11. 2. For each scoring criterion, judge "met / partially met / not met" separately;
    12. 3. The score must not be affected by answer length or formatting aesthetics;
    13. 4. Output only a JSON object, in the following format:
    14. {{"score": <integer>, "checks": [{{"item": <original criterion>, "verdict": "met|partially met|not met"}}], "reason": "<brief rationale>"}}
    15. """

    Model-as-judge requires careful control of four types of bias:

    • Self-preference: The judge model tends to give higher scores to answers generated by models in its own family. The judge model should be clearly different from the candidate model; when conditions allow, multiple judge models are best used for cross-scoring;
    • Position bias: In pairwise comparisons, the answer listed first tends to be favored. Swap the A/B order and repeat the review, or randomly shuffle the order;
    • Length bias: A longer answer is not necessarily better, and the scoring criteria should explicitly prohibit judging quality by length;
    • Unstable output format: Require the judge model to return structured data; if parsing fails, retry or route to manual review.

    Before using model-as-judge, you should also run a consistency check first: sample 30–50 cases, have 2–3 reviewers score independently, then compute the agreement between the judge model and human scores. For example, use quadratically weighted Kappa to measure agreement on ordinal ratings; a value above 0.6 is generally considered to have good reference value. This threshold is only an empirical guideline, and teams should set their own bar based on their own standards.

    (3) Human review: for calibration and arbitration

    Human review is the most expensive and should be concentrated on three scenarios: acceptance of newly launched capabilities, spot checks of model-as-judge results, and contentious cases with large review disagreement. For subjective experience metrics facing end users, the final standard should still be real user preference.

    5. Engineering Implementation of an Evaluation Framework

    A reusable evaluation framework is best organized into five layers:

    • Case layer: a set of cases in a unified format;
    • Adapter layer: hides interface differences between model services, making side-by-side comparison easy;
    • Inference layer: handles concurrent calls, timeouts, and retry control;
    • Scoring layer: three graders—rule-based, model-as-judge, and human review;
    • Reporting layer: outputs statistics grouped by dimension, difficulty, and case tags.

    Below is a minimal framework example:

    1. import asyncio
    2. import hashlib
    3. import json
    4. import random
    5. import statistics
    6. import time
    7. from dataclasses import dataclass, field
    8. from typing import Any, Callable, Optional, Protocol
    9. class ChatModel(Protocol):
    10. """Unified model-calling interface that hides differences between model services"""
    11. async def chat(self, messages: list[dict], **params: Any) -> str: ...
    12. @dataclass
    13. class EvalConfig:
    14. temperature: float = 0.0 # fixed sampling temperature to reduce output randomness
    15. max_output_tokens: int = 2048
    16. repetitions: int = 3 # number of repeated samples for the same case
    17. seed: Optional[int] = 42 # fixed random seed for reproducibility
    18. timeout: float = 60.0 # timeout per call (seconds)
    19. retries: int = 2 # number of retries on failure
    20. concurrency: int = 16 # concurrency level, to avoid triggering rate limits
    21. @dataclass
    22. class EvalCase:
    23. id: str
    24. messages: list[dict]
    25. grader: Callable[[str], float] # returns a score in 0.0 ~ 1.0
    26. dimension: str = "general" # capability dimension, used for grouped statistics
    27. difficulty: str = "medium"
    28. weight: float = 1.0 # case weight; raise it for key business scenarios
    29. @dataclass
    30. class CaseResult:
    31. case_id: str
    32. dimension: str
    33. difficulty: str
    34. weight: float
    35. status: str # ok / error
    36. score: float = 0.0 # mean of scores across multiple samples
    37. stdev: float = 0.0 # standard deviation of scores across multiple samples
    38. outputs: list[str] = field(default_factory=list)
    39. elapsed_ms: float = 0.0
    40. error: str = ""
    41. async def evaluate(model: ChatModel, case: EvalCase, cfg: EvalConfig) -> CaseResult:
    42. base = {"temperature": cfg.temperature, "max_tokens": cfg.max_output_tokens}
    43. if cfg.seed is not None:
    44. base["seed"] = cfg.seed
    45. scores: list[float] = []
    46. outputs: list[str] = []
    47. started = time.perf_counter()
    48. for _ in range(cfg.repetitions):
    49. # when sampling temperature > 0, use a different seed for each sample to estimate output variance
    50. params = dict(base)
    51. if cfg.seed is not None and cfg.temperature > 0:
    52. params["seed"] = cfg.seed + random.randrange(1 << 30)
    53. try:
    54. out = await asyncio.wait_for(model.chat(case.messages, **params),
    55. timeout=cfg.timeout)
    56. except Exception as exc:
    57. for _ in range(cfg.retries):
    58. try:
    59. out = await asyncio.wait_for(model.chat(case.messages, **params),
    60. timeout=cfg.timeout)
    61. break
    62. except Exception:
    63. continue
    64. else:
    65. # failures are not counted toward the score but tracked separately,
    66. # so that call failures are not mistaken for "low quality"
    67. return CaseResult(case.id, case.dimension, case.difficulty,
    68. case.weight, "error",
    69. elapsed_ms=(time.perf_counter() - started) * 1000,
    70. error=repr(exc))
    71. scores.append(case.grader(out))
    72. outputs.append(out)
    73. return CaseResult(
    74. case.id, case.dimension, case.difficulty, case.weight, "ok",
    75. score=statistics.mean(scores),
    76. stdev=statistics.stdev(scores) if len(scores) > 1 else 0.0,
    77. outputs=outputs,
    78. elapsed_ms=(time.perf_counter() - started) * 1000,
    79. )
    80. async def run_suite(model: ChatModel, cases: list[EvalCase],
    81. cfg: EvalConfig) -> list[CaseResult]:
    82. sem = asyncio.Semaphore(cfg.concurrency)
    83. async def guarded(case: EvalCase) -> CaseResult:
    84. async with sem: # control concurrency to avoid triggering server-side rate limits
    85. return await evaluate(model, case, cfg)
    86. return await asyncio.gather(*(guarded(c) for c in cases))
    87. def summarize(results: list[CaseResult], group_by: str = "dimension") -> dict:
    88. """Group statistics by dimension and report the call failure rate separately"""
    89. groups: dict[str, list[CaseResult]] = {}
    90. for r in results:
    91. if r.status == "ok":
    92. groups.setdefault(getattr(r, group_by), []).append(r)
    93. summary = {}
    94. for key, items in groups.items():
    95. scores = [r.score for r in items]
    96. summary[key] = {
    97. "count": len(items),
    98. "mean": round(statistics.mean(scores), 4),
    99. "worst": round(min(scores), 4),
    100. }
    101. failed = sum(1 for r in results if r.status == "error")
    102. summary["_meta"] = {
    103. "total": len(results),
    104. "call_failures": failed,
    105. "call_failure_rate": round(failed / len(results), 4),
    106. }
    107. return summary

    Building on this, there are three further engineering recommendations:

    1. Full traceability of results: For each case, the input, output, score, and elapsed time should all be recorded completely and written to persistent storage such as object storage, making it easy to review afterward and trace disputes.
    2. Pipeline-based scheduling: Evaluation tasks can be decoupled with a message queue and run on a container platform, turning evaluation into a repeatable, standard process that can be triggered on demand rather than a hand-written script.
    3. Monitoring and alerting integration: When an evaluation score or failure rate exceeds a threshold, an alert should fire automatically so that evaluation results genuinely participate in the day-to-day development workflow. On Baidu AI Cloud, these capabilities can be implemented through object storage, message queues, container-related products, log services, and monitoring and alerting services, respectively.

    6. Statistics and Interpretation: Making Conclusions Stand Up to Scrutiny

    Even with a fully automated evaluation process, conclusions can be misread. The points below determine whether evaluation data can genuinely guide decisions.

    1. Control randomness first, then repeat sampling

    The sampling temperature should be fixed and recorded in the configuration. When the temperature is greater than 0, the scores of multiple outputs for the same question inherently fluctuate; repeating the sampling 3–5 times per case lets you estimate the variance and avoid mistaking random fluctuation for a difference in model capability. It is worth noting that some model services do not guarantee full reproducibility—even with identical parameters, outputs may differ—which makes repeated sampling especially necessary.

    2. Pairwise comparison beats absolute scores

    When comparing two models, run both on the same set of cases, then compute the per-question score difference, rather than directly comparing two overall means. Pairwise comparison removes the interference of question-difficulty differences and is usually statistically more sensitive. The 95% confidence interval of the mean difference can be computed approximately as follows:

    1. import math
    2. import statistics
    3. def paired_ci95(a: list[float], b: list[float]) -> dict:
    4. """Mean of paired differences and its 95% confidence interval (approximate)"""
    5. diffs = [x - y for x, y in zip(a, b)]
    6. n = len(diffs)
    7. mean_diff = statistics.mean(diffs)
    8. se = statistics.stdev(diffs) / math.sqrt(n)
    9. half = 1.96 * se
    10. return {
    11. "mean_diff": round(mean_diff, 4),
    12. "ci95": [round(mean_diff - half, 4), round(mean_diff + half, 4)],
    13. "n": n,
    14. }

    The decision rules can be set like this:

    • If the confidence interval lies entirely above 0, you can conclude that “on this evaluation set, model A is stably better than model B”;
    • If the interval straddles 0, the current data is insufficient to support the conclusion that “A is better than B”;
    • If the interval lies entirely below 0, A has regressed and the cause needs to be investigated.

    3. Judge by tiers, not to the decimal point

    Human impressions are inherently coarse tiers, such as “between two generations.” Statistics should correspond to tiered output as well:

    • Clearly better: the interval’s lower bound is above a preset materiality threshold;
    • Comparable: the interval includes 0, or leans to one side but does not exceed the threshold;
    • Clearly worse: the interval’s upper bound is below the negative threshold.

    The “materiality threshold” is decided by the business. For example, you might define a quality-score difference within 0.02 as no material difference, to avoid spending effort on tiny fluctuations that are statistically insignificant and business-irrelevant.

    4. Look at the distribution, not just the mean

    Beyond the average score, pay attention to the share of lowest-scoring cases, the score standard deviation, and the tiered results for each difficulty. A model with a higher average but frequent failures on hard cases may actually pose greater risk in production.

    5. Evaluate quality, latency, and cost together

    Quality, latency, and cost often constrain one another. Plot the trade-off curve of all three on the same chart and look for the configuration with “the lowest cost that still meets the quality bar.” Cost accounting must be based on the real business distribution. Input and output are usually billed at different rates, and output length for tasks like long-document summarization and code generation can be several times the input length, so estimating cost by input length alone often understates it severely.

    7. From Evaluation to Launch: Phased Validation

    A high evaluation score does not mean good performance after launch. It is recommended to proceed through four stages:

    Stage 1: Offline evaluation
    Use a fixed evaluation set to complete both side-by-side comparisons and longitudinal regression. Longitudinal regression means comparing a new version against the current production version before launch to confirm that core metrics have not regressed. This stage focuses on quality scores.

    Stage 2: Shadow traffic
    Duplicate production requests and send a copy to the candidate model without affecting users, comparing output quality, latency, and failure rate under the real distribution. This stage is used to surface long-tail scenarios the evaluation set does not cover.

    Stage 3: Small-traffic canary
    Split traffic by user or request, route a portion of real traffic to the candidate model, and observe changes in business metrics such as task completion rate, user retry rate, and human-fallback ratio. Teams with the capacity can run A/B tests. Business metrics are the final judge; model scores are only a preliminary filter.

    Stage 4: Full launch and continuous monitoring
    After full launch, you still need to establish a routine regression mechanism: whether it is a model service upgrade, a prompt change, or an inference-parameter change, each should trigger a regression evaluation and check metric fluctuations against a baseline.

    8. A Checklist of Common Pitfalls

    1. Looking only at public leaderboards: Leaderboard questions may already be in training data and may differ greatly from your own business distribution, so leaderboard rankings can only serve as a preliminary filter.
    2. Single sample, default parameters: High output randomness plus unfixed prompts and parameters make results non-reproducible and hard to compare side by side.
    3. Testing only “smart,” not “obedient”: Weak instruction-following and structured-output ability can break an entire business chain, and these problems often do not show up in an overall score.
    4. Judging long context only by window size: Without checking information recall and length decay, you risk severe regressions in real long-document scenarios.
    5. Ignoring cost and latency: Slightly higher quality but slower responses and a higher unit price may not be worthwhile in a product serving end users.
    6. Not updating the evaluation set for a long time: Once question types age, scores tend to saturate and lose discriminating power, so new questions need to be added regularly.
    7. Treating call failures as quality deductions: Failures caused by the network or rate limiting should be tracked separately from model output quality; otherwise evaluation conclusions get polluted by environmental factors.

    9. Conclusion

    Back to the original question: how do you evaluate a large language model? A mature approach is not to give a single conclusion like “between two generations,” but to answer a set of more concrete questions:

    • On which tasks is it stronger, and on which does it have weaknesses?
    • Compared with the comparison models, does the difference exist stably in a statistical sense?
    • While meeting the quality threshold, are latency and cost acceptable?

    This method applies equally to evaluating models across different versions, different parameter configurations, and even different deployment forms. By turning subjective experience into a reproducible evaluation process, developers can make data-backed technical decisions that can be continuously re-validated in an environment of rapidly iterating models.