How to Evaluate Large Language Models: Dimensions, Metrics, and Process

    This article explains how to evaluate large language models (LLMs) with a reproducible workflow instead of relying on launch-day claims or leaderboard rankings. It walks through a six-step evaluation process, the core evaluation dimensions (from language understanding and reasoning to code, long-context handling, multimodal and structured output, and safety), the key engineering metrics, and metric definitions such as F1, Exact Match, pass@k, cost, and latency. It also covers evaluation methodologies (standardized benchmarks, human preference comparison, regression on real tasks, and canary releases), common pitfalls like data contamination and leaderboard overfitting, a runnable Python example, an FAQ, and a pre-launch checklist. The goal is not to find a "best all-around" model, but to verify whether a model reliably meets the bar on your own business distribution.

    Start with the conclusion: run the six-step evaluation process

    After a new model is released, it is not advisable to judge it only by launch-day talking points or leaderboard rankings. Instead, you should run a reproducible evaluation process:

    1. Define the task: Clarify whether the model needs to handle question answering, summarization, code completion, structured output, or multimodal understanding.
    2. Build an evaluation set: Sample from production logs, real tickets, and historical successful cases to form a private, traceable dataset with field descriptions.
    3. Offline screening: Use small-scale offline evaluation to quickly rule out obviously unqualified candidate models.
    4. Verify stability: Test output stability under a proper sampling strategy to avoid distortion from a single chance result.
    5. Aggregate metrics: Evaluate quality, engineering, cost, and safety metrics together.
    6. Canary rollout: Run a small-traffic online validation, observe error rate, latency, cost, and safety risk, and then decide whether to roll out fully.

    The goal of this process is not to find a “best all-around” model, but to verify whether the model can reliably meet the bar within your own business distribution.

    1. Why leaderboard scores alone are not enough

    Leaderboard scores can serve as a first-pass reference, but they cannot replace business evaluation, for several reasons:

    • The leaderboard may not match your business distribution: Public benchmark questions often fail to cover the long-tail distribution of real user requests.
    • Data contamination is possible: Some test questions may have been seen by the model during pretraining or fine-tuning, inflating scores.
    • Leaderboard overfitting is possible: Vendors may optimize for specific public benchmarks, and their actual generalization ability may not be strong.
    • Leaderboards usually ignore engineering metrics: Time to first token, throughput, error rate, per-call cost, and concurrency stability often matter more for deployment than a single accuracy number.

    Therefore, evaluation should return to three fundamentals: real tasks, real data, and real cost.

    2. Evaluation dimensions: from model capability to engineering usability

    2.1 Six core dimensions

    Dimension Evaluation goal Sample requirements Failure signals
    Basic language understanding and generation Whether instruction following, contextual coherence, summarization, and style control are stable Real business corpora covering short text, long text, and multi-turn dialogue Off-topic answers, format errors, missing key information
    Reasoning ability Whether logic, math, multi-step planning, and causal analysis are reliable Multi-step questions, boundary conditions, and counterexamples Intermediate steps going off track, contradictory conclusions
    Code ability Whether code completion, function generation, defect localization, and refactoring are capable Alignment with real interfaces, dependency versions, and constraints Syntax errors, missing edge cases, code that cannot run directly
    Long-context handling Information recall within long text and resistance to interference Long inputs with key information placed at different positions Losing key information, being distracted by irrelevant content
    Multimodal and structured output Image-and-text understanding, table parsing, and JSON Schema output Real images, tables, and structured field samples Broken JSON format, missing fields, wrong types
    Safety and alignment Refusals, jailbreak robustness, factuality, and value consistency Sensitive scenarios, leading questions, hallucination-prone questions Harmful output, being induced into overreach, factual errors

    2.2 Engineering metrics matter just as much

    Metric Definition Measurement conditions Example reference threshold
    Time to first token (TTFT) Time from request sent to first token/first content segment returned Fixed concurrency, input length, network environment Below 500 ms or 1 s, defined per business
    End-to-end latency Time from request sent to complete result returned Fixed input length, output length, concurrency Set relative to the timeout threshold
    Throughput Number of requests or tokens processed per unit time Load-testing tool, concurrency, input/output distribution Set based on capacity planning
    Error rate Share of failed requests among total requests Continuous run window Should be close to 0, with alerts
    Timeout rate Share of timed-out requests among total requests Fixed timeout threshold Monitor separately
    Retry strategy Whether to retry after failure, retry count, backoff strategy Combined with idempotency and cost Avoid infinite retries
    Circuit breaking and rollback Whether you can degrade or switch models when metrics go abnormal Monitoring alerts and switch verification Must have an executable plan
    Per-call cost Token cost and infrastructure cost per request Calculated from real input/output lengths Match business gross margin

    3. Metric definitions: F1, Exact Match, pass@k, cost, and latency

    3.1 F1 score

    Suitable for scenarios with reference answers such as summarization, extraction, and classification.

    • Precision: precision = correct predictions / total predictions
    • Recall: recall = correct predictions / total references
    • F1: 2 * precision * recall / (precision + recall)

    If computed at the token level, you can count the tokens overlapping between the prediction and the reference, then compute precision, recall, and F1.

    3.2 Exact Match

    Suitable for scenarios where the answer is highly deterministic, such as code completion, SQL, function signatures, and structured fields.

    Judgment: after normalizing whitespace and case, the output is scored 1 if it exactly matches the expected result, otherwise 0.

    3.3 pass@k

    Suitable for code generation and multi-candidate sampling scenarios.

    Suppose you generate n candidates for the same question and c of them pass the tests. The probability that at least k pass can be written as:

    pass@k = 1 - C(n-c, k) / C(n, k)

    If n - c < k, then there cannot be enough remaining failing samples, so it can be recorded directly as 1.

    Example: generate 3 candidates, 1 passes, and you require at least 1 to pass, then pass@1 = 1 - C(2,1)/C(3,1) = 1 - 2/3 = 0.333.

    3.4 Time to first token and throughput

    • Time to first token: first_token_time - request_start_time
    • End-to-end latency: finish_time - request_start_time
    • Throughput: total tokens / total elapsed time or successful requests / total elapsed time

    3.5 Per-call cost

    Per-call cost = input tokens * input price / 1000 + output tokens * output price / 1000 + allocated infrastructure cost

    Example: with 500 input tokens and 200 output tokens, an input price of 0.01 per 1K tokens, and an output price of 0.03 per 1K tokens, the per-call token cost is:

    500 * 0.01 / 1000 + 200 * 0.03 / 1000 = 0.005 + 0.006 = 0.011

    When selecting a model, you should calculate using the real request length distribution of your business, rather than estimating all scenarios with an average.

    4. Mainstream evaluation methodologies: offline, human, regression, and canary

    4.1 Standardized benchmark testing

    Suitable for quick side-by-side comparison, with low cost and reproducibility. The downside is that it may be disconnected from your business, so it works as a first-pass screen but not as the final decision basis.

    4.2 Human or crowdsourced preference comparison

    Have annotators blind-review or make pairwise comparisons of multiple model outputs, which is closer to subjective experience. Suitable for evaluating generation quality, tone, and readability. The downside is high cost, and it requires clear scoring rules, annotation guidelines, and consistency checks.

    4.3 Regression on real tasks

    Sample real requests from production logs and build a regression set. This is the method closest to the business. The sample fields should include at least:

    • id: unique identifier
    • input: real input
    • expected: expected answer or judgment method
    • metric: metric, such as F1, Exact Match, pass@k
    • weight: weight
    • threshold: pass threshold
    • source: source, such as ticket, log, or test set
    • tags: task type, scenario labels
    • privacy_level: privacy level
    • version: version or date

    4.4 Online A/B testing and canary releases

    Canary releases should follow these principles:

    • Initial split ratio: 1% or lower
    • Ramp-up ratios: gradually increase through 5%, 20%, 50%
    • Metrics to monitor: error rate, timeout rate, time to first token, user feedback, safety block rate, cost
    • Sample size: cover enough requests to avoid chance fluctuations
    • Rollback conditions: error rate exceeds the threshold, latency rises significantly, cost becomes abnormal, or a safety incident occurs
    • Observation period: cover at least one full business cycle, and avoid observing only during off-peak periods

    5. Evaluation pitfalls to watch out for

    5.1 Data contamination

    Test questions may already be covered by the model’s training data, inflating scores. Mitigations include:

    • Use a private evaluation set
    • Dynamically update questions
    • Use data published after the model’s training cutoff date
    • Keep a held-out test set private

    5.2 Leaderboard overfitting

    Vendors may optimize for public leaderboards. Ways to detect this include:

    • Whether reproducible evaluation scripts are provided
    • Whether the sample distribution and failure cases are disclosed
    • Whether they only emphasize leaderboard scores without explaining task boundaries
    • Whether engineering metrics like cost, latency, and safety are missing

    5.3 Distribution mismatch

    If the evaluation set comes from public datasets while business requests come from real users, the mismatch in distributions will distort conclusions. Build evaluation sets from business logs and update them regularly.

    5.4 Ignoring randomness

    The same input may produce different results at different sampling temperatures. Recommendations:

    • For deterministic evaluation, use temperature=0 and run once, unless the model remains uncontrollable.
    • For stability evaluation, use temperature>0 and set a random seed.
    • After multiple samplings, report the mean, standard deviation, and confidence interval.
    • When comparing two models, use the same inputs and the same sampling strategy, and run a statistical significance test if necessary.

    5.5 Ignoring prompt sensitivity

    The same model may perform noticeably differently under different prompts. During evaluation, preserve the prompt version and record how prompt changes affect the results.

    6. An engineering evaluation workflow with runnable code

    Below is a directly runnable Python example. Requirements: Python 3.8 or later, with no third-party dependencies. In real use, you only need to replace MockClient with a wrapper around your actual model calls.

    1. import json
    2. import math
    3. import random
    4. import re
    5. import statistics
    6. from collections import Counter
    7. def normalize(text: str) -> str:
    8. return re.sub(r"\s+", " ", text.strip().lower())
    9. def f1_score(expected: str, output: str) -> float:
    10. exp_tokens = normalize(expected).split()
    11. out_tokens = normalize(output).split()
    12. if not exp_tokens and not out_tokens:
    13. return 1.0
    14. if not exp_tokens or not out_tokens:
    15. return 0.0
    16. common = Counter(exp_tokens) & Counter(out_tokens)
    17. overlap = sum(common.values())
    18. if overlap == 0:
    19. return 0.0
    20. precision = overlap / len(out_tokens)
    21. recall = overlap / len(exp_tokens)
    22. return 2 * precision * recall / (precision + recall)
    23. def exact_match(expected: str, output: str) -> float:
    24. return float(normalize(expected) == normalize(output))
    25. def score_fn(expected: str, output: str, metric: str) -> float:
    26. if metric == "f1":
    27. return f1_score(expected, output)
    28. if metric == "exact":
    29. return exact_match(expected, output)
    30. raise ValueError("unsupported metric: " + metric)
    31. def pass_at_k(n: int, c: int, k: int) -> float:
    32. if n - c < k:
    33. return 1.0
    34. return 1.0 - math.comb(n - c, k) / math.comb(n, k)
    35. def ttft_ms(start_time: float, first_token_time: float) -> float:
    36. return (first_token_time - start_time) * 1000.0
    37. def estimate_cost(
    38. input_tokens: int,
    39. output_tokens: int,
    40. input_price_per_1k: float,
    41. output_price_per_1k: float
    42. ) -> float:
    43. return (
    44. input_tokens * input_price_per_1k / 1000.0
    45. + output_tokens * output_price_per_1k / 1000.0
    46. )
    47. class MockClient:
    48. """Demo client; in real scenarios, replace it with your model call wrapper."""
    49. def generate(self, prompt: str, temperature: float = 0.0, seed=None) -> str:
    50. if "summarize" in prompt:
    51. return "Customer asks about refund conditions; the agent explains the process and guides them to submit a request."
    52. if "complete" in prompt:
    53. return "def add(a, b):\n return a + b"
    54. return ""
    55. def build_eval_suite():
    56. return [
    57. {
    58. "id": "c1",
    59. "input": "Summarize the following support conversation: the user asks about refund conditions, and the agent replies that the product must be intact and guides them to submit a request.",
    60. "expected": "Customer asks about refund conditions; the agent explains the process and guides them to submit a request.",
    61. "metric": "f1",
    62. "weight": 0.5,
    63. "threshold": 0.75,
    64. "source": "prod_log",
    65. "tags": ["support summary"]
    66. },
    67. {
    68. "id": "c2",
    69. "input": "Complete the function: implement addition of two numbers.",
    70. "expected": "def add(a, b): return a + b",
    71. "metric": "exact",
    72. "weight": 0.5,
    73. "threshold": 1.0,
    74. "source": "unit_test",
    75. "tags": ["code completion"]
    76. }
    77. ]
    78. def run_eval(model_client, eval_set, temperature=0.0, n_repeat=1, seed=None):
    79. if temperature == 0.0 and n_repeat != 1:
    80. raise ValueError(
    81. "temperature=0 means deterministic output; "
    82. "if you want to evaluate randomness, set temperature>0 and n_repeat>1."
    83. )
    84. if temperature > 0.0 and seed is not None:
    85. random.seed(seed)
    86. results = []
    87. for case in eval_set:
    88. outputs = []
    89. for _ in range(n_repeat):
    90. outputs.append(
    91. model_client.generate(
    92. case["input"],
    93. temperature=temperature,
    94. seed=seed
    95. )
    96. )
    97. scores = [
    98. score_fn(case["expected"], output, case["metric"])
    99. for output in outputs
    100. ]
    101. mean_score = statistics.mean(scores)
    102. std_score = statistics.pstdev(scores) if len(scores) > 1 else 0.0
    103. results.append({
    104. "id": case["id"],
    105. "metric": case["metric"],
    106. "mean": round(mean_score, 4),
    107. "std": round(std_score, 4),
    108. "threshold": case["threshold"],
    109. "passed": mean_score >= case["threshold"],
    110. "weight": case.get("weight", 1.0),
    111. "sample": outputs[0] if n_repeat == 1 else None
    112. })
    113. return results
    114. def summarize(report):
    115. total_weight = sum(x["weight"] for x in report)
    116. weighted_score = (
    117. sum(x["mean"] * x["weight"] for x in report) / total_weight
    118. if total_weight
    119. else 0.0
    120. )
    121. return {
    122. "weighted_score": round(weighted_score, 4),
    123. "all_passed": all(x["passed"] for x in report),
    124. "cases": report
    125. }
    126. if __name__ == "__main__":
    127. models = {"mock": MockClient()}
    128. eval_set = build_eval_suite()
    129. for name, client in models.items():
    130. report = run_eval(client, eval_set, temperature=0.0, n_repeat=1)
    131. print(json.dumps(
    132. {"model": name, **summarize(report)},
    133. ensure_ascii=False,
    134. indent=2
    135. ))
    136. print("pass@k example:", pass_at_k(n=3, c=1, k=1))
    137. print("cost example:", round(estimate_cost(500, 200, 0.01, 0.03), 4))

    To run it: save the code as eval.py and execute python eval.py. In a real system, you can archive evaluation results to object storage, use a message queue to decouple the sampling and scoring tasks, and track online metric drift through monitoring and alerts.

    7. FAQ: how to choose LLM evaluation

    Q1: How soon after a new model is released should I evaluate it?

    Evaluate it as soon as the official model weights, API, or stable service are available. Do not judge only by launch-day previews; verify real capabilities on a callable version.

    Q2: Can I choose a model directly if its leaderboard score is high?

    No. Leaderboards are only for first-pass screening. The final decision must factor in the business distribution, task type, cost, latency, safety, and maintainability.

    Q3: How large should the evaluation set be?

    There is no universal standard, but it should at least cover the core task distribution. Small teams can start with 50 to 200 real samples and keep expanding. What matters is not the count but the representativeness.

    Q4: Do I still need multiple samples with temperature=0?

    Usually not. temperature=0 aims for deterministic output. If you want to evaluate randomness and stability, use temperature>0, set a random seed, and take multiple samples before computing the mean, standard deviation, or confidence interval.

    Q5: How do I avoid data contamination?

    Use a private evaluation set, dynamic questions, and time-split data, and keep a portion of validation data that has never been made public. Avoid mixing evaluation questions with training data.

    Q6: How do I decide the canary ratio?

    Start at 1%, and after confirming that error rate, latency, cost, and safety metrics are stable, expand to 5%, 20%, and 50%. At any stage, an anomaly should be rollback-able.

    8. Pre-launch checklist

    • Business task type and acceptance metrics are defined
    • A private evaluation set from real business data is built
    • The evaluation set includes the fields: id, input, expected, metric, weight, threshold, source
    • Metric definitions for F1, Exact Match, and pass@k are established
    • Time to first token, end-to-end latency, throughput, error rate, and timeout rate are measured
    • Per-call cost is estimated, including input and output token costs
    • Security and compliance testing is complete, including refusal of harmful content and jailbreak robustness
    • Retry, rate limiting, circuit breaking, and rollback strategies are prepared
    • The canary split ratio and observation period are set
    • Monitoring alerts and log redaction are configured
    • Model versions, prompt versions, and evaluation results are retained for traceability

    Conclusion

    The marketing hype around new models eventually fades; what truly determines business results is whether the evaluation system is rigorous. By upgrading evaluation from “watching launches and chasing trends” to “letting data speak,” you can make rational, stable, and reusable model selection decisions amid rapid model iteration.