LLM Evaluation: Dimensions, Methods, and Engineering Practices
This article explains how to evaluate large language models in a rigorous, repeatable, and comparable way. It covers common evaluation pitfalls, the core dimensions to assess (task capability, controllability and safety, engineering efficiency, and long-term availability), how to design reproducible evaluations with the right judges for each task type, how to control bias in LLM-as-a-judge setups, and how to aggregate results into meaningful metrics. It also provides a reusable Python evaluation framework, guidance on API integration (synchronous, streaming, and structured output), practical optimization techniques, common pitfalls to avoid, and a phased rollout roadmap. The goal is not to crown a single winner but to find the most suitable option for a given business and keep it suitable as the technology evolves.
Introduction
LLM products are shipping updates faster than ever. Many developers have felt this directly: the same prompt that worked well a few days ago can suddenly change in style after switching to a newer model version. As a result, “is this model any good” has become an unavoidable question in tool selection, project planning, and technology upgrades.
People typically rely on three kinds of judgment:
- A “gut feeling” formed from a few conversations;
- Rankings on third-party leaderboards;
- Copying an approach that others have already proven.
All three are useful, but each has clear limitations when used on its own:
- Subjective impressions are unstable. Different models usually adopt different default system prompts and sampling settings, so their outputs vary widely in length, structure, and tone. A single conversation is heavily influenced by sampling randomness and is prone to bias.
- Leaderboard scores are not always reliable. Public evaluation sets carry two risks: questions may already appear in a candidate model’s training data (“data leakage”), and models may be optimized specifically for the leaderboard. On top of that, there is often a distribution gap between leaderboard scores and real business tasks.
- Evaluation is often superficial. Pricing, rate-limiting policies, and versioning change over time, so an approach that worked for someone else is not necessarily directly reproducible.
For these reasons, model evaluation should not be a one-off “trial run” but a sustainable engineering activity. At a minimum, it needs to be: dimensions that can be broken down, a process that can be reproduced, and conclusions that can be compared.
This article discusses that goal across five areas: evaluation dimensions, evaluation design, judging mechanisms, sample code, and a rollout path.
1. Common Misconceptions About Model Evaluation
1.1 Treating model capability as a single score
A single composite score easily hides real differences. A model that is good at code generation is not necessarily good at long-document QA; a model that excels at English tasks may sound unnatural when handling colloquial Chinese. Looking only at the total score makes it easy to “step on a landmine” at the most critical business step.
1.2 Confusing “model capability” with “system capability”
The online experience is determined jointly by the model, prompts, retrieval strategy, post-processing, and parameter settings. When quality drops after switching models, the cause may be one of the following:
- The original prompt was heavily “tuned” for the old model;
- The new model has a different default stopping strategy;
- The structured-output parsing step was not adapted in time.
Comparing only model outputs without controlling these variables yields unreliable conclusions.
1.3 Ignoring “stability”
Even with fixed parameters, large models can still be affected by factors such as floating-point operation order and concurrent batching, so the output for the same input is not necessarily identical word for word. Therefore, looking only at the “best single run” is meaningless. You should also track:
- The variance of answers across repeated sampling;
- The degree of degradation when facing overly long inputs or adversarial prompts;
- How the error rate changes over long-running periods.
2. Evaluation Dimensions: From Task Capability to Engineering Metrics
A complete evaluation system should cover at least four dimensions, each mapping to a clear business question.
2.1 Task capability: can the model “get it right”
| Capability category | Typical tasks | Recommended judging method |
|---|---|---|
| Language understanding | Summarization, information extraction, sentiment classification, coreference resolution | Key-element coverage checks, classification accuracy |
| Knowledge QA | Closed-book knowledge questions, time-sensitive questions | Exact match, comparison against reference answers |
| Logical reasoning | Multi-step math problems, constraint problems, ordering tasks | Extract the final answer, then exact match |
| Coding | Function implementation, defect localization, unit test writing | Run unit tests in an isolated sandbox |
| Long context | Cross-section information integration, long-document localization | Target information retrieval, citation localization |
| Instruction following | Format constraints, role settings, boundary conditions | JSON Schema validation, regex matching |
| Expressing uncertainty | Refusing or advising verification when beyond knowledge scope | Manual spot checks, rule validation |
“Expressing uncertainty” is often overlooked, yet it is directly tied to business risk. A model that always “confidently answers wrong” is more dangerous than one that clearly states its uncertainty. When evaluating, you should specifically design questions the model most likely does not know the answer to, and observe whether it fabricates.
2.2 Controllability and safety: is behavior “controllable”
Getting a task right is only the starting point for “usable.” Production systems also require the model’s behavior to be controllable with clear boundaries, mainly:
- Format stability: Can it strictly output in the specified format, with complete fields and correct types?
- Instruction robustness: When the input contains phrases like “ignore the above requirements,” can it still follow the system constraints?
- Safety and compliance: Are refusals for illegal, discriminatory, or privacy-leaking requests accurate, and is there “over-refusal” that hurts normal business?
2.3 Engineering efficiency: are speed and cost “worth it”
The speed users feel, and the cost enterprises actually pay, are often determined by engineering metrics rather than the model itself. This dimension should measure the following at the same time:
- Time to first token (TTFT): The time from sending a request to receiving the first valid token. It has the biggest impact on the streaming chat experience and is mainly affected by queueing time and prompt length.
- Time per output token (TPOT): After the first token, the average time per output token. It determines the reading wait for long answers.
- End-to-end latency: The overall time including queueing, prefill, and full decoding; it is also the basis for setting timeout thresholds.
- Cost per task: Actual input and output token counts multiplied by their unit prices, plus extra consumption from retries. Cost should be calculated per “completed valid business task,” not per “call.”
In addition, pay attention to these common practices and their effects:
- Context caching: If a long system prompt or reference material is reused, prefer a service form that supports context caching, which can significantly reduce billing and compute overhead for repeated inputs;
- Prefix stability: If the same prefix is reused, keep its position and content fixed in the prompt so caching can be better utilized;
- Batching: For offline tasks that tolerate latency, prefer batch processing to improve throughput and reduce cost.
2.4 Availability and compliance: is it “usable long term”
This dimension determines whether a model can sustainably support a business over the long term:
- Service stability: Service availability, rate-limiting policies, and degradation mechanisms under traffic spikes;
- Data and privacy: Whether input/output is used for training, data storage regions, and log retention and masking capabilities;
- Compliance requirements: Content safety review, industry regulatory requirements, and whether private deployment is needed;
- Versioning and compatibility: Whether the model provides a stable version that can be pinned, and whether new versions come with change notes and transition periods;
- Ecosystem and tooling: Whether structured output, tool calling, batch inference, and other capabilities are complete, and whether documentation and SDKs are mature.
3. Evaluation Design: Making Conclusions Reproducible and Comparable
3.1 Task set: drawn from real scenarios
The quality of the task set directly determines the credibility of evaluation conclusions. Recommended principles:
- Real sources. Extract tasks from desensitized production logs, tickets, and failure cases; this reflects the business distribution better than tasks written from scratch.
- Stratified sampling. Divide layers by business scenario, including high-frequency simple tasks, low-frequency complex tasks, and typical failure cases. Set a minimum sample size per layer to avoid long-tail scenarios being masked by averages.
- Boundary coverage. Specifically add edge cases such as empty input, overly long input, mixed languages, and malformed formats.
- Versioning and deduplication. Assign a stable ID and content hash to each task to detect duplicates and prevent data leakage. For public evaluation sets, assume the candidate model may have “seen” them, and treat them only as a reference rather than the sole basis.
3.2 Judges: different tasks need different “referees”
The judging mechanism is the core of the evaluation system. Choose among five types of judges based on task characteristics:
- Exact match / regex: Suitable for tasks with a unique answer or fixed format. Simple to implement and stable, but not tolerant of wording changes.
- Unit tests: Suitable for coding tasks, running model-generated code in an isolated sandbox. Objective, but the coverage of test cases sets the upper bound of the evaluation.
- Reference answer comparison: Scoring by key-element coverage or text similarity; suitable for summarization and extraction tasks. Watch out for “different wording but correct meaning” being misjudged.
- Model judge: An LLM scores outputs or does pairwise comparison against a rubric; suitable for subjective tasks like open-ended writing. Watch for bias when using it (see 3.3).
- Human review: Used to calibrate automatic judging, handle disputed samples, and perform final acceptance.
In practice, a combined strategy is recommended: judge everything that can be judged automatically, and manually review a fixed proportion of samples that cannot.
3.3 Model judges: controlling bias matters more than who you pick
Using an LLM as a judge is common in the industry, but the judge itself has biases, mainly:
- Position bias: When comparing two answers, favoring the one placed first (or last);
- Verbosity preference: Preferring longer answers when content quality is comparable;
- Self-preference: Preferring answers similar in style to its own.
You can mitigate these biases with the following structured prompt:
You are a rigorous technical reviewer. Evaluate the [model output] against the [scoring rubric]. Do not consider answer length.First explain your reasoning point by point against the rubric, then give an integer score from 1 to 5.Scoring rubric:- 5: Fully satisfies all constraints, accurate expression, no factual errors;- 3: Mostly satisfied, with a few minor issues that do not affect understanding;- 1: Clearly does not satisfy constraints or contains critical errors.Finally output only one line of JSON: {"score": <integer 1-5>, "reason": "<no more than 50 words>"}
For pairwise comparison, also make sure to:
- Present the two candidate outputs anonymously, without revealing their source;
- Swap the order and evaluate once each; count into the results only when both conclusions agree, and send disagreements to human review;
- Periodically verify the judge’s consistency against human review results;
- When averaging the win rates of candidates, use a general method such as Elo rating with ties, rather than simple counting.
3.4 Parameter fixing: controlling variables
To isolate “model differences” from other factors, use the same settings for a given batch of tasks:
- Sampling parameters (such as temperature, top_p);
- System prompt;
- Maximum output length;
- Timeout and retry policy.
At the same time, run multiple independent samples (e.g., n = 5) for key tasks, describing stability with mean and variance rather than a single run. For self-hosted inference services, fix the random seed to improve reproducibility. For closed-source commercial models, estimate random fluctuation through repeated sampling.
3.5 Result aggregation: making differences “judgeable”
Pass or fail on a single task is only raw data; ultimately it must be aggregated into metrics:
- Compute pass rate, mean score, and confidence interval per scenario;
- Report P50 and P95 of latency, not just the mean;
- Convert token consumption into money to get the cost per task;
- Filter out low-discrimination tasks. If all candidate models pass a task, or all fail, it does not contribute to ranking and can be removed or replaced.
In addition, for close gaps like “85% pass rate vs. 82%,” you need to distinguish a real difference from random fluctuation. You can use bootstrap resampling to estimate confidence intervals:
import randomdef bootstrap_pass_rate(results, rounds: int = 1000, sample_size: int = 200) -> tuple:"""Estimate the mean pass rate and 95% confidence interval via resampling with replacement.results: a boolean list like [True, False, True, ...]"""rates = []for _ in range(rounds):sample = [results[random.randrange(len(results))] for _ in range(sample_size)]rates.append(sum(sample) / len(sample))rates.sort()return sum(rates) / len(rates), rates[int(0.025 * len(rates))], rates[int(0.975 * len(rates))]
To judge whether the difference between two models is real, “pairwise comparison” is recommended: on the same batch of tasks, record +1 when one model passes and the other fails, -1 for the reverse, and 0 when both give the same result. The cumulative score is “wins minus losses”; the closer it is to 0, the closer the two models perform on that task set. If the confidence interval overlaps 0 significantly, you cannot conclude that “A is better than B.”
4. Sample Code: A Reusable Evaluation Framework Skeleton
Building on the design above, you can put together a minimal but usable evaluation framework with the Python standard library. It has four core parts:
- Data structures for tasks and results;
- A pluggable judge registry;
- A concurrent executor with caching;
- A stratified report generator.
import asyncioimport hashlibimport jsonimport timefrom dataclasses import dataclass, fieldfrom typing import Any, Optional, Protocol# ========== 1. Data structures ==========@dataclass(frozen=True)class Task:tid: str # Stable IDcategory: str # Scenario category, used for stratified statsprompt: strreference: Optional[str] = None # Reference answermeta: dict = field(default_factory=dict) # Judge name, judging params, etc.@dataclassclass CaseResult:task: Taskoutput: strjudge: strpassed: boolscore: floatlatency_ms: intprompt_tokens: intcompletion_tokens: int# ========== 2. Interfaces and judge registry ==========class ModelClient(Protocol):"""Model client protocol: real implementations should connect to a unified model gateway(responsible for auth, rate limiting, retries, billing, and logging)."""name: strasync def chat(self, prompt: str, params: dict) -> dict:"""Returns {'text': str, 'prompt_tokens': int,'completion_tokens': int, 'latency_ms': int}"""class Judge(Protocol):def verify(self, task: Task, output: str) -> tuple: # (passed, score)...class JudgeRegistry:def __init__(self):self._judges: dict[str, Judge] = {}def register(self, name: str, judge: Judge) -> None:self._judges[name] = judgedef get(self, name: str) -> Judge:if name not in self._judges:raise KeyError(f"Unregistered judge: {name}")return self._judges[name]# ========== 3. Judge implementations (illustrative) ==========class ExactJudge:"""Exact match: extract the answer from the model output and compare it with the reference"""def verify(self, task: Task, output: str) -> tuple:target = (task.reference or "").strip().lower()got = output.strip().lower()return got == target, float(got == target)class SchemaJudge:"""JSON Schema validation: verify that structured output is parseable and satisfies constraints"""def __init__(self, schema: dict):self.schema = schemadef verify(self, task: Task, output: str) -> tuple:try:data = json.loads(output)except json.JSONDecodeError:return False, 0.0valid = validate_against_schema(data, self.schema) # validation function (illustrative)return valid, float(valid)class UnitTestJudge:"""Unit-test judging: run model-generated code in an isolated sandbox.Sandbox requirements: block network access, limit CPU/memory/runtime,prevent reading sensitive host files, and avoid security risks from untrusted code."""def __init__(self, runner):self.runner = runner # runner(code, tests) -> (passed_count, total)def verify(self, task: Task, output: str) -> tuple:code = output.split("```")[-2] if "```" in output else outputpassed, total = self.runner(code, task.meta["tests"])return passed == total, (passed / total if total else 0.0)class LLMJudge:"""Model judge: for open-ended tasks; implement it per the prompt template in 3.3and calibrate it regularly with human spot-check results."""def __init__(self, client: ModelClient, template: str):self.client = clientself.template = templateasync def verify(self, task: Task, output: str) -> tuple:verdict = await self.client.chat(self.template.format(criteria=task.meta["criteria"], answer=output),{"temperature": 0},)try:data = json.loads(verdict["text"])score = max(1, min(5, int(data["score"])))return score >= 4, (score - 1) / 4except (json.JSONDecodeError, ValueError, KeyError):return False, 0.0 # judge output unparseable, forward to human review# ========== 4. Concurrent execution with caching ==========async def eval_one(client: ModelClient, task: Task, judge: Judge,params: dict, cache: dict) -> CaseResult:key = hashlib.sha256(json.dumps([client.name, task.tid, task.prompt, params, type(judge).__name__],ensure_ascii=False, sort_keys=True).encode("utf-8")).hexdigest()if key not in cache:resp = await client.chat(task.prompt, params)passed, score = judge.verify(task, resp["text"])cache[key] = CaseResult(task, resp["text"], type(judge).__name__, passed,score, resp["latency_ms"],resp["prompt_tokens"], resp["completion_tokens"])return cache[key]async def run_suite(tasks: list, client: ModelClient, registry: JudgeRegistry,params: dict, concurrency: int = 16) -> list:sem = asyncio.Semaphore(concurrency) # control concurrency to avoid triggering rate limitscache: dict = {}async def guarded(task: Task) -> CaseResult:async with sem:judge = registry.get(task.meta["judge"])return await eval_one(client, task, judge, params, cache)return await asyncio.gather(*(guarded(t) for t in tasks))# ========== 5. Stratified report ==========def report(results: list) -> dict:stats: dict[str, dict[str, Any]] = {}for r in results:s = stats.setdefault(r.task.category, {"total": 0, "passed": 0, "score": 0.0,"latencies": [], "cost": 0.0})s["total"] += 1s["passed"] += int(r.passed)s["score"] += r.scores["latencies"].append(r.latency_ms)# Unit prices are injected by the unified gateway; cost is priced by input/output tokens separatelys["cost"] += r.prompt_tokens * PRICE_IN + r.completion_tokens * PRICE_OUTfor s in stats.values():s["pass_rate"] = s["passed"] / s["total"]s["avg_score"] = s["score"] / s["total"]s["avg_latency_ms"] = sum(s["latencies"]) / len(s["latencies"])return stats
This code embodies three key designs:
- Decoupling judges from the executor: Adding a new task type only requires registering a new judge, without changing the execution flow.
- A unified gateway hides differences: All model calls go through a unified client protocol; switching models only requires replacing the client implementation, while the evaluation code stays unchanged.
- Caching ensures idempotency: The combination of task ID, prompt, params, and judge determines the cache key. Running the same configuration repeatedly yields identical results.
In real deployment, you can add three more capabilities on top of this:
- Deliver each task’s result as a message to a message queue, where downstream consumers archive it to object storage and generate visual reports;
- Integrate with monitoring and alerting services, so rising task failure rates and latency spikes automatically trigger notifications;
- Hook the evaluation entry point into the CI pipeline, so a model version or prompt change automatically triggers a regression run.
5. Integration: Synchronous, Streaming, and Structured Output
Models are usually exposed as APIs, and the design of the integration layer directly affects how faithful the evaluation results are. There are three common calling modes:
- Synchronous calls: One request returns the complete result, suitable for evaluation and batch processing.
- Streaming calls: Text is returned segment by segment via protocols such as SSE, suitable for chat scenarios. When evaluating, record the time to first token and the full response time separately.
- Structured output: Some services support returning a strict JSON structure to reduce parsing failures. However, services differ in how well they support schemas and enforce constraints, so you still need to validate them with SchemaJudge rather than assuming they are “always compliant.”
For enterprises with the right conditions, they can also deploy open-source models in their own environment. This approach is more flexible for data isolation and cost control, but it requires building your own inference service, autoscaling, and monitoring systems; capacity planning and operational overhead cannot be ignored.
Whether calling external services or self-hosting inference, the integration layer should solidify the general engineering practices accumulated during evaluation:
- Unified auth, rate limiting, timeouts, and retries;
- Separate billing and accounting by input and output tokens;
- Full request logging, to make results reproducible and to trace problems.
6. Optimization: Adapt to Capability Rather Than “Tuning the Model”
Different models follow system prompts and output formats differently, and the same prompt rarely performs equally well across all of them. This is one reason evaluation must control variables. Rather than repeatedly “tuning” the prompt, it is better to lower task difficulty with general methods:
- Break down complex tasks: Split “long-document summarization + structured output” into “section summaries → aggregation,” so each step is easier to validate.
- Make outputs verifiable: Require key conclusions to include their source text, so they can be checked automatically.
- Provide a few examples: For tasks with strict format requirements, giving 1–3 input/output examples in the prompt usually works better than a long textual description.
- Use models in tiers: For simple tasks like intent recognition and format conversion, a lightweight model can be chosen; for complex reasoning and long-form writing, upgrade to a stronger model. This tiered strategy is often better in both cost and latency.
7. Common Pitfalls and How to Avoid Them
- Evaluation set leakage: Candidate models may have already “seen” public questions. The fix is to rely mainly on a self-built task set and to refresh the questions periodically.
- Distortion from a single judge: Automatic judging is overly sensitive to wording. You should periodically sample for human review and track the agreement rate between automatic and human judgments.
- Averages masking the long tail: Two models with similar averages may differ drastically on key scenarios. You must look at metrics per scenario.
- Ignoring cost and latency: When capabilities are comparable, the option with a lower cost per task or lower P95 latency is often more suitable for large-scale rollout.
- Missing a regression mechanism: Without automated regression after a model upgrade, you can only judge by feel. Hook evaluation into CI so it runs automatically when versions change.
8. Rollout Roadmap
- Phase 1 (1–2 weeks): Map out 5–8 core business scenarios, build 30–50 tasks per scenario, decide the judging method, and get a minimal evaluation loop working end to end.
- Phase 2 (3–6 weeks): Build a unified model gateway and evaluation platform, introduce statistical significance and cost analysis, connect evaluation to CI, and link it with monitoring and alerting.
- Phase 3 (ongoing): Establish an online feedback loop. Sample real requests, desensitize them, and add them to the evaluation set; validate online performance through canary releases and A/B experiments, and iterate the evaluation set monthly or per version.
Conclusion
There is no “all-around champion” when it comes to model capability. Just as important as the model itself is a rigorous evaluation system: it determines whether you can see differences, whether you can reproduce results, and whether you can make the right decision quickly amid version changes.
Evaluation itself can also be implemented on cloud infrastructure. Baidu AI Cloud, for example, can combine capabilities such as LLM service platforms, container platforms, object storage, message queues, logging services, and monitoring and alerting to build a complete evaluation workflow spanning task management, automatic judging, report archiving, and online canary releases.
The ultimate goal of evaluation is not to decide “who is number one,” but to find the “most suitable” option for the business right now, and to keep it suitable as the technology evolves.
