Building an LLM Evaluation System: A Practical Engineering Guide

1. Why Single Benchmark Scores Fall Short in Model Selection

The iteration cycle of Large Language Models (LLMs) has compressed from annual to monthly updates, with each version bringing a new wave of benchmark scores. However, in engineering practice, relying on these scores presents several challenges:

1. Misaligned Evaluation Criteria: Vendor-published scores often stem from task distributions and prompt templates favorable to their own models. When comparing across different reports, the task sets, sampling temperatures, and scoring methods are frequently inconsistent, making the scores directly incomparable.
2. Benchmark Saturation and Contamination: Popular benchmarks are widely present in pre-training corpora, meaning models might have simply “memorized the answers.” Simultaneously, high-scoring benchmarks are becoming saturated, leading to a continuous decline in their ability to differentiate model performance.
3. Scenario Misalignment: The capability distribution measured by general benchmarks does not align with the long-tail distribution of real enterprise business tasks (e.g., report generation, contract review, customer service tickets, code repository maintenance). A high general score does not guarantee usability in specific business contexts.

Therefore, mature engineering teams are upgrading their evaluation strategies from simply “checking scores” to “building systems.” A robust evaluation system must serve three primary goals: Selection Assessment (evaluating whether a new model is worth adopting), Version Regression (ensuring capabilities do not degrade after an upgrade), and Deployment Readiness (verifying that a model meets business requirements before full rollout).

2. The Four-Layer Design of an Evaluation System

2.1 Task Pool: Defining “Exam Questions” with Real Work

A high-quality evaluation set should be derived from real work tasks rather than artificial Q&A prompts. Teams can adopt a “real occupational tasks” approach, using the frequent deliverables of knowledge workers as task templates: sales presentation decks, financial estimation models, legal element analysis, project scheduling, and manufacturing drawing interpretation. Each task category can then be stratified by complexity:

• L1 Basic Capabilities: Single-point tasks such as format conversion, information extraction, or single-function implementation, used to quickly identify capability gaps.
• L2 Comprehensive Tasks: Multi-constraint scenarios, such as “generating an analytical report with charts based on given data,” testing instruction following and comprehensive reasoning.
• L3 Long-horizon Agent Tasks: Multi-step, multi-deliverable tasks, such as “locating a bug in a code repository and submitting a fix,” testing planning, tool usage, and self-correction.

Empirically, maintaining an L1 : L2 : L3 test case ratio of around 5 : 3 : 2 supports both frequent regression testing and the coverage of high-value complex scenarios.

2.2 Scoring Mechanism: Cross-Evaluation by Multiple Judges + Human Arbitration

Any single judge (whether human or model) introduces systematic biases: model judges commonly exhibit preferences for length, formatting, or specific phrasing styles. A more robust approach is a three-judge cross-evaluation:

• Three independent judge models (or judge prompts) score the output separately.
• Calculate inter-judge agreement; cases with discrepancies exceeding a threshold are automatically routed to a manual review queue.
• Periodically calibrate judge prompts with human-annotated results to prevent scoring standard drift.

2.3 Metrics System: Quantifying “Good vs. Bad”

Capability Dimension	Core Metrics	Description
General Q&A	Accuracy, Preference Rate	Preference rate is suitable for pairwise comparisons of open-ended generation tasks
Factuality	Single-claim Error Rate, Error-containing Reply Rate	Two complementary granularities (see Section 3)
Code Capability	Test Pass Rate, Review Score, Rework Count	Balances “runs correctly” with “maintainability”
Instruction Following	Constraint Satisfaction Rate	Checks hard constraints (format, word count, structure) item by item
Agent Tasks	First-time Success Rate, Success within k Attempts	Reflects the fault tolerance cost in real-world usage
Efficiency & Cost	Latency, Token Consumption, Cost per Task	Often the deciding factor when capabilities are similar

2.4 Reproducibility Engineering

Evaluation results must be reproducible to be convincing:

• Fix sampling parameters (temperature, truncation length, system prompts) and record them in the test case metadata.
• Run each test case at least 3 to 5 times to gather statistics, reporting the mean and confidence interval rather than a single best score.
• Apply version control to the test case set and perform duplicate detection against public corpora before integration to mitigate data contamination risks.

3. Evaluation Key Points for Critical Capabilities

3.1 Hallucination Control: Testing at Two Granularities

A common pitfall in factuality evaluation is only checking “whether the entire reply is correct,” which masks localized errors. A more granular approach decomposes the output at the claim level:

• Single-claim Error Rate: Decompose the reply into independent factual claims, verify each one, and calculate the proportion of incorrect claims. This reflects the frequency of the model “getting one fact wrong.”
• Error-containing Reply Rate: If a reply contains at least one error, it is counted as a failure. This reflects the probability of a user “receiving an unusable answer.”

Combining both provides a complete picture: the former measures error density, while the latter measures delivery quality. A typical verification pipeline is: Claim Extraction → Evidence Retrieval → Item-by-item Verification → Manual Spot-check Calibration.

3.2 Code Capability: From “Passing Tests” to “Deliverable”

Unit test pass rates are merely the baseline. Repository-level evaluation should focus on: the accuracy of multi-file collaborative modifications, adherence to existing code styles, exception and boundary handling, and change minimization (avoiding unrelated refactoring). For the review dimension, it is recommended to score correctness, robustness, and readability separately, preventing the “it runs, so full marks” mindset.

3.3 Long Context: The Real Test Under Massive Tokens

Long-context evaluation cannot rely solely on “needle in a haystack” retrieval probes—that only proves the model can “find” the information, not “use” it well. A more comprehensive combination includes:

• Retrieval Probes: Embed markers in ultra-long documents to test precise localization capabilities.
• Multi-hop Synthesis: Require drawing conclusions by synthesizing information across documents and chapters, testing deep utilization capabilities.
• Positional Robustness: Place key information at the beginning, middle, and end respectively to observe the “lost in the middle” effect.
• Cost and Latency: The cost of ultra-long context calls increases significantly and must be weighed alongside capability gains.

3.4 Computer Control and Agents: The New Frontier of Evaluation

Currently, the control capabilities of general models are typically evaluated along two technical paths: the Vision Path, where the model understands the screen via screenshots and outputs mouse and keyboard commands to operate the desktop; and the Structured Path, where the model directly drives the page through browser automation libraries to complete form filling and system queries. Setting up this type of evaluation environment requires addressing four key aspects:

1. Environment Isolation and Resetting: Use sandboxes to host tested tasks and perform snapshot rollbacks after each test round to ensure no cross-contamination between cases.
2. Task Verifiability: Define machine-determinable success conditions for every task (e.g., a specific state appears on the target page, a target record is successfully written) to avoid “looking like it’s done.”
3. Efficiency Metrics: In addition to success rates, record average attempts, session duration, and token consumption—the cost difference between succeeding on the first try versus retrying three times can be exponential, even with the same 90% success rate.
4. Security Evaluation: Verify that confirmation strategies effectively intercept high-risk operations like payments, deletions, and external communications, and test resilience against in-page prompt injection attacks.

3.5 Interactive Reasoning: Goal Maintenance Under Multi-turn Interference

In real-world usage, users often inject new information or adjust directions mid-inference. Evaluations can design “mid-course correction” cases: inject new constraints halfway through a task to test whether the model can correctly abandon the old path, restructure its plan, and retain the valid work already completed.

4. Evaluation Engineering Implementation

The engineering form of an evaluation system is typically a pipeline: Test Case Management → Concurrent Invocation of Tested Models → Raw Result Archiving → Judge Scoring → Reporting and Alerting. When building on the cloud, container platforms can handle the elastic scheduling of evaluation tasks, object storage can archive raw conversations and screenshots, log services can aggregate call records, and monitoring/alerting systems can track metric anomalies. Utilizing a one-stop LLM platform like Baidu AI Cloud allows teams to leverage its built-in model services and supporting tools, focusing in-house development costs on the task pool and scoring standards—the areas that truly reflect business differences.

Below is a dependency-free code sample demonstrating the aggregation of cross-judge scores and a simplified calculation for success within k attempts:

import statistics
def aggregate_judges(scores: dict, threshold: float = 1.0) -> dict:
    """Three-judge cross-evaluation score aggregation.
    scores looks like {"judge_a": 4, "judge_b": 5, "judge_c": 4}
    """
    vals = list(scores.values())
    mean = statistics.mean(vals)
    std = statistics.pstdev(vals) if len(vals) > 1 else 0.0
    return {
        "mean": round(mean, 2),
        "std": round(std, 2),
        "needs_human_review": std > threshold,  # Escalate to human review if discrepancy is too high
    }
def success_within_k(attempts: list, k: int = 3) -> float:
    """Whether there is at least one success within k attempts (single case determination).
    attempts is a list of booleans ordered by attempt; aggregate by averaging across all cases.
    """
    return 1.0 if any(attempts[:k]) else 0.0
def claim_error_rate(claims: list) -> float:
    """Single-claim error rate: claims is the item-by-item verification result, True means incorrect."""
    return sum(claims) / len(claims) if claims else 0.0

In a production environment, teams should also record the model version, prompt version, sampling parameters, and judge version for each test case, ensuring that any given score can be traced back to its complete execution context.

5. From Evaluation to Selection: A Decision Framework

With reliable metrics in place, the model selection decision can follow four steps:

1. Assign Dimension Weights by Business Scenario: Increase preference rate and factuality weights for content generation businesses; increase code and Agent weights for engineering tasks; include cost-per-task for cost-sensitive operations.
2. Use the Custom Evaluation Set as the Primary Criterion: Use general benchmarks as a reference only—the correlation between a custom business evaluation set and real-world distribution will always be higher than that of public benchmarks.
3. Canary Validation: Conduct low-traffic A/B testing to compare candidate models against the live model based on real user feedback, rolling out fully only after metrics converge.
4. Establish Continuous Regression: Trigger the evaluation pipeline for every model vendor upgrade and every prompt modification, blocking capability degradation before it reaches production.

6. Conclusion

The essence of model evaluation is translating ambiguous questions like “is this model good?” into a set of measurable, reproducible, and attributable engineering problems. The task pool determines what to test, the scoring mechanism determines how to judge, the metrics system determines what to observe, and engineering determines if the system can run sustainably. Once these four layers operate stably, model selection shifts from “chasing scores and public opinion” to “data-driven infrastructure decision-making”—which is the key to maintaining technical initiative in an era of rapid model iteration.