How to Build a Reliable Comprehensive Evaluation System for Large Language Models

Why Comprehensive Evaluation Matters

Evaluating a Large Language Model (LLM) should not be limited to checking whether a single response is fluent, nor should conclusions about a model’s capability be drawn from a handful of examples. Real-world applications typically require the model to understand instructions simultaneously, adhere to formatting constraints, execute multi-step reasoning, and explicitly state limitations when information is insufficient or constraints conflict. If an evaluation covers only one of these aspects, it is easy to mistake local performance for overall capability.

A valuable evaluation system should answer three fundamental questions:

1. What to test: Clearly define the capability dimensions corresponding to the task, rather than making a vague assessment of “intelligence.”
2. How to judge: Establish reviewable scoring rules for correctness, formatting, and constraint adherence.
3. What the results mean: Explain the capability boundaries reflected by the scores and their relationship to actual application scenarios.

The goal of evaluation is not to create an seemingly authoritative leaderboard, but to identify the strengths, weaknesses, and uncertainties of a model in specific tasks, providing a basis for model selection, prompt engineering, and risk control.

1. Designing Test Sets from Capability Dimensions

1.1 Instruction Following and Format Control

These tasks test whether a model can simultaneously satisfy multiple explicit requirements. For example, a prompt might require generating a fixed number of sentences, arranging them in a specific order, meeting specific ending rules, and avoiding certain characters or words. The difficulty lies not in the content itself, but in ensuring that multiple constraints are met simultaneously.

During evaluation, requirements should be broken down into checkable conditions:

• Is the quantity accurate?
• Is the order correct?
• Is the formatting consistent?
• Do any forbidden items appear?
• Is each piece of content still fluent and meaningful?

A combination of rule-based checks and manual review is ideal for these tests. Quantity, character positions, and forbidden words can typically be verified automatically, while semantic naturalness and content reasonableness are better checked by human reviewers. Relying solely on whether a sentence “looks good” can easily miss detail violations; conversely, using only programmatic checks for characters cannot determine if the text carries actual meaning.

1.2 Mathematical Reasoning and Combinatorial Search

Math problems are useful for observing whether a model can consistently understand limitations, propose valid derivations, and verify results. Taking a number combination problem as an example, the task might require using each input digit exactly once, allowing operations within a specified range, while prohibiting scripts, brute-forcing, or extra digits.

Scoring should distinguish between several levels:

• Correct result: Does the final expression equal the target value?
• Input compliance: Were all and only the given digits used?
• Operation compliance: Were the allowed or prohibited operation rules followed?
• Credible process: Is the explanation consistent with the expression, and are there skipped steps or post-hoc rationalizations?
• Answer quantity: If multiple distinct solutions are required, are the quantity and distinctness criteria met?

A common mistake is treating a “correct final value” as the sole standard. In fact, if an answer introduces numbers not provided in the prompt, it is unacceptable even if the calculation result is correct. Whether multiple solutions are genuinely distinct also requires pre-definition: this can be judged by expression structure or by requiring different core operational paths, preventing the same solution from being counted multiple times after superficial modifications.

1.3 Multi-Digit Passwords and Constraint Satisfaction

Password deduction puzzles are excellent for testing a model’s ability to handle repeated elements, positional relationships, and counting rules. These problems usually contain multiple guesses and feedback, such as certain digits being in the correct position, while others are present but in the wrong position. If repeated digits are allowed, the counting method must be exceptionally clear to prevent a single digit from being counted multiple times.

It is recommended to divide the verification process into two steps:

1. First, compare the password and the guess digit by digit, recording the digits that match exactly in position.
2. For the remaining positions, count the occurrences of the digits. The number of times a digit can be counted as “present but in the wrong position” should be limited by the smaller of the remaining occurrences on either side.

Formally, for a digit (d), first subtract the quantity matched by position, then compare the occurrences of (d) in the unmatched portions of both sets. This prevents over-scoring when one side has multiple identical digits, and avoids counting a digit already credited for a correct position as a wrong-position match.

Evaluators should also verify that the final candidate password satisfies all clues simultaneously, rather than just checking a few. For this type of problem, listing a verification checklist row by row is usually more reliable than providing a single conclusion:

Clue	Exact Position Matches	Correct Digits, Wrong Position	Compliant (Yes/No)
Guess 1	Counted by rules	Counted by rules	Yes/No
Guess 2	Counted by rules	Counted by rules	Yes/No

If an answer does not show the verification process, evaluators should independently calculate it and should not mark it correct simply because the reasoning text is coherent.

2. Establishing Clear and Reproducible Scoring Rules

Comprehensive evaluations should not compress all tasks into a simple “right or wrong” and sum them up. A more robust approach is to define scoring dimensions for each question and lock in the scoring criteria before testing. For example, if a full score is divided into several tiers, you can evaluate:

• Is the core answer correct?
• Are input and output constraints adhered to?
• Is the reasoning process verifiable?
• Are any necessary conditions missing?
• Are there unsupported assertions?

Scoring rubrics should be established before running tests to avoid adjusting standards after seeing the model’s answers. If a question has multiple reasonable solutions, the acceptable range of answers should be stated in advance; if the question itself is ambiguous, the question should be revised first, rather than leaving the interpretation to the scorer’s discretion.

For automated checks, prioritize deterministic rules, such as quantity, format, character sets, and formula verification. For semantic judgments, the review basis should be recorded; when multiple reviewers are involved, they can score independently and then review disputed samples together. This improves consistency and helps identify ambiguities in the problem descriptions.

3. Controlling Variables in the Testing Process

To compare different models or different versions of the same model, the testing process must be as consistent as possible. At a minimum, the following should be recorded:

• Test set version and question content;
• Prompts and their context;
• Generation parameters and runtime;
• Whether multi-turn follow-ups are allowed;
• Answer truncation, exceptions, or failure situations;
• Scoring rules and reviewer information.

If the same question uses different prompts, different interaction rounds, or different scoring criteria, the results are no longer directly comparable. For generation processes involving randomness, the test should be repeated when conditions permit to observe result stability. A single success does not indicate stable capability, just as a single failure does not necessarily represent the overall level.

The test set itself must also be version-controlled. Adding new questions, revising wording, or changing scoring criteria should all be documented in a changelog. If the difficulty of the new and old versions differs, the two total scores cannot be simply compared side-by-side; a set of fixed questions can be retained as an anchor to help observe trends.

4. Analyzing Error Structure Beyond Total Scores

Total scores provide a quick overview but can mask capability differences. Two models might achieve similar scores, but one might frequently lose points due to formatting details, while the other consistently provides flawed reasoning. The former might be improved through output validation, while the latter may require more cautious task decomposition and manual review.

Therefore, the report should at least include:

• Scores for each capability dimension;
• The rationale for each question’s judgment;
• Main error types and their frequency;
• Typical success and failure cases;
• The uncertainty and applicability scope of the results.

Errors can be categorized by type, such as: omitted constraints, misreading the prompt, arithmetic errors, repeated counting, inconsistency between arguments and conclusions, non-compliant formatting, and over-assertion of uncertain information. Mapping error types to task scenarios is usually of more practical value than simply reporting a single composite score.

5. Best Practices for Enhancing Evaluation Quality

First, avoid data leakage. Publicly available questions may enter the model’s training corpus, causing test results to reflect memorization rather than generalization. You can use regularly updated private questions, parameterized generated variants, or questions that change their surface form while maintaining the same reasoning structure.

Second, avoid an overly skewed test set. If the questions are primarily concentrated on short arithmetic or format generation, the results should not be interpreted as broad, comprehensive capabilities. The test set should cover the tasks that the target application truly cares about, and the report should explicitly state the domains not covered.

Third, avoid equating evaluation scores directly with production performance. Real-world workflows also include context quality, tool calling, data permissions, response latency, failure recovery, and manual review. Offline task performance provides only limited evidence and must be combined with small-scale trial runs and risk assessments for the target business.

Fourth, retain auditable materials. Original prompts, complete outputs, automated check results, and manual scoring records should be saved correspondingly. In the event of a dispute, evaluators can reproduce the judgment process and pinpoint whether the issue lies with the model, the question, or the scoring engine.

Conclusion

A reliable LLM evaluation relies not on stacking more questions or pursuing a single high score, but on clearly defining capabilities, accurately describing constraints, establishing reviewable judgment rules, and structurally analyzing errors. By covering different tasks such as instruction following, mathematical reasoning, and constraint solving, combined with version control, repeated runs, and manual review, evaluation results can genuinely help readers understand under what conditions a model is reliable, where it is prone to errors, and what validations are still needed before deployment.