Large Language Model Evaluation: A Five-Dimension Practical Guide and Pitfall Checklist
How should you evaluate large language models, choose between them, and spot common failure modes? As LLMs move beyond conversational Q&A into composite tasks like code generation, agent execution, web retrieval, front-end reconstruction, and physics simulation, single-point impressions no longer reflect real capability. This article breaks evaluation into five dimensions—visual and front-end rendering, spatial reasoning, physics simulation, algorithm implementation with reinforcement learning, and multi-round retrieval—each with concrete pass criteria, failure-mode examples, and scoring suggestions. It includes a corrected Q-learning convergence-check function in Python, a retrieval verification table, and a deployable evaluation checklist template, along with guidance on compliance, tiered scoring, and building a regression-ready evaluation pipeline.
How Do You Evaluate Large Language Models?
How do you evaluate large language models? How do you choose a model? What are the common failure modes? As LLMs move from conversational Q&A toward composite tasks like code generation, agent execution, web retrieval, front-end reconstruction, and physics simulation, single-point experience can no longer reflect true capability. A model might generate a beautiful interface but fail to get physical common sense right; it might answer fluently yet reuse stale results across multiple search rounds, or even fabricate parameters. Evaluation therefore needs to shift from “looks good” to dimensional verification that is quantifiable, reproducible, and regression-testable.
Before starting an evaluation, define the compliance boundaries first: evaluation inputs should be anonymized to avoid submitting real personal information, sensitive enterprise data, or unauthorized documents; web retrieval tasks need source labeling, with verification of source authorization, link accessibility, and factual reliability; evaluation logs, prompts, model outputs, and judgments should be stored under tiered access controls to prevent privacy leaks or the secondary spread of false facts.
1. Why Single-Point Experience Is Not Enough for LLM Evaluation
Traditional evaluation often focuses on static capabilities such as text comprehension, knowledge Q&A, and style generation, but real applications have become highly composite. A model with a “high pass rate” might only be good at mimicking interface appearance and organizing fluent copy, yet lack the ability to validate spatial relationships, physical laws, algorithm convergence, and factual sources.
The core of multi-dimensional evaluation is not simply adding more questions, but breaking out different capability facets and defining pass criteria for each one. For example:
- Front-end rendering should not be judged only by “is it cool-looking,” but also by whether it faithfully executes reasoning constraints;
- Spatial layout should not be judged only by “is the final result correct,” but by whether every step is legal;
- Physics simulation should not be judged only by “does the picture look real,” but by whether it matches basic common sense;
- Algorithm implementation should not be judged only by “can it run,” but by whether it converges and whether coordinates align;
- Web retrieval should not be judged only by “did it give an answer,” but by whether sources are real and parameters are traceable.
This approach is closer to software testing: each dimension needs inputs, outputs, pass criteria, and failure modes.
2. Visual Rendering and Front-End Generation: Avoiding “Looks Good but Doesn’t Work”
Visual rendering and front-end generation are high-frequency scenarios in LLM applications and also the areas most prone to misjudgment. Many models can generate seemingly complete particle effects, lighting animations, card layouts, or mini-game interfaces, but “looks good” does not equal “works well.”
A more noteworthy failure mode is the “chain-of-thought and implementation mismatch.” For example, a model might declare in its reasoning that it will present a certain vintage aesthetic, add drag-and-drop interaction, and keep a responsive layout, yet the final output reflects none of it. This can be viewed as a form of reasoning hallucination: the model “says” in the planning stage what it will do, but does not deliver in the generation stage.
Evaluation Checklist
- Whether the code runs directly;
- Whether the page or effect is legible;
- Whether the key constraints the model states about itself are implemented;
- Whether the interaction is genuinely usable;
- Whether the visual style matches the task requirements.
Recommend treating “declaration-implementation consistency” as an independent scoring item. When evaluating, first have the model list its key features, then check the final output item by item. For example, if the model claims to support “particle trails, button hover, and mobile adaptation,” check each of these three items individually, rather than only looking at whether the page overall looks cool.
Failure Mode Examples
- The model claims to implement particle effects, but only generates a static background;
- The model claims to support mobile adaptation, but the layout overflows on narrow screens;
- The model claims to have interactive buttons, but the buttons have no event bindings;
- The model claims to render 3D effects, but it is actually just image replacement.
The root cause of these problems is that the model completed the “description” but never closed the “implementation loop.”
3. Spatial Reasoning: Judging Whether Layout Tasks Truly Pass
Spatial reasoning tasks commonly appear in sliding puzzles, maze navigation, board layouts, block placement, and similar scenarios. Taking sliding puzzles as an example, the model must not only generate the final layout but also ensure every move satisfies the constraints.
Judging only by whether the final result is correct can lead to misjudgment. A model might reach the correct layout by chance while its reasoning process contains illegal moves; it might also just apply a template without truly understanding discrete spatial constraints. Therefore, spatial tasks should check at least three pass criteria:
- Whether each move is legal;
- Whether the final layout is fully aligned;
- Whether it passes under multiple sets of random initial layouts.
Recommend not using a single test case to determine success. For the same class of spatial task, prepare multiple sets of random initial states across three tiers: easy, medium, and hard. If the model succeeds only under a fixed prompt but fails after switching to random layouts, that indicates insufficient generalization.
Recommended Scoring Items
- Number of illegal moves: should be 0;
- Final target alignment rate: should be 100%;
- Pass rate across multiple random test cases: recommend at least 70%;
- Whether the reasoning process explains the constraints: can be a bonus item;
- Signs of “piecing together via templates”: treat as a risk item.
The key to spatial reasoning evaluation is not whether the model gives the correct answer, but whether it has the ability to stably solve constrained problems.
4. Physical Common Sense and Simulation Reasonableness: Turning “Looks Real” into Quantifiable Acceptance
3D physics simulation is well-suited as a comprehensive capability test—examples include bowling pin collisions, ball collisions, liquid flow, and spring systems. It can simultaneously test graphics rendering, physics engine usage, interaction control, and common-sense reasoning.
The common failure point is not “the picture is not pretty enough” but a violation of basic common sense. For example, the generated spacing between bowling pins is too large, making it physically impossible to knock down all pins in a single throw; or after a collision, objects penetrate the ground, trajectories do not match the gravity direction, or interaction keys do not work. This indicates that the model completed the rendering but did not validate logical self-consistency.
Quantifiable Check Recommendations
For bowling-pin-collision-type tasks, you can check:
- Whether the pin spacing matches real-world proportions;
- Whether the ball’s center of mass, speed, and trajectory are reasonable;
- Whether the number of pins knocked down after impact meets the target;
- Whether the picture is legible, not too dark, and objects are visible;
- Whether user interaction is genuinely operable;
- Whether the simulation can end within a finite time;
- Whether there are issues such as object clipping, flying off screen, or physics engine errors.
If a task has no fixed business parameters, you can also apply relative standards: the physical parameters generated by the model must be consistent with common sense, with no obvious errors such as “distances too close,” “collisions ineffective,” or “objects falling infinitely.”
Physics simulation evaluation should avoid pursuing only visual effects. A truly qualified evaluation confirms whether the model understands whether the goal can be achieved, and whether the simulation matches intuition.
5. Algorithm Implementation and Reinforcement Learning: Checking Self-Correction and Convergence
Compared with rendering, algorithm implementation more strongly tests the model’s ability to close the logic loop. A typical task is to have the model implement a reinforcement-learning-based agent, such as training Q-learning in a maze to find the optimal path. This kind of task usually requires multiple rounds of correction, so it is well-suited for examining the model’s self-correction ability.
A typical failure chain might include:
- First generation: the agent’s size is abnormal, occupying most of the maze;
- Second generation: the maze map is not drawn at all;
- Third generation: the maze appears but cannot run;
- Fourth generation: the agent’s action layer and the terrain layer have misaligned coordinates.
The common thread across these problems is layer alignment and state consistency. When evaluating, you cannot just look at whether the code exists; you also need to see whether the agent can act stably in the environment, whether training produces effective updates, and whether the training process converges.
Evaluation Checklist
- Whether the code runs;
- Whether the environment initializes correctly;
- Whether the agent can move within the maze;
- Whether the reward function is reasonable;
- Whether the Q-values undergo effective updates;
- Whether the training process converges;
- Whether the agent’s and the environment’s coordinate systems are strictly aligned.
Convergence judgment needs a clear criterion. Below is a corrected, illustrative check function. Before using it, note the following: q_values_history should be a list of numeric array snapshots, each item being a numpy.ndarray or an object convertible to a numeric array; all snapshots must have the same shape; epsilon and window need to be calibrated against the Q-value scale, reward scale, learning rate, and training noise. Typically, you can start with a small-scale grid search—for example, window set to 10, 20, and 30, and epsilon set to roughly 0.1% to 1% of the reward scale.
import numpy as npdef check_convergence(q_values_history, epsilon=1e-3, window=20):"""Determine whether reinforcement learning Q-values have converged.Preconditions:1. q_values_history is a list of numeric array snapshots; each item shouldbe an np.ndarray or an object convertible to a numeric array;2. The shape of every Q-table snapshot must be identical;3. epsilon and window need to be calibrated against the Q-value scale,reward scale, learning rate, and training noise.Calibration suggestions:- If the reward range is large or the absolute Q-values are large, epsiloncan be increased appropriately;- If training fluctuates significantly, window can be increased appropriately;- It is recommended to run a small-scale parameter search first beforeusing it in formal evaluation."""if len(q_values_history) < window:return Falserecent = [np.asarray(q, dtype=float) for q in q_values_history[-window:]]if any(q.shape != recent[0].shape for q in recent):raise ValueError("Inconsistent Q-table snapshot shapes; please unify the state-action table structure")max_delta = 0.0for i in range(1, len(recent)):delta = float(np.sum(np.abs(recent[i] - recent[i - 1])))max_delta = max(max_delta, delta)return max_delta < epsilon
To further reduce misjudgment, you can combine a moving-window mean, variance, and the reward curve. For example, do not only check whether the Q-value change magnitude is below the threshold, but also check whether the reward steadily increases or stays flat in the later stages of training. Many models’ Q-values still oscillate violently after dozens of rounds and cannot converge, which is itself an important failure signal for algorithm-implementation ability.
6. Multi-Round Retrieval and Information Verification: Detecting “Pretend Searching” and Factual Hallucination
Web retrieval is a high-frequency demand for current LLMs and also a hotspot for hallucination and lazy behavior. A common problem: when asked to “perform four rounds of search,” a model might actually search only once yet repeatedly reuse the first result in the text, while falsely claiming to have completed multiple rounds.
More dangerous are factual errors, especially on key information such as model numbers, parameters, configurations, prices, and release dates. The model might invent a nonexistent configuration item, or wrongly attribute a feature to the target object. Once users take this at face value, it could cause real losses.
Retrieval Task Verification Table
| Verification Item | Example |
|---|---|
| Required search rounds | 4 rounds |
| Actual number of cited sources | 4 different sources |
| Whether sources are traceable | Each source has a title, link, or document ID |
| Whether key parameters are accurate | Model numbers, parameters, and configurations consistent with sources |
| Whether old results are reused | Whether content across different rounds is clearly repetitive |
| Whether negative facts are stated | When a feature genuinely is not present, whether it is proactively flagged |
When evaluating, focus on three points:
- Whether the number of searches matches the content. If the model claims to search N times, does it actually produce information from N different sources?
- Whether key facts can be cross-verified. For core information such as model numbers, parameters, and configurations, require the model to label sources and allow traceability.
- Whether negative facts are detected. If a feature “genuinely is not present,” a high-quality model should proactively flag it rather than silently skip it or fabricate it.
For enterprise scenarios, you can add a high-risk field list—for example, device models, interface protocols, security levels, price ranges, and service availability zones. Errors in these fields should be judged directly as critical failures, not just minor deductions.
7. Practical Checklist: A Deployable Evaluation Template
To turn evaluation from experience-based judgment into an executable process, recommend building a checklist for each task. Below is a general template that can be extended per business scenario.
| Task Type | Input Samples | Pass Criteria | Common Failure Modes | Scoring Items | Sample Count |
|---|---|---|---|---|---|
| Front-end rendering | Page description, interaction requirements, visual reference | Code runs, all key constraints implemented | Declaration-implementation mismatch, style failures | Runnability, consistency, legibility | 10 sets |
| Spatial reasoning | Puzzle initial layout, movement rules | Each move legal, final layout fully aligned | Illegal moves, accidental correctness | Legality, alignment rate, generalization rate | 5+ sets |
| Physics simulation | Scene description, object rules, interaction requirements | Reasonable physics, interactive and terminable | Clipping, spacing errors, unable to knock down | Common-sense reasonableness, interactivity, stability | 5 sets |
| Algorithm implementation | Algorithm goal, inputs/outputs, training rounds | Runnable, coordinate-aligned, training converges | Run failure, coordinate misalignment, non-convergence | Runnability, convergence, alignment | 3 sets |
| Multi-round retrieval | Search questions, round requirements, parameter verification table | Sources real, parameters accurate, negative facts flagged | Reusing old results, fabricating parameters, untraceable sources | Source consistency, factual accuracy | 10 sets |
Recommend a tiered scoring system—for example, three tiers of 0, 0.5, and 1 per item—to avoid subjective “looks good” scoring. For critical errors such as fabricating parameters, illegal moves, or being completely non-runnable, you can set a veto item or a major-deduction item.
At the engineering level, you can leverage mainstream cloud service capabilities (logging, monitoring, object storage) to structurally retain evaluation inputs, model outputs, judgments, and manual review comments. This forms a regression-sustainable evaluation dataset that supports quick comparison after model version upgrades, and helps enterprises build model admission, canary release, and issue rollback mechanisms.
At the same time, the evaluation process itself must mind compliance: sample anonymization, access control, desensitized log storage, web source authorization verification, and approval workflows for accessing sensitive fields. Only under compliance premises do evaluation results carry long-term reference value.
8. Conclusion
Evaluating a large language model cannot stop at “seems okay.” Truly professional evaluation turns vague “feelings” into clear dimensions, explicit criteria, and reproducible processes. Visual rendering must check declaration-implementation consistency, spatial reasoning must check whether constraints are stably satisfied, physics simulation must check whether common sense is reasonable, algorithm implementation must check whether training converges, and multi-round retrieval must check whether sources are real.
Building a multi-dimensional, quantifiable, and traceable evaluation system not only helps teams choose more suitable models, but also continuously surfaces problems in later iterations, pinpoints failure modes, and accumulates high-quality samples. For developers and enterprises, this is far more valuable than a one-off demo.
