Large Model Evaluation Guide: 3D Scenes, Physical Common Sense, and Multi-Turn Retrieval
A practical framework for evaluating large models that generate runnable front-end applications. It covers interaction reliability, task constraints, physical plausibility, algorithmic convergence, retrieval traceability, and factual consistency, with evaluation workflows, scoring criteria, and TypeScript examples for turning subjective impressions into reproducible assertions.
1. The Core Problem: Three Signals, One Root Cause
A practical evaluation of a new generation of large models—focused on their ability to generate runnable front-end projects—revealed several issues that matter more than whether the model can generate code at all.
The generated result may look acceptable while failing the task constraints. The design intent described in the reasoning may not match the final artifact. A model may claim to have performed multiple retrieval rounds when it actually performed only one, and its final answer may still contain factual errors.
These failures appear at different layers, but they share the same root cause: the evaluation process does not translate natural-language requirements into verifiable acceptance criteria.
This article organizes these observations into three evaluation layers and eight common failure modes. It also provides an evaluation workflow, metric definitions, and sample code to help teams build reproducible and regression-ready evaluation systems.
| Signal type | Meaning | Typical symptoms | Typical consequence |
|---|---|---|---|
| Interaction signal | Whether the code runs and the scene can be operated | Blank page, console errors, inability to start the game | The artifact is unusable, and the problem is usually obvious |
| Constraint signal | Whether the result satisfies the rules and common-sense requirements in the prompt | Level elements outside the board, incorrect collision rules | The application runs, but the gameplay or simulation is invalid |
| Factual signal | Whether the model’s claims and reasoning match the actual artifact and execution record | Claimed effects are not implemented, retrieval rounds do not match tool calls, configuration parameters are fabricated | The highest-risk category because confident explanations can hide the problem |
Traditional evaluations focus mainly on interaction signals. As a result, “it runs” is often mistaken for “it meets the requirements.”
Constraint signals require the prompt to be translated into assertions. Factual signals require independent verification of the model’s reasoning, tool calls, and final answer. The third category is particularly dangerous because it often appears detailed and confident, making errors difficult to detect through visual inspection alone.
2. Evaluating Generated Interactive 3D Projects
Tasks such as a sliding-block puzzle, a bowling physics simulation, maze reinforcement learning, and multi-turn web retrieval share several useful properties:
- They require a moderate amount of code.
- The output can run in a real environment.
- The results are observable.
- Each task can be converted into explicit acceptance criteria.
This makes them effective benchmarks for evaluating generated applications.
2.1 Discrete State-Space Puzzles: Constraint Satisfaction and Solvability
For sliding-block puzzles, the primary evaluation target is not visual appearance but rule correctness.
The evaluation should verify:
- Whether movement direction matches the drag gesture or keyboard input.
- Whether every move keeps the pieces within the board boundaries.
- Whether the relative positions of the empty space and target block are preserved correctly.
- Whether the final state can be detected accurately.
Because the state space is discrete and finite, a breadth-first search can be used to calculate the shortest solution path. That path can then be used to test whether:
- Random operations can eventually solve the puzzle.
- The hint function returns legal moves.
- The puzzle configuration is solvable.
The visual presentation also introduces soft constraints. If the interface declares a particular visual style, the piece textures, color palette, and background should be consistent with that style. Otherwise, the implementation may be inconsistent with the design intent described by the model.
2.2 Continuous Physics Scenes: Common Sense Matters More Than Code Validity
In a bowling scene, the key elements include the ball trajectory, pin arrangement, and collision behavior.
The evaluation should not focus only on whether the ball can knock down every pin in a single shot. It should first verify whether the geometry matches reality. With standard pin spacing, the gap between adjacent pins is much larger than the ball’s diameter. A ball traveling along the center line can directly hit only some pins. The remaining pins must be knocked down indirectly by pins from the front row.
Therefore, the following are signs of incorrect geometry or causal reasoning:
- Pins are placed unrealistically close together.
- The model claims that a single shot clears all pins because the spacing is “reasonable.”
- The explanation of the collision result does not match the actual layout.
Visual readability is also essential in a physics demonstration. If lighting is missing or materials are too dark, users cannot clearly see the trajectory or determine whether a collision occurred.
A basic interface acceptance checklist should verify:
- Whether there are clear Start and Restart controls.
- Whether the ball and pins are distinguishable by size and contrast.
- Whether important state information, such as score and remaining pins, is visible.
These may appear to be interface details, but they are prerequisites for evaluating the physics simulation itself.
2.3 Algorithmic Tasks: Turning Training Results into Convergence Metrics
A maze reinforcement learning task helps distinguish between a model that can write code and one that can implement an algorithm correctly.
The agent can usually be modeled as occupying a discrete grid state, with four possible actions: up, down, left, and right. A reward function might assign a positive reward for reaching the goal and negative rewards for hitting walls or wasting steps.
At minimum, the evaluation should check four areas:
- Whether the maze is actually rendered and whether the logical grid corresponds one-to-one with the displayed grid.
- Whether the agent’s rendered size is appropriate relative to the cell size. An agent that is too large blocks the view, while one that is too small is difficult to identify.
- Whether training converges, meaning that returns across multiple evaluation episodes become stable.
- Whether the learned policy can actually solve the maze.
A further consistency check can be applied to the demonstration. If the training process is presented as grid-based reinforcement learning, the interface should not display unrelated animations or data visualizations. Such unrelated output may indicate a mismatch between the reasoning process and the implemented artifact.
2.4 Multi-Turn Retrieval: Making Retrieval Claims Observable
Multi-turn retrieval is especially vulnerable to misleading claims. Three types of checks are useful:
- Round-count consistency: Does the number of retrieval rounds claimed by the model match the actual tool-call record?
- Query diversity: Do later queries retrieve new information, or do they repeatedly return the same results?
- Factual verification: Can every important claim in the final answer be supported by the retrieved sources?
Behavior may also vary significantly across access channels. For example, the same model may behave differently in a web interface and a mobile application because of differences in retrieval orchestration, prompt templates, or tool availability.
Such differences should not automatically be interpreted as differences in the underlying model. Evaluation reports should record the access channel and configuration so that engineering differences are not mistaken for model capability differences.
3. Common Failure Modes and Verifiable Criteria
| Failure mode | External symptom | Root cause | Verification method |
|---|---|---|---|
| Incorrect state maintenance | Illegal moves or unsolvable levels | Rules were not encoded as assertions | Use a solver to verify solvability and run random-operation stress tests |
| Incorrect geometric intuition | Unrealistic spacing makes gameplay invalid | No constraints representing real-world scale | Validate coordinates and collision radii against standard parameters |
| Mismatch between visuals and claims | The interface does not match the described design | Reasoning and implementation are not aligned | Use an interface checklist and compare screenshots |
| Training does not converge | The policy cannot consistently reach the goal | Incorrect reward or state encoding, or insufficient evaluation episodes | Analyze return curves and success-rate thresholds |
| Grid alignment errors | Objects are offset from the background | Logical and rendering coordinate systems are inconsistent | Assert that logical-to-rendered coordinate mappings are correct |
| Interaction-layer defects | The scene cannot start or respond | Initialization failure or missing render loop | Capture screenshots with a headless browser and collect console errors |
| False retrieval-round claims | The model claims multiple rounds but performs one | Retrieval orchestration or counting logic is incorrect | Parse tool-call logs and compare them with the model’s claims |
| Fabricated parameters | Important configuration values are incorrectly attributed | Information was generated without verification | Check each claim against authoritative sources |
The last failure mode is often underestimated. Incorrect core configuration information can influence user decisions, so the consequences are usually much more serious than a visual defect.
4. A Practical Evaluation Workflow
A reliable process can be organized into three stages: a fixed task set, automated assertions, and human review.
4.1 Define Tasks and Acceptance Criteria Together
A prompt should not simply say, “Build a maze game.” It should specify requirements such as:
- Board dimensions.
- The required ratio between agent size and cell size.
- The number of training episodes.
- Whether the training process must be displayed.
This gives “completion” an objective definition.
4.2 Freeze the Environment and Artifacts
Each evaluation should produce:
- Complete runnable source code.
- Screenshots.
- The runtime environment.
- The model version.
- The access channel.
- Relevant configuration details.
This makes results reproducible and comparable.
4.3 Score the Layers Separately
Interaction usability, constraint satisfaction, and factual consistency should receive separate scores. If any layer fails its minimum requirement, the overall evaluation should fail.
This prevents attractive visuals from masking logic defects.
4.4 Run Regression Comparisons
Historical evaluation results should be added to a regression set. This helps determine whether a new model version represents a genuine improvement or has simply traded performance in one dimension for performance in another.
A simplified scoring table can use three levels: pass, partial pass, and fail.
| Dimension | Check | Passing standard |
|---|---|---|
| Runability | Initial loading, console errors, interaction response | No blocking errors |
| Rule correctness | Legal state transitions and reachable goals | A solver can produce a solution path |
| Physical plausibility | Scale ratios, collision radii, and spacing | Within the defined tolerance of standard parameters |
| Interface readability | Contrast, object size, and control visibility | Key elements are identifiable and usable |
| Algorithmic effectiveness | Convergence curve and success rate | Meets the predefined threshold |
| Consistency | Reasoning versus artifact, claims versus execution | No substantive contradictions |
5. Sample Assertions: Turning Impressions into Tests
The following examples illustrate evaluation principles. They are not tied to any particular product implementation.
5.1 Validate Grid and Coordinate Alignment
This function checks whether the mapping from logical grid coordinates to screen coordinates is unique and remains within the canvas. It can help identify cases where the rendering layer is misaligned with the action layer.
type Grid = { rows: number; cols: number };type Point = { row: number; col: number };// Logical coordinates -> screen coordinates.// Verify that the mapping is one-to-one and stays inside the canvas.function assertGridAlignment(grid: Grid, canvas: { width: number; height: number }) {const cellW = canvas.width / grid.cols;const cellH = canvas.height / grid.rows;const project = (p: Point) => ({ x: p.col * cellW, y: p.row * cellH });const seen = new Set<string>();for (let r = 0; r < grid.rows; r++) {for (let c = 0; c < grid.cols; c++) {const { x, y } = project({ row: r, col: c });const key = `${x},${y}`;if (seen.has(key)) throw new Error('Mapping collision: ' + key);seen.add(key);if (x < 0 || y < 0 || x >= canvas.width || y >= canvas.height) {throw new Error('Coordinate out of bounds: ' + key);}}}return { cellW, cellH, project };}
5.2 Check Bowling-Pin Geometry
This example uses commonly accepted pin dimensions and spacing values to define a reasonable range, then checks whether the generated layout deviates from real-world geometry.
// Values are in meters and represent commonly used bowling dimensions.const BALL_RADIUS = 0.108;const PIN_RADIUS = 0.0605; // Approximately half of a 12.1 cm maximum pin widthconst PIN_SPACING = 0.3048; // Approximately 12 inches between adjacent pin centersfunction checkPinLayout(pins: { x: number; y: number }[]) {const problems: string[] = [];for (let i = 0; i < pins.length; i++) {for (let j = i + 1; j < pins.length; j++) {const d = Math.hypot(pins[i].x - pins[j].x, pins[i].y - pins[j].y);if (d < PIN_RADIUS * 2) problems.push(`Overlapping pins: ${i}/${j}`);if (Math.abs(d - PIN_SPACING) > 0.03) {problems.push(`Unexpected spacing: ${i}/${j} = ${d.toFixed(3)}`);}}}return problems; // A non-empty result indicates an implausible layout.}
5.3 Evaluate Training Convergence
This function compares the average return across recent evaluation episodes with the best observed return. It also checks whether every recent episode achieved a positive outcome.
type Episode = { steps: number; totalReward: number };function assertConverged(episodes: Episode[], window = 20, threshold = 0.95) {const tail = episodes.slice(-window);const avg = tail.reduce((s, e) => s + e.totalReward, 0) / tail.length;const best = Math.max(...episodes.map(e => e.totalReward));if (best <= 0) return { converged: false, reason: 'No positive return observed' };const ratio = avg / best;const allReachGoal = tail.every(e => e.totalReward > 0);return { converged: ratio >= threshold && allReachGoal, ratio };}
5.4 Audit Retrieval Rounds and Query Diversity
This function compares the claimed number of retrieval rounds with the actual number of calls. It also calculates the duplicate-query ratio, helping identify cases where the model performs one effective search but claims to have conducted several rounds.
type SearchCall = { query: string; results: number };function auditSearchRound(claimedRounds: number, calls: SearchCall[]) {const uniqueQueries = new Set(calls.map(c => c.query.trim().toLowerCase())).size;const emptyResults = calls.filter(c => c.results === 0).length;return {roundsConsistent: calls.length === claimedRounds,duplicateRatio: calls.length ? 1 - uniqueQueries / calls.length : 0,emptyResultRatio: calls.length ? emptyResults / calls.length : 0,verdict: calls.length === claimedRounds && uniqueQueries >= claimedRounds ? 'pass' : 'fail',};}
6. Three Hard Rules for Multi-Turn Retrieval Evaluation
6.1 Verify Before Accepting Claims
Any statement that the model performed multiple retrieval rounds should first be compared with the actual tool-call record. If no execution record exists, the default evaluation result should be “not executed.”
6.2 Make Round Requirements Observable
Instead of asking the model to describe retrieval rounds only in prose, require it to return a structured record containing the retrieval plan and actual execution results. Structured output is easier to audit than verbal claims.
6.3 Trace Critical Facts to Sources
For parameters, configuration values, prices, and specifications, require clickable or otherwise identifiable sources. Any claim that cannot be supported by the available sources should be marked as a separate risk item.
Differences across access channels should also be recorded as an independent variable. When the same model behaves differently across platforms, the true evaluation target is often the combination of the model, retrieval orchestration, and prompt template. Conclusions are meaningful only when tied to a specific configuration.
7. Conclusion
A seemingly casual “try it and see” evaluation can cover state-machine correctness, physical common sense, reinforcement learning implementation, and retrieval factuality. These capabilities correspond closely to areas that frequently fail in real-world applications.
When subjective impressions are converted into executable assertions, an evaluation can move from “it looks acceptable” to a measurable result such as “six checks passed, one failed, and one remains a risk.”
For development teams, the value of this approach extends beyond scoring. It moves acceptance criteria into the requirements phase by explicitly defining inputs, outputs, constraints, and verification methods. Whether the goal is model selection, version regression testing, or capability validation through a model service on a cloud platform such as Baidu AI Cloud, the same evaluation standards can be reused consistently.
