LLM Deliverable Evaluation: A Technical Guide to Generation and Search-Augmented Behavior
This article explains why demo-level output is not the same as engineering readiness for LLM-generated deliverables. It maps vague quality concerns to seven failure modes, defines layered acceptance checks for visual and motion generation, designs automated verifiers for spatial and physics-based tasks, and evaluates reinforcement learning scripts and multi-round search behavior for correctness, convergence, traceability, and truthfulness. It also outlines a CI-ready evaluation pipeline, three capability tiers, and a practical delivery checklist.
LLM Deliverable Evaluation: A Technical Guide to Generation and Search-Augmented Behavior
1. Starting Point: Why “Looks Right” Does Not Mean “Usable”
When evaluating a large model’s front-end generation and interaction capabilities, the most common misjudgment is to mistake a demo-level result for an engineering-ready capability. An interface may render impressive lighting and particles, and the slider puzzle tiles may appear neatly aligned. Visually, everything seems to pass. But the conclusion changes completely once three questions are asked:
- Does the implementation intent stated by the model actually appear in the deliverable?
- Do the deliverable’s runtime parameters satisfy the common-sense rules and constraints of the gameplay itself?
- When a task requires multiple rounds of tool calls and fact-checking, does the model’s behavioral trace match its textual narrative?
These three questions point to the same issue: deliverable evaluation and answer evaluation are different capabilities. Answer evaluation only checks whether the final answer is correct. Deliverable evaluation must inspect a runnable, interactive, reproducible artifact. The former allows a model to “sound right”; the latter requires it to “do right.” Most experiences of unmet expectations occur at the boundary between the two.
Therefore, the first step in evaluation design is not to switch models, but to split tasks into layers that can be judged automatically and to define clear failure modes and verification signals for each layer.
2. Reconstructing Evaluation Dimensions: Mapping Vague Impressions to Seven Failure Modes
Start with a failure-mode reference table. Every subsequent test case should fall into at least one of these categories. This table is also the skeleton for automated scoring.
| Failure Mode | Typical Symptom | Automatable Verification Signal |
|---|---|---|
| Declaration–deliverable mismatch | The reasoning claims a certain style or mechanism, but the final artifact does not contain it | Coverage of keywords or style features in the deliverable |
| Spatial alignment error | Grid coordinates do not match the rendering coordinate system; elements overlap or are misaligned | Uniqueness of position mapping; overlap area ratio |
| Physical common-sense error | Layout parameters make the target action physically unreachable | Parameter sweep + solvability search |
| Runnability defect | Black screen, failure to start, or unresponsive interaction | Startup error capture; non-empty first-frame pixel ratio |
| Algorithm non-convergence | Policy shows no improvement after training iterations | Reward curve slope; successful policy ratio |
| Untruthful tool-call narration | Claims N calls while the actual count is M (M≠N) | Comparison between call-log counts and narrative claims |
| Factual hallucination | Completes a non-existent configuration item or parameter | Citation coverage; negative-constraint checks |
The value of this table is that it translates “feels bad” into conditions that can be written as assertions. The result can then be integrated into CI instead of relying on manual video review every time.
3. Perception Layer: Layered Acceptance for Visual and Motion Generation
For requirements such as “generate an animated page with lighting particles,” evaluation should be split into four progressive layers rather than assigned a single overall impression score.
Layer 1: Structural validity. Does the page have a complete document structure? Are styles inline or loadable? Is there obvious layout collapse, such as all elements stacked in the upper-left corner?
Layer 2: Observability. This is the most frequently overlooked yet most critical layer. Many deliverables produce a near-black background with low-luminance foreground elements, resulting in “rendering complete but invisible.” Observability can be automatically judged by taking a headless-browser screenshot and performing pixel statistics, as illustrated below:
function checkVisibility(screenshot):img = decode(screenshot)lum = perPixelLuminance(img) # 0..1hist = histogram(lum)if mean(lum) < LOW_THRESHOLD: # overall too darkfail("insufficient overall luminance")foreground = pixelsWhere(lum > backgroundLum + DELTA)if areaRatio(foreground) < MIN_AREA: # valid content ratio too lowfail("insufficient visible area")contrast = contrastRatio(bgLum, fgLum)if contrast < WCAG_AA: # contrast below readable standardfail("insufficient foreground-background contrast")return pass()
Layer 3: Timing and performance. Does the motion have an explicit timeline (entrance, sustain, exit)? Is the frame rate stable? Is first-frame time within threshold? Is there a persistent resource leak, such as monotonically decreasing frame rate after long runtime?
Layer 4: Style consistency. This layer must return to “declaration–deliverable consistency.” If the model promises a specific visual language during reasoning—for example, ink-wash diffusion and negative space, or wood-grain texture—the deliverable must contain recognizable implementation means such as texture maps, edge softening, negative-space ratio, or brushstroke noise. If the reasoning mentions it but the deliverable contains none of it, it should be recorded as style hallucination. This is not an aesthetic issue; it is direct evidence that planning and execution have diverged. Such records are more diagnostically valuable than any single score.
4. Spatial Layer: Designing an Automated Verifier for Puzzle Tasks
Slider puzzles (such as the 15-puzzle) appear to be visual tasks, but they are essentially constraint satisfaction + state search tasks. Their advantage is that the terminal state is fully automatically judgeable, so they should serve as benchmark cases for spatial capability.
The verifier needs to answer four questions:
- Completeness: Are all tiles present? Do their count and dimensions match the layout declaration?
- One-to-one correspondence: Is each tile’s position mapping unique? Are two tiles overlapping, or is one missing?
- Validity: Is the gap between adjacent tiles the agreed unit distance? Is there subpixel jitter that causes “looks aligned but is actually misaligned”?
- Solvability: Is the terminal state the target arrangement, and is the move sequence from the initial layout legal?
Illustrative pseudocode:
function verifyPuzzle(state, goal, grid):# 1) completenessassert count(state.tiles) == count(goal.tiles)# 2) one-to-one correspondence + no overlapoccupied = {}for t in state.tiles:cell = grid.cellOf(t.center) # world coordinates -> grid coordinatesassert cell not in occupied, "position overlap"occupied[cell] = t.id# 3) gap and alignment: allowed error must be explicitfor (a, b) in adjacentPairs(state.tiles):gap = distance(a.edge, b.edge)assert abs(gap - grid.unitGap) <= EPS, "gap mismatch"# 4) target-state consistencyassert multiset(occupied) == multiset(goal.tiles), "arrangement not equivalent"# 5) move legality (if the model outputs a solution sequence)s = initialStatefor mv in solution:assert isLegalMove(s, mv), "illegal move"s = apply(s, mv)assert s == goal, "solution does not reach the goal"return pass()
This verifier introduces an important methodology: for any task with a judgeable terminal state, do not rely on manual visual inspection. Visual inspection is easily persuaded by appearances such as “the positions are all aligned,” while automated verification immediately exposes subpixel misalignment and duplicate occupancy.
5. Physics and Common-Sense Layer: Playability Is a Hard Constraint
3D physics interactions, such as rolling a ball to knock down pins, expose the most representative problems: the rendering pipeline runs, geometry loads, and collisions take effect, but the spacing between pins is too large, so no throw can clear the full set. Technically, no error is reported; from an engineering perspective, it is unusable.
The root cause of this type of defect is not rendering capability but missing common-sense constraints: the model has learned “how to generate a physics scene,” but it has not translated the gameplay objective back into inequality constraints on layout parameters. Evaluation should treat “playability” as a separate item:
- Parameter sweep: Sample layout parameters (spacing, mass ratio, initial velocity range) in batches within reasonable intervals, and count the proportion of cases where “a successful path exists.” If the success rate is 0, the parameter domain has been set in an unreachable region.
- Reachability proof: Exhaustively or heuristically search key actions to explicitly determine “whether there exists a throw that achieves the goal.” This is the only reliable way to distinguish “high difficulty” from “logically impossible.”
- Reasonable parameter intervals: Compare constants such as gravity, restitution, and friction with widely accepted physical ranges in the industry; values outside the range are recorded as constant errors.
- Playability acceptance: Do not require the result to be “fun,” but require that “at least one legal operation sequence exists that completes the goal.”
In the evaluation report, conclusions should be expressed as “the implementation has solvability 0/1 within parameter intervals P1..Pn,” not as “bad experience.” The former can drive fixes; the latter only expresses sentiment.
6. Algorithm Layer: Correctness and Convergence of Reinforcement Learning Scripts
“Implement a grid-world maze and train an agent with tabular Q-learning” is a highly valuable evaluation case because it simultaneously examines algorithm correctness, rendering-logic consistency, and interactivity. In practice, this type of task has the highest failure rate, and the failure modes are highly structured. They can be grouped into four categories:
- Startup-layer failure: The script cannot run due to missing dependencies or syntax errors. Signals: non-zero exit code, stack trace exceptions.
- Rendering-layer failure: The maze cannot be drawn, or the agent size is far larger than the corridor width and occupies half the scene. Signals: non-empty pixel ratio; ratio of the agent bounding box to the grid cell size.
- Alignment-layer failure: The maze grid and the agent’s action coordinate system are inconsistent (for example, rendering uses cell indices while logic uses pixel coordinates), causing “the agent walks through walls” or displacement not equal to one cell. Signals: bijection assertions between logical coordinates and rendering coordinates.
- Algorithm-layer failure: Reward design is too sparse, the state-action space is defined incorrectly, or the update formula is implemented incorrectly, causing training not to converge. Signals: slope of the moving average of the reward curve; success rate of the greedy policy.
Among these, “alignment-layer failure” is especially worth adding to the regression set because it is often invisible visually—the agent is indeed moving, just not correctly. Automated assertions should explicitly check coordinate mapping:
# The logic layer and rendering layer must share the same coordinate definitionfunction toWorld(cell): # cell index -> world coordinatesreturn Vector2(cell.col * CELL_SIZE, cell.row * CELL_SIZE)function toCell(world):col = floor(world.x / CELL_SIZE)row = floor(world.y / CELL_SIZE)assert col != -1 and row != -1, "out of bounds"return Grid(col, row)# Validate immediately after each interaction stepbefore = toCell(agent.worldPos)action = policy.choose(before)agent.worldPos = agent.worldPos + delta(action)after = toCell(agent.worldPos)assert manhattan(before, after) == 1, "one-step displacement not aligned to grid"assert grid.isWalkable(after), "entered a wall"
For the training loop itself, a minimal trusted implementation can be used as a reference baseline, focusing on termination conditions, reward shaping, and convergence judgment:
initialize Q[s][a] = 0for episode in 1..N:s = env.reset()done = falsewhile not done:a = epsilonGreedy(Q[s], epsilon)(s2, reward, done) = env.step(s, a)target = reward + GAMMA * max(Q[s2]) * (1 - done)Q[s][a] += ALPHA * (target - Q[s][a]) # terminal state must zero future returns = s2if episode % EVAL_EVERY == 0:record(greedySuccessRate(policy(Q))) # evaluate with greedy policy, not training-time policyif meanRewardSlope(recentWindow) <= 0:fail("training did not converge")
One evaluation detail deserves special emphasis: training-time success rate is not equal to policy quality, because exploration noise can mask the true level of the learned policy. Convergence judgment must be measured with the greedy policy on independent evaluation episodes. Many implementations whose “training curve looks like it is rising” are actually only showing pseudo-improvement caused by exploration decay.
In addition, presenting Q-values as a heatmap or numeric overlay on the grid is an excellent visualization choice worth encouraging—it turns “whether learning actually occurred” from an abstract curve into a verifiable spatial distribution. During evaluation, “whether the visualization is consistent with internal state” can be listed as a bonus item.
7. Fact Layer: Multi-Round Search Counts, Provenance, and Consistency
When a task enters the scenario of “run multiple retrieval rounds and provide a structured conclusion,” the evaluation focus shifts from the deliverable to behavioral truthfulness. There are two high-frequency defect types.
Type 1: Call count does not match the narrative. The task explicitly requires four retrieval rounds, but only one actually occurs. The model then repeatedly reuses the first round’s results in the body while claiming that multiple rounds were completed. This type of inconsistency is a serious credibility problem and is fully automatable:
- Record tool-call events on both the client and gateway sides (timestamp, query terms, returned summary, hit count);
- Compare “I performed N retrieval rounds” in the model output with the actual event count;
- Output a consistency matrix: whether each round’s conclusion cites evidence newly added in that round. If all rounds’ conclusions can be derived from first-round evidence, classify it as “pseudo multi-round.”
Type 2: Factual completion and loss of negative constraints. For example, when asked to list a category of equipment configurations, the model completes a configuration item that does not exist, and this item is a key parameter. The repair cost of such errors is extremely high because it directly contaminates downstream decisions. The corresponding evaluation method is to construct negative-constraint cases: explicitly provide exclusion conditions in the prompt, check whether the model obeys them, and check whether it proactively states that “the item does not exist.”
A reusable structured-output template can significantly reduce such errors by forcing the retrieval process to be externalized:
{"round": 2,"query": "keywords and constraints for this retrieval round","evidence": [{"source_ref": "S2-1", "snippet_digest": "key hit points", "supports": "conclusion id"}],"conclusion": "new conclusion from this round","gaps": ["information gaps still uncovered"],"negative_constraints": ["confirmed non-existent configuration items and basis"]}
Three accompanying verification points:
- Citation coverage: Can every factual statement in the answer be traced to a
source_ref? The proportion of uncited statements is the hallucination-risk indicator. - Incremental information rate: Does round k introduce new evidence relative to rounds 1..k-1? If it is continuously zero, multi-round search has degenerated into single-round search.
- Consistency sampling: Sample key configuration items in the answer and compare them item by item with authoritative sources, focusing on “extra” items rather than “missing” items—fabrication often appears as invention from nothing.
It is also recommended to define the execution semantics of call counts explicitly. In requirements, distinguish “at least N rounds” from “exactly N rounds.” Under “at least” semantics, most models produce variable counts (possibly far more than N); only under “exactly” semantics do they align strictly. If the requirement side does not specify the semantics, evaluation results have no comparable baseline. In practice, asking the model to explicitly state before the end of each round “whether this round has covered the previous round’s information gaps” is often more effective than repeatedly adding prompts.
8. Engineering the Evaluation Pipeline
Only by connecting the above dimensions into a pipeline can version-to-version comparison and longitudinal regression be supported. It is recommended to build in five steps:
Layered case set. Prepare a set of task templates for each dimension: perception layer (motion and theme style), spatial layer (puzzle and grid layout), physics layer (3D interaction), algorithm layer (reinforcement learning script), and fact layer (multi-round retrieval). Each layer should contain at least one benchmark case with an automatically judgeable terminal state to provide hard metrics.
Sandbox execution. Run model-generated code in isolated containerized environments, use unified headless-browser rendering, and uniformly collect four types of data: runtime logs and error stacks, screenshot sequences, resource and frame-rate metrics, and tool-call event streams. Logs and traces enter a logging service for structured storage; artifacts and screenshots are archived in object storage for version comparison.
Automated verification first. For each case, run assertions first (layout verifier, solvability search, convergence judgment, count comparison), and then have humans review only boundary samples determined by automated checks. The goal is to reduce manual review volume by an order of magnitude while retaining judgment on subjective dimensions such as style consistency.
Metrics dashboard and alerts. Recommended core metrics: task pass rate, first-frame time, average frame rate, failure-mode distribution, average retrieval rounds and incremental information rate, citation coverage, hallucination rate, token usage, and latency. Set thresholds and alerts for key metrics, and automatically trigger regression when anomalous fluctuations occur.
Failure-case accumulation. Every real failure should be converted into a new regression case. The accumulation rate of failure samples often predicts the pace of capability improvement better than the absolute score of any single evaluation.
Finally, one point must be acknowledged: it is normal for different models to have different strengths and weaknesses across dimensions, and a single aggregate score easily hides critical defects. Evaluation reports should output a dimension radar + failure-mode details, not a single total score.
9. Expectation Management: Three Capability Tiers and an Acceptance Checklist
Many cases of “expectations too high” come from failing to distinguish three levels of model capability:
- Demo level: Visuals and interaction run, but parameters, alignment, and common sense have defects and require extensive manual correction;
- Iterable level: Core logic is clear, terminal states are correct, specific defects can be localized, and delivery requires only minor manual modification;
- Production level: Passes all automated assertions, including playability, coordinate alignment, convergence, and citation coverage, and can enter business workflows directly.
Mapping measured results onto these three tiers changes the discussion from “does it work at all?” to “which tier is it in now, and which item is missing?” The following acceptance checklist can be used directly for delivery gatekeeping of generative tasks:
- Every implementation claim in reasoning has a corresponding implementation in the deliverable
- Startup has no exceptions, the first frame is non-empty, and visible area and contrast meet standards
- Terminal arrangement is unique, has no overlap, and gaps match the agreement
- Physical parameters fall within reasonable intervals, and at least one legal operation sequence exists that achieves the goal
- Logical coordinates and rendering coordinates are bijective, and step-by-step displacement can be asserted
- Training is evaluated with a greedy policy, and convergence is judged by moving-average slope rather than the training-time curve
- Tool-call counts match the narrative, and each round’s conclusion cites evidence newly added in that round
- Every key configuration item in the answer is traceable, and negative constraints are explicitly obeyed
- All assertions are integrated into CI, and failure cases are added to regression storage
Overall, the right approach to evaluating these models is not to search for “the strongest one,” but to split capabilities into fine-grained parts using automatically judgeable assertions, assign failures to specific layers, and use a regression set to protect already-solved problems. Visual and interaction completeness is only an entry metric; the real dividing line is whether declarations match deliverables, whether parameters satisfy common sense, whether algorithms converge, and whether retrieval behavior is truthful. Making these items hard gates will produce stable, comparable evaluation conclusions and provide a defensible calibration of “expected value.”
