Evaluating LLMs on Interactive Generation Tasks: Failure Modes and Acceptance Methods
This article examines how to evaluate large language models (LLMs) on interactive generation tasks—tasks that go beyond visual demos to require correct behavior under constraints. Drawing on a comparative spot test of mainstream models across five task types (static visuals, geometric puzzles, 3D physics, reinforcement learning, and retrieval QA), it argues that visual polish is no longer a meaningful differentiator and that the real gaps appear in the parameter and constraint layers. The article categorizes common failure modes, explains why reasoning chains and prose cannot serve as acceptance criteria, and proposes a five-layer acceptance model (P0–P4) with executable assertions, multi-run statistical metrics, prompt constraint templates, and an automated evaluation pipeline. It is aimed at developers and enterprise users who want reproducible, decision-ready measurements of model capability rather than impressive but unreliable demos.
1. Introduction: When “Looks Cool” No Longer Means “Usable”
After a wave of new large language models (LLMs) shipped, many vendors highlighted the same capability in their demos: generate an interactive app from a single sentence. You type a one-line request, and the model returns a page you can open directly in a browser—complete with rendering, animation, physics effects, and sometimes an entire small game. These demos spread quickly, each more polished than the last with light and particles, 3D scenes, and so on.
But once you apply stricter acceptance criteria, visual quality turns out to be the wrong thing to measure. The real dividing line is this: can the model work correctly the whole time on a task that has clear constraints, inputs, and state?
This article uses a comparative spot test of several mainstream LLMs as a thread. The tasks cover five categories:
- Static visual presentation: themed page generation, testing expressiveness in lighting, materials, and composition;
- Geometric placement: a Klotski sliding-block puzzle, requiring a correct, solvable, interactive layout;
- 3D physics: a bowling scene, testing collision simulation and default scene parameters;
- Reinforcement learning: a Q-learning demo in a maze, testing algorithm implementation, rendering alignment, and visualization;
- Retrieval QA: forcing the model to run a specified number of retrieval rounds, testing whether it follows the count constraint and whether the conclusions are factually accurate.
All tested models were the latest versions called through official online entry points, and the “web/mobile” tests were real usage scenarios of publicly available entry points. They are referred to below as “a certain domestic LLM” and “a certain cloud vendor’s model.” The conclusions are not aimed at any single product; they point to a more general question: by what standard should we measure how capable LLMs are on interactive tasks?
2. The Nature of Generative Interactive Tasks: Three Overlapping Layers and Error Amplification
To understand why failures happen, first understand the nature of this kind of task. A “small app” generated from a single request can be broken into three layers:
- Rendering layer: DOM layout, Canvas/WebGL drawing, shaders and materials, animation loops;
- Logic layer: state machines, event handling, coordinate transforms, collision detection, solving or training algorithms;
- Parameter layer: concrete values such as object sizes, physical constants, color and contrast, retrieval scope.
The three layers are strongly coupled. A single wrong default value in the parameter layer can still look “nice” at the rendering layer, while the result at the logic layer is already wrong. The model compresses all three layers into one generation pass, so any omission in any layer is amplified into something “unusable.”
Another widely overlooked factor is the “black-screen interface.” In pure visual mode, if the layout or logic is broken, the model cannot see the rendered result and therefore cannot detect problems like imbalanced sizing or misaligned layers on its own. This limitation must be accounted for when designing evaluation tasks.
3. Visual Presentation: A Dimension Where Differentiation Is Disappearing
The conclusion first: on pure visual presentation, the gap between mainstream models has clearly narrowed. Lighting, particles, materials, and gradients all look “convincing,” and some models don’t even aim for these effects—which reflects product positioning more than capability. Visual presentation should not be the main differentiator in evaluation.
A more valuable question lies in the mismatch between what models “say” and what they “do.”
In one Klotski task, the model’s chain of thought clearly stated it would present an ink-wash (shuimo) style, but the generated page had nothing to do with that and remained a generic graphic style. Such an “ink-wash” look can hardly be implemented directly through a general drawing API; the so-called style is closer to a verbal hallucination in the reasoning process: the model wrote down the design intent in its thinking but never realized it in code.
A similar phenomenon appears in retrieval tasks (see Section 6): the model claims it ran several rounds of retrieval but actually completed only one.
The common thread is that the model outputs “narrative-layer” content, while acceptance must look at “artifact-layer” content. This yields the first evaluation principle: neither the chain of thought nor the prose can serve as an acceptance basis; they can only be clues for locating problems.
4. Geometry and Physics Tasks: Knowledge Gaps Surface at “Common-Sense Parameters”
4.1 Klotski: A Correct Layout Does Not Equal Correct Interaction
The Klotski portion performed relatively best: blocks were placed neatly, with no overlap or out-of-bounds. But “correct layout” only clears the first hurdle. An interactive puzzle must at least also satisfy:
- Solve correctness: the initial layout is solvable, and states remain consistent after moves;
- Interaction loop: clicks or drags trigger moves, collision rules are correct, and the end state can be recognized;
- Visual readability: blocks have clear boundaries and the target block is identifiable.
Many models simply write a preset layout when generating this kind of page. That guarantees a correct opening but not correct interaction.
4.2 Bowling: Failure Is in the Parameters, Not the Code
The bowling scene runs and 3D rendering is fine, but it has two typical problems:
- The pins are spaced too far apart, so a single throw cannot knock down all the pins. This is a common-sense error: real pin spacing is far smaller than on the order of the pin body’s diameter. When the model lacks physical common sense or lacks parameter constraints, it tends to pick a “looks reasonable” value for each object independently, rather than making parameters obey physical laws;
- The scene is overall too dark, making the lane, pins, and ball hard to distinguish. This shows the model optimized for “atmosphere” rather than “playability.”
Both points reflect a missing parameter layer: there is no explicitly declared parameter table, and no step that checks whether parameters satisfy physical common sense.
5. Reinforcement Learning Tasks: The “Deep Water” of Interactive Tasks
5.1 Four Failures, Four Different Failure Modes
The maze Q-learning task best illustrates the problem. For fairness, the same task was attempted four times, with these results:
- First: the maze was drawn, but the agent was too large, nearly occupying half the maze—showing a lack of a ratio constraint between “character size and scene scale”;
- Second: the maze was not rendered at all—an overall rendering-layer failure;
- Third: the maze could be displayed but could not be operated—meaning only the rendering pipeline was completed, not the closed loop from input to state;
- Fourth: both the maze and the agent could be displayed, but the agent was not aligned with the terrain and “walked through walls” or floated outside cells when moving—a typical symptom of a broken coordinate-transform chain.
5.2 What Deserves Recognition
Even though all four attempts failed to pass, the task had a bright spot: the model encoded Q-values as a heatmap on the maze floor, so the agent could intuitively see the current value distribution upon reaching each cell; the training process eventually converged.
This is a good design idea. Existing research (visualization analyses of Atari agents, for example) shows that the spatial distribution of the value function is an important basis for understanding agent behavior. Being able to complete “algorithm + visualization” in a single generation indicates the model’s understanding of reinforcement learning is real.
5.3 Quick Reference for Typical Problems
| Failure type | Typical symptom | Root cause | Acceptance checkpoint |
|---|---|---|---|
| Scale mismatch | Agent too large, object proportions off | Missing size and scale constraints | Ratio of entity size to scene scale |
| Missing rendering | Maze not drawn, blank scene | Init order, resource loading | First-frame screenshot and error logs |
| Broken interaction | Frame renders but can’t be operated | Event binding, focus management | Whether state changes after input |
| Coordinate misalignment | Agent not aligned with terrain | Inconsistent coordinate-transform chain | Mapping between pixel and state coordinates |
| Style not realized | Claims ink-wash but no matching implementation | Chain of thought disconnected from artifact | Item-by-item check of claims vs. artifact |
6. Multi-Turn Retrieval: From “Retrieval Capability” to “Constraint Following and Fact-Checking”
The last set of tests focused on “multi-turn retrieval.” The same question was tested separately on web and mobile, requiring the model to run four rounds of retrieval.
6.1 Web: Claimed Execution Does Not Match Reality
The model claimed in the body that it performed multiple rounds of retrieval, but actually executed only one, with all later conclusions reusing the first-round results. This is not a capability problem but a constraint-following problem:
- The number of retrieval rounds is a hard instruction, and the model should treat it as a program constraint, not a descriptive goal;
- There is a systematic gap between “claiming to execute” and “actually executing.” Generative models tend to make the narrative self-consistent rather than the behavior self-consistent.
6.2 Mobile: Out-of-Control Count and Key-Parameter Hallucination
The same task on mobile performed as follows: the number of retrievals far exceeded the requirement (more than a dozen), yet it still erred on a key parameter—adding a dual-chamber air suspension configuration to a vehicle model that does not have it.
This kind of error is especially worth watching, for three reasons:
- It falls in the parameter layer, which is exactly the information users rely on most directly when making decisions;
- Vehicle configuration is itself verifiable objective fact, yet the model filled in unknown items with “common configurations”;
- It shows that after multiple retrievals, the model did not cross-validate the retrieved snippets against its conclusions—parameters were “inherited” rather than “verified.”
6.3 A Reference for Good Practices
In comparable tasks, the better implementations shared two traits:
- They executed exactly the specified number of rounds, no more, no less, showing that retrieval depth was treated as a hard constraint;
- They proactively flagged corrections for key parameters, explicitly stating that the vehicle had no dual-chamber air suspension configuration. This shows the model cross-checked on top of the retrieval results rather than accepting them outright.
Accordingly, evaluation of retrieval tasks should cover four dimensions:
- Constraint following: whether the number of rounds is exact and verifiable;
- Result completeness: whether retrieved information was actually used;
- Factual accuracy: whether key parameters and configuration items match the sources;
- Proactive correction: whether the model flags uncertainty or gives a negative answer when facing conflicting or unreliable retrieval results.
7. Evaluation Methodology: Layered Acceptance and Executable Assertions
The preceding analysis can be distilled into an executable methodology: turn “is it good?” into “does it pass?”
7.1 Five-Layer Acceptance Model
Adopt a five-level, progressively stricter standard; a failure at any layer marks that layer as not passed:
- P0 Loadable: the page has no errors and the first screen finishes rendering within a time limit;
- P1 Interactive: input causes a state change (movement, click, drag, or training start);
- P2 Semantically correct: no overlap or out-of-bounds in the layout, coordinates aligned, physical parameters satisfy common-sense constraints;
- P3 Presentation adequate: training converges, visualization is readable, color contrast is compliant;
- P4 Claim-consistent: every description in the body has a corresponding implementation in the artifact.
7.2 Writing Assertions into the Evaluation Script
Layered standards only have meaning when written as assertions. Sample code:
def verify_interactive_app(app_url: str, cfg: dict) -> dict:"""Automated acceptance for interactive generation tasks (sample)cfg: scene scale, max entity size, retrieval rounds, color contrast threshold, etc."""page = open_headless_browser(app_url)page.wait_until_loaded(timeout=cfg["load_timeout_ms"])result = {}# P0 Loadable: no uncaught errorsresult["P0"] = len(page.console_errors()) == 0# P1 Interactive: after one input, state should changebefore = page.snapshot_state()page.dispatch_input(cfg["test_action"]) # e.g. arrow key "up"page.wait_for_timeout(200)after = page.snapshot_state()result["P1"] = before != after# P2 Alignment: sample entity and background pixels uniformly, verify coordinate consistencyresult["P2_aligned"] = check_layer_alignment(page, cfg["anchor_grid"])# P2 Scale: agent bounding box must not exceed the configured ratio of cell side lengthresult["P2_scale"] = page.entity_bbox("agent") <= cfg["max_agent_cell_ratio"] * cell_size(page)# P3 Readability: foreground/background contrast meets thresholdresult["P3_contrast"] = contrast_ratio(page, region="playfield") >= cfg["min_contrast"]# P4 Claim consistency: each capability claimed in the body is verifiable in the artifactresult["P4_claim_match"] = match_claims(app_url, claims=app.description)return result
The value of this script is that it turns the subjective judgment of “I looked at it and it seems fine” into repeatable, regression-friendly automated checks. When a model iterates, the same script can be reused directly to compare version differences.
7.3 Statistical Metrics: Use Multiple Sampling Instead of Single Success/Failure
The four failures in the maze task show that outputs for interactive tasks have significant variance. A single test can both underestimate capability and overestimate stability. Recommended metrics:
- pass@1: the proportion of single generations that pass all assertions, measuring practical usability;
- pass@k: the proportion of at least one pass across k independent attempts, measuring the capability ceiling;
- First-pass rate and average number of attempts: measuring operability.
Combining the three is what distinguishes a “capability ceiling” from “stable output.”
8. Prompt Engineering: Put Constraints Up Front
Beyond evaluation, improving generation success on the business side is equally important. The core idea: make explicit the implicit constraints scattered in the chain of thought, turning them into verifiable conditions.
8.1 Constraint Template for Interactive Tasks
[Task] Build an interactive Q-learning maze demo as a single-file HTML page.[Hard constraints]1. Maze is 21×21, walls take up ~22%, and a valid path from start to goal is guaranteed;2. Agent diameter ≤ 60% of cell side length, using the same coordinate conversion as the walls;3. Floor encodes Q-values as a heatmap, legend fixed to [0, 1];4. Must support arrow-key movement, collision detection, reset, and step-count display;5. After 500 training episodes, show a per-episode return convergence curve.[Self-check list] Before output, confirm each item: □ no console errors □ entities aligned with terrain□ foreground/background contrast meets standard □ description maps one-to-one to implementation[Delivery note] The body describes only implemented capabilities; unimplemented items are explicitly marked "not implemented."
Three points matter here:
- Replace adjectives like “good-looking” and “reasonable” with measurable thresholds;
- Require the model to output self-check results so that claims align with the artifact;
- Specify the rule for marking “not implemented” to avoid exaggeration in the body.
8.2 Constraint Template for Retrieval Tasks
[Retrieval constraints]1. Retrieve for exactly the specified number of rounds; in each round, list the query terms and sources in the body;2. All parameter-type conclusions must cite verifiable sources; anything that cannot be cross-verified is marked "unconfirmed";3. If a key configuration lacks a reliable source, explicitly state "refer to the official specification sheet."
These constraints directly target the two earlier problems: out-of-control retrieval counts and parameter hallucination.
9. Engineering Implementation: How to Build the Evaluation Pipeline
Model capability evaluation should not stop at manual spot checks; it should be consolidated into a pipeline:
- Task orchestration: use configuration to manage the task set, constraints, and acceptance scripts, with support for grouping by scenario;
- Sandbox execution: run generated artifacts in an isolated container environment, automatically opening them with a headless browser to screenshot, inject input, and collect console logs;
- Automated assertions: run check scripts layer by layer per P0–P4 and output structured results;
- Attribution analysis: categorize failure samples by “rendering layer / logic layer / parameter layer / constraint layer” to form failure profiles;
- Regression monitoring: automatically re-run after each model version update and compare changes in key metrics.
In practice, you can rely on general cloud infrastructure capabilities:
- A container platform hosts the batch execution environment;
- Object storage archives page snapshots and run logs;
- A logging service collects assertion results centrally;
- Monitoring and alerting set thresholds for metrics such as pass-rate drops.
Providers such as Baidu AI Cloud offer these same general-purpose capabilities in their model services—not a custom solution built for one specific evaluation scenario. The key point is that this pipeline relies on general engineering capability, not on the traits of any one model, and can be reused across any model service.
10. Conclusion: From “Stunning Generation” to “Rigorous Acceptance”
The overall judgment from this spot test: mainstream LLMs have generally met the bar on visual presentation, but still show clear gaps on “working correctly under constraints,” and this gap will not disappear automatically with model version updates.
Specifically:
- The real difficulty in generative interactive tasks lies in the parameter and constraint layers, not in the amount of code;
- Neither the chain of thought nor the prose can serve as an acceptance basis—the artifact must be the standard;
- The capability ceiling of multi-turn retrieval depends on constraint following and fact-checking, not on the number of retrievals;
- Translating “is it good?” into executable assertions is the only way to make evaluation sustainable.
For developers and enterprise users, the sensible approach is not to bet on the “most dazzling demo,” but to first break down your own scenario into assertable acceptance points, then measure continuously against layered standards. That way, no matter how models iterate, you get conclusions that are comparable, regression-friendly, and usable for decisions.
