Evaluating LLM-Generated Interactive Demos: Key Acceptance Criteria and Common Pitfalls
This article explains how to evaluate the front-end code generation capabilities of mainstream large language models through interactive demos. It covers why interactive demos act as a high-fidelity test of model ability, breaks the work into three task categories—visualization, geometry and physics, and algorithm visualization—and presents a unified acceptance framework built on five key criteria and a three-layer checking process. It also analyzes common failure modes and their root causes, offers prompt engineering strategies for producing verifiable output, and describes an automated evaluation pipeline. Readers will learn how to decompose interactive generation tasks, design acceptance criteria, identify which layer a failure occurs in, and improve first-pass delivery quality through prompt engineering and automation.
When an interactive demo opens, runs, and can be played through, does it actually look the way it was supposed to? I recently tested the front-end code generation capabilities of several mainstream large language models using a few common visualization scenarios. The overall impression can be summed up as: better than expected, but not fully delivered. Clean, transparent interfaces in light themes, complete technical roadmaps laid out in the chain of thought, and reinforcement learning algorithms whose training curves genuinely converge — all of these stood out at first glance. Yet once you actually run the output, or check it against the standards given in the chain of thought, the gap between what was promised on paper and what was delivered keeps surfacing. This article does not evaluate any specific product. Instead, it abstracts a reusable evaluation method from these observations: how to decompose interactive generation tasks, how to design acceptance criteria, at which layer common failure modes occur, and how to improve the quality of first-pass delivery through prompt engineering and automated pipelines.
1. Why Use Interactive Demos to Evaluate Code Generation
1.1 Interactive Demos Are a High-Fidelity “Mirror” of Capability
Compared with answering an algorithm question or explaining an error message, asking a model to “build a demo that runs, is playable, and looks right” tests several capabilities at once:
- End-to-end engineering ability: you have to organize the HTML structure, styling, canvas rendering, and event handling in a single output, and any single omission leaves a blank page.
- Commonsense reasoning: whether the physics look realistic, whether the geometric proportions are reasonable, and whether interface elements are placed sensibly all end up displayed directly on screen.
- Cross-concept integration: the algorithm, visualization, and interaction must be unified into one coordinate system and one state model — for example, making the agent stand exactly in the center of a path cell.
- Self-consistency: whether the effects promised in the chain of thought (such as “using an ink-wash style”) are actually delivered in the output.
More importantly, the results are highly objective. Whether it opens correctly, whether the operations behave as expected, and whether the screenshot matches the requirements can all be judged directly, with almost no subjective interpretation needed.
1.2 A Key Premise: Non-Deterministic Output Must Be Evaluated Statistically
Evaluating code generation is not like evaluating a deterministic program, where a single run settles the question. With the same sampling settings, the same model facing the same prompt can produce noticeably different output across rounds. Even within a single round, the quality of the reasoning stage and the actual generation stage may not match — the reasoning may be sound while the code that gets written falls short. So a reasonable approach is to:
- Fix the prompt, sampling parameters, and runtime environment;
- Run each task multiple times and record the first-pass rate;
- Report the pass rate together with the distribution of failure types, rather than drawing conclusions from a single “crash” or “wow” moment.
2. Three Typical Task Types: Each With Its Own Difficulty
2.1 Visualization Generation: Testing “Delivery of the Visual Promise”
For tasks like “generate a set of flowing particle light-and-shadow effects,” the point is not how complex the shader is, but three things:
- First-frame appearance: right after the page loads, it should immediately show a layered scene, not a single lone moving blob.
- Stable performance: driven frame by frame via requestAnimationFrame, it should output near the screen refresh rate on an ordinary laptop without noticeable frame drops.
- Theme fidelity: if the prompt asks for an “ink-wash” or “Chinese traditional” mood, the output should express it through color, brushwork, and negative space, not just in the text of the chain of thought.
A common observation in practice is that the chain of thought describes “ink bleeding and rice-paper texture” in detail, yet the final code contains nothing but a few circular particles moving around. This shows a disconnect between the model’s planning stage and its implementation stage.
2.2 Geometry and Physics Commonsense: Sliding-Block Puzzles and Collision Scenes
Sliding-block puzzles (such as a numbered puzzle in a Klotski-style layout) look simple but actually test both geometry and logic:
- Geometric correctness: the board size, cell side length, and border width calculations must be consistent; otherwise cells won’t line up and pointer positions will drift.
- Rule correctness: when pieces are of different sizes (such as a 2×2 block versus a 1×1 block), the movable range of each piece must be computed individually.
- The initial layout must be guaranteed solvable. You can generate the shuffle by making several random moves backward from the goal state, which avoids producing an unsolvable arrangement.
Collision scenes (such as an array of bowling pins) test physics commonsense:
- Real pins are arranged in an equilateral triangle, with the centers of any two adjacent pins about 30.5 cm (12 inches) apart and a ball diameter of about 21.8 cm. When the ball enters the pin array it triggers a chain reaction of collisions — this is exactly why “clearing everything with one ball” is possible.
- If the generated pin spacing is noticeably too large, it often means the developer (here, the model) confused “the distance between pins” with “the gap within the pin formation” during modeling, reflecting a weak grasp of the real scene.
- Even if the model scales the sizes down to fit the screen, it should preserve the real-world relative proportions. In a schematic lane scene, the visual gaps between pins usually look larger, but that is a matter of proportion trade-offs; the feel should still be kept reasonable through collision parameters (friction, coefficient of restitution, collision radius), rather than abandoning collision propagation.
In addition, a nearly all-black interface with illegible details is essentially a dark theme lacking layered design. It is advisable to define at least three gray levels — background layer, main content layer, and dividers — and to ensure the contrast between foreground text and background is no lower than 4.5:1, a general requirement in accessible design.
2.3 Algorithm Visualization: The Q-Learning Maze Combines “Algorithm + Rendering + Interaction”
In a maze scene, the model must accomplish three things at once:
- Implement Q-learning so that the agent converges to a feasible path;
- Render the maze, the agent, and the Q-value heatmap;
- Provide interactive controls such as start, pause, reset, and parameter adjustment.
The convergence step best reflects model quality. Q-learning converges when the following conditions are met:
- The state and action spaces are finite;
- Every “state-action” pair is visited infinitely often;
- The learning rate satisfies the Robbins-Monro conditions, meaning the sum of all learning rates is infinite while the sum of their squares is finite. In practice this is often approximated by decaying based on visit count, for example alpha = 1 / n.
Two points in practice are worth praising:
- Displaying the Q-values on the cells of the floor, making “how the agent evaluates each direction” intuitively visible, is a good visualization idea.
- Many models generate training processes that converge normally, showing that mainstream models already have a fairly solid understanding of standard algorithms.
The failures, though, usually happen in engineering details:
- The agent size doesn’t match the cell size (for example, the agent’s side length is about twice the cell), taking up most of the screen;
- Only the maze and the goal are generated, with no runnable main body;
- Walls and paths are computed separately during rendering, causing misalignment, so the agent can’t move or keeps passing through walls.
A practical way to distinguish the cases: if the algorithm core is correct but rendering and interaction have problems, prompt engineering plus a small scaffold can usually fix them; if even the state transitions or reward design are wrong, you need to revisit the spec from scratch.
3. A Unified Acceptance Framework: Five Key Points and Three Layers of Checks
To avoid “it looks good, so it passes,” it is advisable to break each demo’s acceptance into five key points:
| Acceptance item | Core question | How to check | Typical criterion |
|---|---|---|---|
| First render | Does it work the moment it opens? | Screenshot after load + console check | No blank area on the first screen, no errors |
| Logic correctness | Are the rules and algorithm right? | Interaction scripts + assertions | Operations follow the rules, training converges |
| Visual and experience | Is it clear and smooth to use? | Human review + screenshots | Contrast meets the standard, layout aligned, no obvious visual defects |
| Code quality | Is it maintainable? | Static reading + structure stats | Single file, clearly layered, no redundant dependencies |
| Stability | Does it run correctly across multiple runs? | Multi-round regression + fixed random seed | Stable first-pass rate, no random failures |
On this basis, the checking process can be divided into three layers:
- Parts a machine can judge automatically: whether the page loads successfully, whether the console reports errors, the number of DOM elements, whether interaction scripts exist, image size proportions, color contrast, and so on;
- Semi-automatic parts: using a headless browser to simulate drag, click, and keyboard operations, then judging based on the resulting state — for example, whether a game is completed, how many episodes were trained, whether it converged;
- Parts that require human judgment: aesthetics, theme fit, playability, and the consistency between the visuals and the prompt description.
It should be noted that “code quality” in the table above refers to evaluating the quality of the model’s output code itself. In concrete engineering, mature teams usually apply this together with unified development standards and engineering capabilities (such as static analysis, code review, and pipeline quality gates).
4. Typical Failure Modes and Root-Cause Analysis
4.1 Chain-of-Thought Promises Don’t Match the Final Output
Failure symptom: the chain of thought says “ink-wash bleeding,” but the actual output has only basic particles. This kind of problem is especially common in visual tasks.
Root-cause analysis:
- The text description in the planning stage is not converted into executable implementation constraints, such as specific color values, transparency gradients, or texture overlay layers;
- Visual style is a “hard-to-quantify” goal, and the model tends to gloss over it with abstract wording.
Improvement approach: break visual requirements into verifiable concrete metrics, for example:
- The main color palette (ink color, rice-paper base);
- At least how many texture overlay layers;
- Whether negative space is preserved.
And the corresponding implementation code must be pointable-to in the output.
4.2 Canvas Layering Causes Content to Overlap
Failure symptom: when a page has multiple canvases or WebGL scenes, the non-transparent regions of the upper canvas cover the underlying content, so either “the screen is black and nothing is visible” or “the view is mostly blocked.”
Root-cause analysis: multiple canvas layers default to being laid out in document flow, with each layer’s element positions computed independently and no unified layer management.
Improvement approach:
- Use one container to manage dimensions uniformly, with all layers using the same width and height and the same device-pixel-ratio scaling;
- Set pointer-events: none on opaque layers to prevent an upper canvas from intercepting mouse events meant for the layer below;
- Make layering explicit via zIndex, and give the canvas container a fixed aspect ratio, such as 16:9, to avoid repeated size changes.
Example code (using native Canvas, no third-party libraries):
<!-- Stack multiple canvas layers in one container with unified positioning and scaling --><div class="stage" style="position:relative;width:100%;aspect-ratio:16/9;"><canvas id="bg" style="position:absolute;inset:0;width:100%;height:100%;"></canvas><canvas id="main" style="position:absolute;inset:0;width:100%;height:100%;"></canvas></div><script>function setupCanvas(id) {const canvas = document.getElementById(id);const ratio = window.devicePixelRatio || 1;const { width, height } = canvas.getBoundingClientRect();canvas.width = Math.round(width * ratio);canvas.height = Math.round(height * ratio);const ctx = canvas.getContext('2d');ctx.setTransform(ratio, 0, 0, ratio, 0, 0); // keep logical coordinates aligned with CSS pixelsreturn { canvas, ctx, width, height };}</script>
4.3 Rendered Object Sizes Don’t Match the Scene
Failure symptom: in a maze, the agent takes up nearly half the maze; on a bowling lane, the pins are too large or too small, so collision detection doesn’t match the visual impression.
Root-cause analysis: the model lacks the concept of “relative proportion” and tends to use fixed pixel values or values given by intuition, without computing dynamically based on the container size.
Improvement approach:
- Base all sizes on the cell side length, for example cellSize = Math.min(available width / columns, available height / rows);
- Scale elements such as the agent, ball, and pins uniformly by proportion, and reserve a safety margin;
- Add proportion assertions in the self-check step, for example the agent’s width must not exceed 60% of the cell side length.
4.4 Reward Design and Convergence Failure
Failure symptom: after dozens of episodes of training, the agent circles in place and the return curve never converges.
Root-cause analysis:
- The reward is set only as “reach the goal +1,” making exploration extremely inefficient;
- The step penalty is too large or too small, causing the policy to degenerate into going in circles;
- The exploration rate of ε-greedy doesn’t decay with training, so the model stays in random exploration;
- The learning rate doesn’t decay with visit count, causing Q-values to oscillate back and forth.
Reference implementation (illustrative):
// Core Q-learning update: r is the immediate reward, Q[s][a] is the estimated value of action a in state sfunction qLearn(states, actions, env, opts) {const { gamma = 0.95, alpha0 = 0.2, eps0 = 1.0, epsMin = 0.05, episodes = 3000 } = opts;const Q = Object.create(null);const n = Object.create(null);const get = (s) => (Q[s] ||= Object.fromEntries(actions.map(a => [a, 0])));let eps = eps0;for (let ep = 0; ep < episodes; ep++) {let s = env.reset();while (!env.isDone(s)) {// ε-greedy: explore randomly with probability eps, otherwise pick the highest-valued actionconst a = Math.random() < eps? actions[(Math.random() * actions.length) | 0]: actions.reduce((best, x) => (get(s)[x] > get(s)[best] ? x : best), actions[0]);const { s: s2, r, done } = env.step(s, a);// Sample average: alpha = 1/n, which satisfies the Robbins-Monro conditionsn[s + '|' + a] = (n[s + '|' + a] || 0) + 1;const alpha = 1 / n[s + '|' + a];const target = r + (done ? 0 : gamma * Math.max(...actions.map(x => get(s2)[x])));get(s)[a] += alpha * (target - get(s)[a]);s = s2;}eps = Math.max(epsMin, eps * 0.995); // decay the exploration rate}return Q;}
Engineering suggestion: for deterministic environments like grid worlds, you can factor the MazeEnv state transition into a pure-function module (input state and action, output next state and reward) that both the visualization renderer and the automated tests can call. This avoids the problem of “the tested logic and the running logic being two separate pieces of code.”
4.5 Acceptance Instability Caused by Randomness
Failure symptom: generating the same prompt over multiple rounds, some rounds work while others have misplaced canvas layers or an abnormal agent size.
Root-cause analysis: the output has randomness, and combined with the lack of explicit acceptance constraints, some “low-probability defects” recur.
Improvement approach:
- Write key constraints as hard check items in the prompt (see Section 5);
- Fix the random seed for the generation process (if the platform supports it);
- Run multiple rounds of regression for the same prompt, and measure stability with the pass rate rather than a single-run verdict.
5. Prompt Engineering: Making Generated Output “Verifiable”
5.1 Describe the Task With a Structured Spec
Compared with “make a nice maze Q-learning demo,” a structured description can significantly raise the success rate of a single generation. Example (pseudo-YAML):
task: maze-qlearning-demooutput: a single HTML file, vanilla JS + Canvas, no external dependencieslayout:- top title bar; canvas in the middle (aspect ratio 16:9); control bar and metrics area at the bottomcanvas:- grid maze: rows and columns configurable, default 21×21- agent size = 60% of the cell side length, center-aligned- drawing layers: background -> walls/paths -> Q-value heatmap -> agent -> trajectoryinteraction:- buttons: start / pause / reset / single-step- parameters: learning rate, discount factor, exploration rate (sliders, effective in real time)algorithm:- Q-learning, state is the cell coordinates, actions are up/down/left/right- rewards: reaching the goal +10, each step -0.05, hitting a wall -0.5 and no movement- termination: do not compute future returns when doneacceptance:- no console errors after the page loads- the agent always stays at the cell center after running- converges within 3000 episodes; after convergence the optimal path length ≤ shortest path × 1.2- contrast ratio between all text and background ≥ 4.5:1self_check:- list the implementation location in the file for each acceptance item
5.2 Ask the Model to Output a “Self-Check List”
State this explicitly in the prompt: “After generating the code, list item by item how you satisfy each acceptance item below and point out the corresponding code location; if you cannot satisfy one, explain why.”
This step forces the model to align the “promises” in its chain of thought with the code implementation, noticeably reducing the “sounds nice but coded wrong” cases.
5.3 Staged Generation and Closed-Loop Repair
- First round: generate only a runnable skeleton (layout, canvas, basic interaction), and confirm there are no errors and no blanks.
- Second round: fill in the algorithm and rendering details.
- Third round: take the problems exposed during the first round’s run (such as console errors, overlapping elements, imbalanced proportions) as context, feed them back to the model, and ask for targeted fixes.
This “generate—run—feedback—regenerate” loop is usually more effective than a single, longer prompt. It takes more time, but yields more stable results.
6. An Automated Evaluation Pipeline: Turning Subjective Experience Into Regressable Metrics
6.1 Headless Browser + Assertions + Screenshots
Illustrative flow:
Prompt template --> generate code --> load in headless browser--> collect: console errors / page dimensions / element coordinates--> interaction scripts: drag, click, keyboard--> assertions: rule correctness, convergence, proportion constraints--> screenshot archive --> metrics dashboard
Example assertion (pseudo-code):
# Check whether the agent always stays aligned with the cell centerdef assert_agent_aligned(page, cell_size, tolerance=1):pos = page.eval("window.__demo.agentPos()") # read-only state exposed by the democx, cy = pos["cell"]px, py = pos["pixel"]assert abs(px - (cx + 0.5) * cell_size) <= toleranceassert abs(py - (cy + 0.5) * cell_size) <= tolerance
Having the demo expose a read-only state interface (for example window.__demo) lets the test script directly read the agent’s position, training episodes, and convergence status, without having to guess the DOM structure.
6.2 Suggested Metrics Dashboard
- First-pass rate: broken down by task group;
- Average number of repair rounds: the number of feedback iterations needed from first generation to passing;
- Failure-type distribution: how much of the total is rendering, logic, and experience;
- Average response time and token consumption (the cost dimension).
These metrics answer a more important engineering question: for a given task type, is it a “model capability problem” or a “prompt spec problem”? If the pass rate rises significantly after adding structured specs and a self-check step, it means the problem is mainly in the requirement description, not in the model itself.
7. Conclusion and Practical Recommendations
Back to the original question: where are the strengths and weaknesses of LLM-generated interactive demos?
On the strengths side, they can be summarized as:
- Mainstream models can already reliably produce single-file demos that “open and are interactive”;
- The chain of thought usually breaks down the task reasonably, and the convergence ability for algorithm tasks has clearly improved;
- In visualization (such as drawing Q-values on the cells), the designs often exceed expectations.
The weaknesses are equally clear:
- There is a gap between the visual promise and the actual output;
- Physics and geometry commonsense still rely on explicit constraints in the prompt;
- Engineering details like canvas layering and size proportions are high-frequency failure areas;
- The output has randomness, so a single result isn’t enough to represent the true level.
Three practical recommendations for teams:
- Put acceptance criteria in the prompt, not in post-hoc review comments. A structured spec plus a self-check list is often more effective than repeatedly appending “optimize it some more.”
- Build automated regression, and break “does it look good” into “does it run, are the proportions right, did it converge, is the contrast sufficient,” turning subjective experience into quantifiable, traceable metrics.
- Keep a “layered mindset” toward model-generated front-end output: the rendering layer, algorithm layer, and interaction layer can each be verified independently, so when something goes wrong you can quickly locate it rather than rewriting the whole page.
The ceiling of a tool often depends on how precisely the user defines the task. Describing the requirements clearly and writing the acceptance criteria explicitly is the first step to bringing generative code into a production workflow.
