Interactive Task Evaluation for LLM Code Generation: Four Levels of Difficulty
Interactive task evaluation is a stronger signal for testing an LLM's code generation ability than static accuracy scores or standardized benchmarks, which leave room for memorization and template reuse. By asking a model to generate interactive, visualizable programs that run in a browser environment, evaluators can expose real weaknesses in structured output, state management, geometric alignment, and commonsense reasoning at once. This article breaks the evaluation into four levels—visual presentation and intent-to-output consistency, physics commonsense and constraint satisfaction, algorithm implementation, and interaction completeness—summarizes common failure modes, and recommends a multi-attempt, structured-scoring method that turns subjective impressions into quantifiable observations.
Introduction: Why interactive tasks are a good yardstick for code ability
When evaluating large language models, static problem accuracy and standardized scores often leave room to “game the leaderboard”: a model can memorize answers and reuse templates without actually producing code that runs and holds together logically. Compared with multiple-choice-style evaluation, asking a model to generate an interactive visualization in a browser environment—for example, a slider puzzle, a small 3D physics scene, or a reinforcement learning demo—can expose its real shortcomings across structured output, state management, geometric alignment, and commonsense reasoning in a single pass.
The value of these tasks is that “results are visible and correctness is clear-cut”: whether the page renders, whether the animation is smooth, whether the rules make sense, and whether the algorithm converges are all obvious at a glance and hard to fake. Below, we use several typical interactive scenarios to break down the dimensions worth focusing on during evaluation.
Level 1: Visual presentation and “intent-to-output” consistency
The most direct checkpoint is the rendering result. When a model generates a program with lighting, particles, gradients, and other atmospheric effects, it can reproduce the visual quality common in modern web experiences, and mainstream models generally handle this part well.
What deserves real scrutiny, however, is the problem of inconsistency between the reasoning chain and the final output. A typical symptom: the model clearly states in its reasoning that it will present an “ink-wash diffusion” effect, yet the generated code contains no related implementation at all—a classic case of reasoning-chain hallucination, where the model “says” it will do something but omits or fabricates the corresponding logic when it comes time to implement it.
For developers, this suggests that evaluation cannot rely only on whether the reasoning text sounds confident; it must be checked against the actual code and the running screen. Recommended practices:
- Extract the key intent points the model states about itself (such as “ink wash” or “wood grain”), and search the code one by one to confirm whether there is a corresponding implementation (gradients, noise textures, blend modes, and so on);
- Do a manual or screenshot-based comparison at the rendering layer to confirm whether the visual effect matches the description;
- Make the “intent fulfillment rate” a separate scoring item, specifically to capture this kind of self-contradiction.
Level 2: Physics commonsense and constraint satisfaction
The second frequent point of failure is missing physics commonsense. Take a 3D scene where a ball strikes bowling pins: a typical problem that shows up in evaluation is that the initial spacing between the pins is too large, making it impossible to knock down all the targets at once—fundamentally an error in geometric and physical constraints.
The model’s difficulty is that it is good at calling rendering frameworks to produce code that “looks like it moves,” but it may not understand the implicit constraints of the game rules themselves (the coupling between collider sizes, initial layout, and scoring logic). When evaluating such tasks, “commonsense correctness” should be listed separately:
- Whether the initial layout satisfies the rules (spacing, symmetry, reachability);
- Whether the physics parameters are reasonable (gravity, friction, collision response);
- Whether the interaction loop holds together (action → collision → judgment → feedback).
In addition, problems like a screen that is “so dark you can’t see anything” also belong to presentation-layer defects, and can be avoided by setting baseline requirements for contrast, default visibility, and so on.
Level 3: The real difficulty of algorithm implementation
If visual presentation and physics are the “surface,” then algorithm implementation is the “deep end” that most clearly separates models. Take a grid maze plus tabular Q-learning: it simultaneously tests three things—data structures (maze generation and representation), algorithm logic (state-action-reward, Q-value updates), and visualization overlay (mapping Q-values onto cells).
The failure chain that appears in evaluation is highly representative:
- The agent is far larger than a cell, occupying half the maze (out-of-control proportions);
- The maze structure is not rendered correctly at all;
- The maze is drawn but cannot run (interaction breaks);
- The agent’s action layer and the terrain layer are not aligned (inconsistent coordinate systems / grid indices).
The last point is especially critical: a layer alignment error causes the agent’s position to be offset from the actual cells, so even if the algorithm is correct, it visually appears as “glitchy, jittery movement.”
One commendable practice is to encode Q-values directly into the floor cell colors so the training process is visible; more importantly, whether training converges—most model-generated reinforcement learning demos end up with diverging or oscillating Q-values, while a few genuinely converge, which precisely reflects different levels of understanding of the algorithm’s essence. Convergence can be automatically checked with a simple quantitative criterion:
# Example: determine whether Q-learning converges after trainingdef is_converged(rewards, window=50, tolerance=0.02):if len(rewards) < window:return Falserecent = rewards[-window:]# Treat as converged if reward variation within the recent window is small enoughreturn (max(recent) - min(recent)) < tolerance
Level 4: Interaction completeness and playability
The final level is “does it actually work.” Even if a program renders and the algorithm is correct, if it can’t be clicked, doesn’t respond, or has misaligned coordinate systems, it is still an unqualified product. Playability / interactivity should be treated as a hard gate: consistent coordinate systems across all layers, effective event binding, and a closed-loop state flow. Any single break will make a demo that “looks complete” collapse instantly.
Summary of common failure modes
Combining the scenarios above, the typical failures of large models in interactive code generation can be grouped into five categories:
- Reasoning-chain hallucination: an effect that is claimed to be implemented does not exist in the code;
- Missing physics/commonsense: the initial layout or parameters violate the game rules;
- Spatial alignment errors: inconsistent coordinate systems or grid indices across layers;
- Non-convergent algorithms: the training process diverges, losing the point of the demo;
- Rendering or interaction breakdown: black screen, unclickable, or non-runnable.
Recommended evaluation method: multi-attempt retries plus structured scoring
A single failed generation cannot fully negate a model’s ability, because interactive tasks are inherently complex and subject to chance. A more robust approach is to allow a limited number of retries and record each attempt in a structured way, rather than drawing conclusions from a single impression:
# Example: multi-attempt evaluation with structured result loggingdef evaluate_interactive_task(model, prompt, max_attempts=4):history = []for attempt in range(1, max_attempts + 1):output = model.generate(prompt)record = {"attempt": attempt,"renders": check_renders(output), # Whether it renders correctly"playable": check_playable(output), # Whether interactions work"aligned": check_alignment(output), # Whether layer coordinate systems align"converges": check_convergence(output), # Whether the algorithm converges"intent_match": check_intent(output), # Whether intent matches the output}history.append(record)if all(record.values()):breakreturn history
With this logging in place, evaluation can be upgraded from “I feel it’s good or not” to quantifiable observation: was the first attempt out of proportion, did the second fail to render, was the third un-runnable, was the fourth misaligned—the trajectory of failure modes is itself an important signal for measuring model stability. If you need to run evaluations at scale, you can deploy this on a container platform for batch execution and use logging services and monitoring/alerting to track the success rate and failure distribution of each generation.
Conclusion
Using interactive, visualizable tasks to test an LLM’s code generation ability is closer to real development scenarios than pure score comparison. The core is not “who generates more flashy effects,” but whether the model can keep intent and output consistent, obey physics and rule commonsense, correctly implement algorithms, and guarantee complete interactivity. By splitting visual presentation, physics constraints, algorithm correctness, and interaction completeness into four independent levels, and supplementing them with multi-attempt retries and structured scoring, developers can form a solid, reproducible judgment on whether an LLM can “actually write code.”
