Interactive Visualization Tasks: A New Way to Evaluate LLM Code Generation

    Pass@k and compilation checks cannot show whether generated code runs well or matches real-world intuition. This article explains how interactive visualization tasks fill that gap. It covers three representative task types, common failure modes, a minimal Q-learning reference implementation, a multi-dimensional scoring framework, and principles for designing an evaluation set.

    1. Why Evaluation Needs to Be Closer to Real Use

    Two families of metrics dominate how the industry evaluates code generation in large language models. The first is unit-test pass rate, such as Pass@k. The second is the rate of syntactically valid, compilable code. Both can be automated and reproduced, but both have clear limits:

    • They check whether code is correct, not whether it is usable. Code can pass every assertion and still show layout errors, sluggish interaction, or a black screen at runtime.
    • They rarely capture common-sense correctness. For physics simulation or spatial layout, code that runs does not guarantee that the result matches physical intuition.
    • They ignore process quality. Traditional evaluation looks only at the final output. Whether a model’s chain of thought is internally consistent, and whether its repeated attempts are stable, often matters more for engineering adoption.

    Interactive visualization tasks offer a complementary approach. The model generates runnable web or canvas code in one pass, using Canvas, CSS animations, or Three.js-style 3D rendering. That output is then run and observed directly. Because the results can be seen and used, abstract coding ability becomes visual and interaction quality that people can judge.

    2. Three Representative Task Types

    Using common graphical challenges as examples, three representative task types emerge. Each one tests a different capability.

    Type 1: State-space tasks (e.g., sliding puzzles and Klotski). The core challenge is correctly representing discrete states and their transitions. The model must maintain accurate coordinates and occupancy for each block, then redraw the board after each move. Evaluation focuses on whether all elements align strictly to the grid, whether moves are legal, and whether the end condition is detected correctly.

    Type 2: 3D physics simulation (e.g., bowling and billiards). These tasks test physical common sense. A common failure is unreasonable initial spacing. If the gaps between pins are too wide, a strike cannot happen in one roll, which is impossible in real bowling. Evaluation asks whether the model has a basic sense of size and spacing, and whether its approximations of collision, gravity, and friction are reasonable.

    Type 3: Algorithm visualization (e.g., a Q-learning agent in a maze). This is the hardest type. It requires both algorithmic correctness, meaning the agent converges and the value function is sensible, and visualization correctness, meaning the agent layer aligns with the terrain layer. Many models fail here. The maze may not render, the agent may be so large it fills half the maze, the program may render but not respond to interaction, or training may never converge.

    3. Failure Modes Found in Typical Problems

    Based on many observed cases, failures in visualization tasks fall into several categories. These categories are useful for designing evaluation criteria.

    1. Missing physics and common sense. The bowling gap example shows that the model did not encode real-world constraints into its parameters. The code runs, but its results contradict common sense.

    2. Misalignment between rendering and logic. An agent whose action layer does not align with the terrain layer is a representative case. The logical coordinates are correct, but the rendering uses an inconsistent coordinate system, scale, or origin. As a result, the character floats above the terrain or clips through it. Evaluation must score logical correctness and visual correctness separately.

    3. Chain-of-thought hallucination. The model’s reasoning says it will implement a visual effect, such as an ink-wash style or a wood-grain texture, but the final output shows none of it. A mismatch between stated plans and actual output should be penalized as its own item in visualization tasks.

    4. Usability problems. The output may show a black screen, unreadable content, or no way to start interacting. Informal user reports describe this as the page “crashing” or being “unable to play.” These problems often come from failed initialization, missing event bindings, or a broken render loop. They determine whether the output can be used at all.

    5. Failure to converge. In reinforcement learning visualizations, a training curve that does not trend upward usually points to a flaw in the reward design, learning rate, or state encoding. Convergence separates an implementation that looks correct from one that actually learns.

    4. A Minimal Runnable Q-learning Training Skeleton

    To make evaluation reproducible, the model needs a clear algorithmic reference. The following simplified tabular Q-learning update can be used to check whether a generated implementation is mathematically correct.

    1. import numpy as np
    2. def q_learning_step(q_table, state, action, reward, next_state,
    3. alpha=0.1, gamma=0.95, epsilon=0.1):
    4. """Standard single-step Q-learning update.
    5. alpha: learning rate gamma: discount factor epsilon: exploration rate for the ε-greedy policy
    6. """
    7. # Explore or exploit
    8. if np.random.rand() < epsilon:
    9. next_action = np.random.choice(q_table.shape[1])
    10. else:
    11. next_action = np.argmax(q_table[next_state])
    12. # TD target: immediate reward plus the discounted maximum future value
    13. td_target = reward + gamma * np.max(q_table[next_state])
    14. td_error = td_target - q_table[state, action]
    15. # Update the value of the current state-action pair
    16. q_table[state, action] += alpha * td_error
    17. return next_action

    During evaluation, check three things. First, confirm that the states are discretized correctly. Second, confirm that the TD update formula is implemented correctly. Third, confirm that total reward trends upward over several training episodes. An implementation that appears to train but does not converge usually has a problem with its reward function or state encoding. Displaying Q-values directly on the maze floor is an intuitive and creative presentation, and it can earn credit on visual quality.

    5. Designing a Practical Scoring Framework

    A binary pass or fail judgment cannot describe a model’s capability well. A multi-dimensional score table is a better choice. It splits the total score into several independent dimensions:

    Table 1. Multi-dimensional scoring framework for visualization tasks

    Dimension What to observe Suggested weight
    Runnability Whether the output renders correctly and whether interaction can start High
    Logical correctness Whether state transitions and algorithm updates are correct High
    Common-sense plausibility Whether sizes, spacing, and physics match reality Medium
    Visual presentation Layout, clarity, aesthetics, and fit with the theme Medium
    Chain-of-thought consistency Whether stated reasoning matches the actual output Medium
    Stability Pass rate and variance across repeated attempts Medium

    Stability deserves particular attention. In practice, a model may need several attempts before a task works at all. The first attempt produces an oversized agent. The second produces a maze that never renders. The third renders but cannot be interacted with. The fourth finally aligns but still contains logical errors. This high variance is itself a key signal. It shows that the success rate is too low to support engineering deployment. Evaluation should record the number of attempts, not only whether the final attempt passed.

    6. Designing a High-Quality Evaluation Set

    For evaluation results to be meaningful, the task set itself must be carefully designed:

    1. Graded difficulty. Progress from simple static canvases to interactive animations, then to simulations that combine several algorithms. This step-by-step design shows where a model’s capability ends.
    2. Clear acceptance criteria. Each task needs a pass condition that can be measured or judged by eye, such as “all elements align strictly to an 8×8 grid” or “the reward curve rises monotonically over 200 training episodes.”
    3. A mix of automated and human review. Runnability and logical correctness suit scripted checks. Visual aesthetics, theme fit, and chain-of-thought consistency require human review.
    4. Reproducibility. Fix the inputs and runtime environment, and keep complete logs for every attempt. This avoids misjudging a model that succeeded only once by chance.
    5. Score the algorithm layer and the presentation layer separately. This is the core difference between visualization tasks and traditional evaluation. It shows developers whether a model cannot compute correctly or cannot draw correctly.

    7. Conclusion

    Interactive visualization tasks move code generation evaluation from pass rates on black-box assertions to observable runtime behavior. They expose deep defects such as missing physical common sense, coordinate misalignment, and failure to converge. They also assess visual presentation and chain-of-thought consistency, which are key factors for engineering adoption. Teams that want to evaluate code generation models systematically, and build automated evaluation pipelines on cloud infrastructure such as container platforms, object storage, logging, and monitoring and alerting, will benefit more from a multi-dimensional, reproducible system that separates algorithm and presentation than from optimizing a Pass@k number alone. The purpose of evaluation is to pinpoint a model’s capability gaps, which guides model selection, prompt engineering, and later iterations.