Evaluating LLM Creative Coding: Visual, Physical, and Algorithmic Tests

    Creative coding tasks—generative visuals, physics simulations, and algorithm visualizations—are becoming a practical yardstick for LLM reasoning and engineering ability, because they can run, can be observed, and can be judged. This article reviews a comparative evaluation across three dimensions of increasing difficulty. It covers why visual quality should be judged from the final runnable output rather than the model's chain of thought, why physics tasks fail on missing commonsense parameters such as pin spacing and contrast, and why grid-maze Q-learning implementations break down over coordinate systems, reward design, and visualization of convergence. It includes the standard Q-learning update rule plus code examples for grid-to-pixel mapping and reward shaping, and closes with a reusable checklist for evaluating LLMs on creative coding: judge artifacts over descriptions, add plausibility checks, stress-test coordinate alignment, verify convergence and visualization, and cap the retry budget.

    1. Why Creative Coding Has Become a Litmus Test for LLMs

    Creative coding tasks—generative visuals, physics simulation, algorithm visualization—naturally carry three properties: they run, they can be observed, and they can be judged. That combination is making them a practical yardstick for an LLM’s overall reasoning and engineering capability. Unlike Q&A-style benchmarks, these tasks require a model to handle requirement interpretation, architecture design, mathematical modeling, coordinate math, and code generation all at once. Any weak link in that chain gets directly amplified in the final output.

    There is another reason these tasks are hard to fake: you cannot bluff your way through them with wording. A text answer can stay vague, but an interactive program that runs and behaves intuitively has to be logically self-consistent. Creative coding therefore exposes real weaknesses in commonsense constraints, spatial alignment, and logical closure. The sections below walk through three tasks of increasing difficulty.

    2. Visual Output: From “Looks Right” to “Makes Sense”

    The first dimension tests visual expressiveness, with typical tasks such as light-and-particle effects, ink-wash style rendering, and UI texture. Most mainstream models reach the level of surface resemblance here—particles, glows, gradients, and animations all get generated, and the overall look passes. The dividing line is not whether an element is present, but whether it is plausible.

    One trap deserves attention: a disconnect between chain-of-thought and the final artifact. In one generation, the model explicitly stated in its reasoning that it would render an ink-wash texture, yet nothing of the sort appeared on screen. The model has the semantic concept but failed to convert it into executable rendering instructions—essentially a generative hallucination: promised at the language layer, not delivered at the code layer. Visual evaluation must therefore be based on the final runnable output, not on the model’s self-description.

    3. Physical Commonsense: The Hidden Test

    The second dimension is physics simulation, using bowling as a classic scenario. Tasks like this tend to “pass” without passing well: ball motion, collision detection, and 3D rendering all run, but the physics falls apart under closer inspection.

    The most typical problem is missing commonsense. When the spacing between pins is set too wide, a line that would clear every pin in the real world can never clear the rack in the program—the model has the kinematic formulas for collision but has not internalized domain facts such as standard pin array spacing. Similarly, insufficient contrast between objects and a dark background, with no lighting or outlines to separate them, leaves the scene “too dark to read,” which is a recurring readability issue.

    The takeaway: physics evaluation cannot stop at “did it move.” The parameters have to fall within plausible real-world ranges. A dedicated “parameter plausibility” item in the scorecard is worth adding, specifically to check whether gravity, friction coefficients, object dimensions, and spacing are close to reality.

    4. Algorithmic and Spatial Reasoning: Why Q-Learning Tends to Fail

    The third dimension is the hardest, using Q-learning in a grid maze as the task. It stresses three things at once: correctness of the algorithm implementation, spatial alignment of the coordinate system, and visualization of training convergence. Most models fail here, and the causes are highly concentrated.

    The first is loss of control over layout. Typical failures include an agent drawn so large that it physically blocks nearly half of the maze, leaving no viable path; maze walls that are never rendered at all; or an agent layer whose coordinates are not aligned with the terrain layer, so that the agent visually walks “through walls” or “floats in the air” even though the logic reached the correct cell. The root cause is a failure to separate two coordinate systems—logical grid coordinates and rendering coordinates—without a unified grid-to-pixel mapping.

    The second is how implementations handle alignment and convergence. Stronger implementations ground abstract information in the scene: rendering each cell’s Q value directly on the corresponding floor tile makes the learning process visible and verifiable, while the training curve converges as expected. Many models cannot do this—either the Q values never update, or a poorly designed reward function causes oscillation and divergence.

    The core Q-learning update rule itself is not complicated; the standard form is:

    1. import random
    2. def q_learning_step(q_table, state, action, reward, next_state, alpha=0.1, gamma=0.9):
    3. # Current state-action value
    4. old_q = q_table[state][action]
    5. # Best value of the next state
    6. next_max = max(q_table[next_state].values())
    7. # Bellman update
    8. q_table[state][action] = old_q + alpha * (reward + gamma * next_max - old_q)
    9. return q_table
    10. def choose_action(q_table, state, epsilon=0.1):
    11. if random.random() < epsilon: # ε-greedy exploration
    12. return random.choice(list(q_table[state].keys()))
    13. return max(q_table[state], key=q_table[state].get) # Exploit the learned best action

    What separates “runs” from “runs correctly” is exactly coordinate alignment and reward design:

    1. # Unified grid -> screen pixel mapping to avoid misalignment between the agent layer and the terrain layer
    2. CELL_SIZE = 40
    3. def grid_to_pixel(row, col):
    4. return col * CELL_SIZE, row * CELL_SIZE
    5. # Reward design: strong positive reward on reaching the goal, negative reward for hitting a wall,
    6. # and a small step penalty otherwise to encourage shorter paths
    7. def reward_fn(next_row, next_col, goal, wall_set):
    8. if (next_row, next_col) == goal:
    9. return 10.0
    10. if (next_row, next_col) in wall_set:
    11. return -1.0
    12. return -0.05 # Encourage shorter paths

    Give this kind of task enough retries and then evaluate, and models repeatedly fail in the same way: right intent, wrong execution. That is the real waterline of current spatial reasoning ability.

    5. Evaluation Methodology: A Reusable Checklist for Developers

    Across all three dimensions, the following practical recommendations help you assess LLMs on creative coding more rationally:

    1. Judge the artifact, not the description. However polished the chain of thought, only the final runnable, observable output counts; stay alert for generative hallucinations.
    2. Add a commonsense check for physics tasks. Keep a separate parameter plausibility score covering dimensions, spacing, gravity, and similar physical constants.
    3. Stress-test coordinate alignment in spatial tasks. Check specifically whether logical coordinates and rendering coordinates are unified, and whether agents, terrain, and numeric labels land in the right positions.
    4. For algorithm tasks, watch convergence and visualization. Look beyond whether training converges to whether abstract information such as value functions is visualized effectively.
    5. Keep a retry budget, but cap it. A moderate number of retries distinguishes genuine capability gaps from one-off failures, while excessive retries mask real defects.

    6. Conclusion

    From visuals to physics to algorithms, LLMs can now cover most surface-level “looks right” requirements in creative coding, but they still show clear and reproducible weaknesses in deeper capabilities such as commonsense constraints, spatial alignment, and logical self-consistency. For developers, rather than chasing a single impressive demo, the better path is to build a multi-dimensional evaluation framework grounded in runnable artifacts—that is what pinpoints a model’s boundaries and provides a reliable basis for model selection and engineering deployment. Combined with supporting capabilities such as the container platform, log service, and monitoring and alerting offered by Baidu AI Cloud, this kind of evaluation can also be automated and run continuously, keeping track of how model capabilities evolve over time.