LLM Code Generation Testing: A Practical Evaluation of Visualization and Algorithm Reasoning
Recent LLM releases often lead with eye-catching visual demos, but a demo that runs is not the same as one that is correct. This article presents a hands-on test of a new-generation large model's code generation across four high-discrimination tasks—visual rendering, spatial logic (Klotski), physical common sense (bowling), and algorithm reasoning (Q-learning)—and uses the results to distill a reusable evaluation methodology. It introduces a four-layer scoring framework (runs, usable, reasonable, reproducible), a plan–execution consistency check for catching chain-of-thought hallucinations, and practical guidance such as multiple sampling, domain-constraint checklists, and a reproducible containerized workflow. The goal is not to rank a specific product, but to help teams detect the hidden defects that only surface under deeper scrutiny.
Introduction: Why “It Runs” Isn’t “It’s Correct”
Recent releases from several vendors have introduced a new generation of large models, and their official demos tend to focus on eye-catching visual effects. As a developer with a long-standing interest in real-world engineering, I put a new-generation model’s code generation ability through a hands-on test using a set of high-discrimination tasks. I chose this particular set because each task probes a different capability dimension of the model: visual presentation, spatial logic, physical common sense, and algorithm reasoning.
I want to stress that the goal here is not to rank any specific product. Instead, I hope to distill a reusable evaluation methodology from this exercise. Many teams run into the same confusion: the model’s output “looks fine” and the code runs, yet the reasonableness of the result does not hold up under scrutiny. This article is about how to identify these hidden defects.
Designing the Evaluation Dimensions: From “Output” to “Deep Correctness”
I tend to break down a large model’s code generation ability into four layers, each one deeper than the last:
- Can it run? Syntactically correct, no errors, and it starts in the target environment. This is the lowest bar.
- Is it usable? The interface is interactive, the rendering logic is complete, and the structure is clear.
- Is it reasonable? Does it conform to common sense, physical laws, and business constraints?
- Is it reproducible and maintainable? Are results stable across multiple runs, and is the code extensible?
Most evaluations stop at layers 1 and 2, but layer 3 is where the real gap opens up. The four tasks below are designed around exactly this point.
Task 1: Visual Rendering — Look Past “Looking Good” to “Implementation”
The first task was to generate a dynamic visual page with particle and light/shadow effects. The tested model landed somewhere in the “above passing” range: it delivered particle motion, glow gradients, and layered stacking, with a visual result that was decent enough.
There is a point here that is easy to overlook: a visual that looks good often masks an implementation that is hollow. Many generated results rely on stacking large numbers of fixed parameters and show no real understanding of the rendering pipeline (shaders, blend modes, frame-rate control). When evaluating, I would focus on:
- Whether it uses a sensible rendering loop and releases resources properly, avoiding memory leaks;
- Whether the animation is configurable rather than hardcoded;
- Whether there is a degradation strategy for low-end devices.
This is especially critical if you are generating scenes with WebGL or a mainstream 3D rendering library—whether it correctly handles coordinate spaces, lighting models, and occlusion relationships is what separates “watchable” from “professional.”
Task 2: Spatial Logic — Klotski’s “Correct Placement” and Chain-of-Thought Hallucination
The second task was to implement Klotski (a sliding-block puzzle). The tested model was largely correct in the final layout—the blocks were roughly back in place, which is worth acknowledging. What was genuinely interesting was its behavior in the chain-of-thought: it announced that it would “render an ink-wash style,” yet the final code contained no related implementation at all.
This is a typical case of a disconnect between the chain-of-thought and the execution result—a hallucination at the reasoning level. The model describes things beautifully in the “planning” stage but fails to deliver in the generation stage. For this kind of task, I recommend a “plan–execution consistency check”:
- For each feature promised in the chain-of-thought, can you find a corresponding implementation in the final code?
- Is there a systematic gap between the model’s “self-description” and its “actual output”?
A practical approach is to have the model do a self-review after finishing the implementation, checking its plan item by item. This exposes hallucinations quite effectively.
Task 3: Physical Common Sense — The Bowling “Common Sense Gap”
The third task was to generate a bowling scene (built on a 3D rendering solution). The problem: the spacing between the balls was too large, so a single throw could never clear them all. This looks like a parameter issue, but it is really a lack of common sense and physical constraints.
A bowling pin setup is a standard equilateral-triangle arrangement: the 10 pins are laid out row by row in a 1-2-3-4 progression, with spacing and ball diameter precisely designed to guarantee a chain reaction on impact. The model failed to recognize this constraint when generating the scene and simply “placed the pins.” The correct layout logic can be summarized as:
# Standard 10-pin triangular rack for bowling (illustrative)def bowling_rack(ball_radius, gap_ratio=0.02):rows = 4pins = []spacing = ball_radius * 2 * (1 + gap_ratio) # spacing slightly larger than the ball diameterfor r in range(rows):count = r + 1for i in range(count):x = (i - r / 2) * spacingz = r * spacing * 0.866 # longitudinal spacing for the triangular layoutpins.append((x, z))return pins # total 1+2+3+4 = 10 pins
The fact that the scene came out “pitch black and hard to see” also exposes a lack of default visual-environment handling: no basic lighting, camera framing, or background contrast. When evaluating physics-simulation tasks, I would focus on whether the model has internalized the domain common sense (layout rules, collision chains, energy conservation) rather than merely satisfying “there is a scene.”
Task 4: Algorithm Reasoning — The Maze Q-Learning Test
The most challenging task was to use Q-learning for maze pathfinding. Reinforcement learning spans state space, action space, reward function, exploration vs. exploitation (ε-greedy), and Q-table updates—so it places the highest demands on a model’s grasp of algorithms. In the experiment, the comparison model performed poorly, and I held high expectations for the tested model—but it, too, failed to converge stably within a limited number of attempts, still “crashing” after a few tries.
The core update logic of Q-learning is not complicated, but it is easy to get wrong in the details:
# Core Q-learning update (illustrative)def q_learning_update(q_table, s, a, r, s_next, alpha=0.1, gamma=0.9):best_next = max(q_table[s_next].values())q_table[s][a] += alpha * (r + gamma * best_next - q_table[s][a])return q_table
Common failure points include: poor reward design (e.g., only penalizing without guidance, which makes convergence extremely slow), unreasonable state encoding, a missing exploration-rate decay strategy, and incorrect termination conditions. This shows that the hard part of an algorithm task is not “writing the formula” but “designing an experiment that actually converges.” When evaluating such tasks, you should give multiple attempts and watch the convergence curve rather than judging by a single run—algorithms are inherently stochastic, and one result is not enough to draw a conclusion.
A Few Recommendations on the Evaluation Methodology
Drawing on this hands-on test, I have distilled several actionable evaluation principles:
- Layered scoring: score “runs / usable / reasonable / maintainable” separately so you are not misled by “it runs.”
- Plan–execution consistency check: focus on chain-of-thought hallucinations, which are a common weakness of current models.
- Multiple sampling: especially for algorithmic and stochastic tasks, take at least 3–5 samples to get a distribution rather than judging on a single run.
- Domain-constraint checklist: for each task type, list in advance the common-sense constraints that must be satisfied (e.g., bowling pin layout, collision chains) and verify them one by one.
- A reproducible, engineered workflow: containerize the evaluation tasks, pair them with a unified logging service and monitoring/alerting, and record each run’s inputs, outputs, and convergence metrics so that results are traceable and comparable.
If your evaluation pipeline needs to run at scale, you can use a container platform to manage the evaluation environment uniformly, use a logging service to capture the full trace of each run, and use monitoring/alerting to catch convergence anomalies or timeouts—this is a common combination of capabilities that cloud platforms such as Baidu AI Cloud offer for putting models into engineering production.
Conclusion
The biggest takeaway from this test is this: evaluating a large model’s code generation ability is not about how dazzling the demo looks, but about how it performs on the “hard problems”—common sense, logic, and algorithms. Visuals can be dressed up, but physical laws, spatial logic, and convergence do not lie. For developers and enterprises alike, building a layered, reproducible, consistency-focused evaluation system is far more valuable than chasing the wow factor of a single demo. I hope this evaluation framework gives you a practical reference when selecting and deploying large models.
