LLM Code Generation Evaluation: Visual Quality, Physical Plausibility, and Algorithm Convergence
This article lays out a dimension-based framework for evaluating how well large language models generate code, going beyond the simple question of whether the output runs. It covers interactive visualization demos, physics and common-sense modeling, agent-based iterative generation such as maze navigation, and reinforcement learning code such as Q-learning. It groups the typical failure modes into four categories — semantic hallucination, missing common sense, layer misalignment, and algorithm non-convergence — and closes with a scoring rubric and practical engineering recommendations for verifying generated code in production environments.
1. Why Large Models Need Systematic Code Generation Evaluation
Large models have moved well past “generate a snippet of runnable code” toward “generate a complete, interactive, verifiable program.” But running is only the lowest bar. What actually separates models is how complete the output is across dimensions like visual quality, physical common sense, logical layering, and algorithm convergence.
Take a typical task: “generate an interactive visualization demo.” It naturally combines several capabilities at once — graphics rendering, geometric layout, physical behavior, interaction logic, and even algorithms such as reinforcement learning that only converge after multiple iterations. Tasks like this are a genuine testbed for assessing model code capability. In practice, many users hit the same pattern of high expectations followed by disappointing results: a single generated attempt looks impressive at first glance, but falls apart in the details and reveals a shallow understanding. A dimension-by-dimension evaluation method is therefore more useful than simply asking “does it run?”
2. Interactive Visualization Generation: Scoring the Visual Layer
The first impression of a visualization demo comes from its visuals — particle systems, layered lighting, smooth transitions. When evaluating, separate two levels:
- Surface appearance: does it include common effects such as particles and lighting, and does it render smoothly without flicker?
- Underlying intent: does the result actually deliver what the prompt promised?
A typical problem is “visual hallucination”: in its reasoning trace or explanation, the model claims it will render a specific style (for example, a particular traditional aesthetic), but the generated scene has nothing to do with that description. This kind of “says one thing, builds another” stems from a gap between the model’s semantic understanding of the target visuals and its ability to realize them in code — a textbook hallucination. Instead of judging only whether the output looks flashy, score description-to-implementation consistency as its own metric.
3. Physics and Common-Sense Modeling: Checking Whether Generated Code Is Reasonable
When a task involves physics simulation, the plausibility of the code says more about model quality than its runnability. Consider a billiards or bowling scene. The point is not whether the model can call a 3D rendering library to place the objects, but whether the result respects basic physical common sense:
- Is the spacing between pins wide enough for a moving object to collide and knock them down as intended?
- Do collision detection and force transfer follow momentum and geometric relationships?
- Are there gaps so large that the targets cannot all be knocked down in a single shot — a layout that violates common sense?
The root cause is usually that the model “placed the objects” without establishing correct spatial scale or physical relationships. For physics tasks, define quantifiable plausibility criteria — number of objects knocked over after impact, target reachability — instead of a subjective “does it look right.”
4. Agent-Based Iterative Generation: Layer Alignment and Playability
Harder tasks ask the model to generate a program in which an agent acts within an environment, such as maze navigation. These raise the bar for layer alignment and playability, and they are where multi-round iterative generation tends to break down.
Suppose the model is given several chances to revise. Typical failure modes appear in sequence:
- Scale imbalance: the agent is drawn too large and takes up a significant share of the scene, destroying the overall layout;
- Missing rendering: the environment is not drawn correctly at all;
- Playability collapse: the visuals are there, but the interaction logic does not work and the demo becomes unusable;
- Layer misalignment: the agent’s action layer and the terrain layer do not line up, so movement and visuals disagree.
These four problems escalate one by one, mapping onto four dimensions of the generated program: layout reasonableness, rendering completeness, interaction usability, and layer consistency. Layer misalignment is the most insidious, because it does not show up in a static screenshot — it only surfaces at runtime. Evaluation should emphasize judging after the program runs, and should check whether coordinates and scales are consistent across logical layers.
5. Reinforcement Learning Code Generation: Convergence Is the Hard Metric
Across all these scenarios, reinforcement learning (for example, training an agent with Q-learning to escape a maze) puts code correctness to the hardest test. You cannot judge quality by glancing at the screen; the hard criterion is whether training converges.
Key evaluation points:
- Reward function design: do the rewards and penalties steer the agent toward learning the right behavior?
- State and action space: are the state encoding and action set reasonable, and do they cover the necessary information?
- Training curves: do loss or return stabilize over iterations rather than oscillate or diverge?
It is worth noting that many models produce Q-learning code that looks complete but never converges during training — a sign that their grasp of the algorithmic details is shallow. Conversely, if a model not only makes training converge but also visualizes the abstract Q-values inside the environment (for example, annotating traversable cells with colors or numbers), it demonstrates a deeper understanding of the algorithm’s internal state and the ability to express it. “Does it converge?” should therefore be the core passing criterion for reinforcement learning tasks.
6. Failure Modes: A Unified View of Four Problem Classes
Pulling these scenarios together, typical failures in model-generated code fall into four categories:
- Semantic hallucination: the promised effect does not match the final implementation (for example, claiming a particular aesthetic and delivering none of it).
- Missing common sense: physical layout and scale relationships defy intuition (for example, collision targets that cannot be reached).
- Layer misalignment: coordinates, scales, or timing are inconsistent across logical layers — correct in a static view, broken in motion.
- Algorithm non-convergence: algorithm code that looks complete never reaches its intended training goal.
These map to “says one thing but does another,” “does not match reality,” “internally inconsistent,” and “goal not achieved” — together forming an analysis framework for code generation quality.
7. Evaluation Dimensions and Engineering Practice
To make evaluation more objective and reusable, build a scoring rubric around these dimensions:
| Dimension | Focus | Passing criterion |
|---|---|---|
| Runnability | Runs without errors | Runs as-is or after minor fixes |
| Visual consistency | Does the result match the description | No significant hallucination; effect matches expectations |
| Physical plausibility | Layout and collisions follow common sense | Key targets reachable; no common-sense-violating layout |
| Layer consistency | Logical layers align | No misalignment at runtime |
| Algorithm convergence | Does training converge | Loss/return converges stably |
A few recommendations for putting this into practice:
- Run first, judge second: many defects only appear at runtime; static code review is not enough to reach a verdict.
- Give objective metrics, not subjective descriptions: replace “does it look good” with quantifiable criteria such as objects knocked over, convergence curves, and reachability.
- Watch for quality decay across iterations: more chances does not mean better results — observe whether each revision moves in the right direction.
- Separate “runs” from “runs correctly”: the first is the passing line; the second reflects true capability.
Conclusion
Evaluating large models’ code generation is shifting from “can it produce runnable code” to “can it produce reasonable, consistent, convergent complete programs.” Rather than obsessing over whether a single attempt is impressive, build a dimension-based evaluation framework and treat common defects — visual hallucination, missing common sense, layer misalignment, and algorithm non-convergence — with clear-eyed rigor. For teams introducing AI-assisted programming into production, using general cloud capabilities such as mature container platforms, logging services, and monitoring and alerting to automatically verify generated results and run regression tests is an effective way to improve engineering reliability. Only by standardizing and quantifying evaluation can teams set expectations that are truly trustworthy during tool selection and rollout.
