Testing LLM-Generated 3D Scenes: The Gap Between Visuals and Implementation

Why a Single-File 3D Scene Is a Better Test

As LLM code generation becomes more common, asking a model to “generate a frontend page” no longer separates capabilities effectively. Static pages have limited task complexity, and most leading models can produce runnable HTML+CSS+JS in one pass. By contrast, generating an offline single-file 3D scene is a much more discriminating benchmark because it stresses four capabilities at once:

1. Knowledge completeness: The task requires a single HTML file using native WebGL2 only, with no external images, model files, or third-party libraries. The model must write shaders, matrix transforms, camera controls, and other graphics code from memory. Any gap in that knowledge leads directly to a black screen or runtime error.
2. Graphics engineering breadth: A convincing scene involves geometric modeling, lighting, materials, animation, water rendering, and other subsystems. This tests how the model makes architectural trade-offs within a limited amount of code.
3. Long-range constraint adherence: For example, “when the tide rises and falls, the boats, waterline, and plaza flooding state must respond together.” Cross-module dynamic constraints like this are a useful probe for long-range programming ability.
4. Aesthetics and composition: How the initial view organizes the main subject and supporting elements reflects the “taste” aligned during post-training.

One additional background point matters: while the industry widely introduces lightweight, fast, low-cost model tiers, the frontier models are more worth observing. The marginal benefit of distilling from strong models is diminishing, and the progress of newer generations increasingly comes from choices made in post-training and reinforcement learning. Comparing two generations of the same model family can reveal the structure of capability improvement, not just score changes.

Test Design and Three Scoring Dimensions

To make the comparison reproducible, strict variable control was applied:

• Task: Implement a scene with a giant circular water gate, a terraced seaside city, and adjustable tide levels in a single offline HTML file.
• Two hard constraints in the prompt: First, establish the composition, architecture, and materials before adding dynamics. Second, the rising and falling tide cannot be simulated by merely moving a blue plane up and down; boats, the waterline, and plaza flooding states must all respond.
• Controlled variables: The old and new versions were generated within the same time window using the same deep-thinking mode. This matters because online model behavior can fluctuate across versions and serving strategy changes, and comparisons across time windows introduce noise.

Scoring covers three dimensions, corresponding to what can be seen, what requires reading the code, and what must be numerically verified:

Dimension	What It Examines	Verification Method
Visual layer	Initial composition, distribution of primary and secondary elements, material recognizability	Direct observation
Mechanism layer	Whether effects such as water reflections are approximations or correct implementations	Source code review
Consistency layer	Whether quantified prompt constraints are actually satisfied	Parameter calculation

Dimension 1: Composition and First Impression

The most obvious difference between the two outputs appears in how the scene is organized under the default camera.

The previous generation did not suffer from a small main subject—the giant ring was massive enough—but the mountains on both sides occupied too much of the frame. A large portion of the initial view was filled by nearby slopes and distant peaks. The city was squeezed deep inside the bay, and the building clusters and plaza were hard to identify. The result was that the giant ring stood at the center while the seaside city required effort to make out. The relationship between the main subject and supporting elements was unbalanced.

The new generation organized the giant ring, bridge arch, beacon tower, and terraced housing into a coherent landscape. Buildings spread along both shores, the central plaza had clearly defined concentric terraces, and the warm-cool contrast between oxidized copper and stone was easier to distinguish. It came closer to the “complete world” requested in the prompt and moved beyond the stiff feeling of placing a single ring object on terrain.

This reflects two levels of understanding. Treating a scene as a stack of independent objects is object-level understanding. Organizing primary and secondary relationships, visual guidance, and color rhythm is scene-level understanding. The latter is a direct result of post-training alignment on aesthetic signals, which explains why first impressions are often where the new generation shows the most visible progress.

A practical suggestion for developers: explicitly state frame proportions in the prompt—for example, “the main structure should occupy about 40% of the frame, the midground city 30%, and distant mountains no more than 20%.” Such quantified descriptions are easier for models to execute than vague statements like “the composition should have depth.”

Dimension 2: Water Reflection Mechanism

Beneath the visuals, the real engineering gap lies in how water reflections are implemented.

Approximation approach (previous generation): Reflections were simulated analytically by blending a sky gradient, sun highlight, and light streaks. The advantage is low computational cost and simple code. The drawback is that building reflections have no real geometric source, so the reflection and the object do not correspond spatially, and the flaw becomes visible on closer inspection.

Correct approach (new generation): An additional mirror camera is introduced. The entire scene, including the city and boats, is redrawn from a mirrored viewpoint and then sampled onto the water surface. The core flow can be summarized as follows:

// Core flow of mirror reflection (illustrative)
const mirrorCam = mainCamera.clone();
mirrorCam.reflect(waterPlane);          // 1. Mirror the camera across the water plane
renderer.setRenderTarget(reflectFBO);   // 2. Render the scene offscreen to a texture
renderer.render(scene, mirrorCam);
renderer.setRenderTarget(null);
// 3. During the main pass, water fragments sample this texture
//    using screen coordinates, then add normal-based distortion
//    to produce a realistic wavering reflection

This is an upgrade from “looks like a reflection” to “geometrically correct reflection,” a shift at the implementation mechanism level. For developers, the takeaway is that when reviewing generated code, this kind of mechanism choice should take priority over a screenshot impression. If the mechanism is correct, later iterations have a reliable foundation. If the mechanism is an approximation, requirement changes will likely force a rebuild.

Dimension 3: Parameter Consistency—the Most Overlooked Regression

At the highest tide setting, the two versions reversed their performance.

The prompt required that “at high tide, most of the stone steps are submerged.” The verification method is to extract the step height, total step count, and the world coordinate corresponding to the highest tide from the generated source code, then perform a simple division:

// Parameter calculation (illustrative values)
const stepH = 0.6, steps = 9;   // Each step is 0.6 high; 9 concentric steps total
const maxTide = 2.8;            // Highest tide level (world Y coordinate)
// Submerged steps at high tide ≈ floor(maxTide / stepH) = 4
// That means only 4 out of 9 steps are below average water level.
// If the requirement is "most (≥2/3) submerged,"
// then maxTide ≥ 6 * stepH is needed to satisfy the constraint.

The calculation result is counterintuitive: the visually superior new version had only the lowest four of nine plaza steps below the average water level at high tide, failing to satisfy the “most steps submerged” requirement. The previous generation, despite its weaker initial composition, covered the outermost steps with the highest water range.

Why would better visuals come with consistency regression? Three plausible explanations:

1. Reward signal bias: Human evaluation feedback in post-training naturally favors overall visual impression. Reviewers first see composition and atmosphere, while parameter-level constraints carry lower weight in the reward.
2. Generation budget competition: The model allocated more computational budget to high-complexity modules such as mirror reflections and layered materials, while numeric validation of the dynamic system was marginalized.
3. Insufficient attention allocation: When constraints are buried at the end of a long sentence, model adherence drops noticeably.

Capability Layering and a Three-Step Acceptance Workflow

Across the three dimensions, a layered model of current LLM code generation capability emerges:

• Layer 1: Global organization and aesthetics—improving fastest, with increasingly strong first impressions.
• Layer 2: Local mechanism selection—such as the correctness of the rendering approach. Still improving, with the direction trending positive.
• Layer 3: Cross-module numeric consistency—most prone to regression, and the strongest test of “backend-level” engineering depth.

This suggests neither concluding from a stunning screenshot nor dismissing overall progress because of one parameter flaw. A more reliable approach is to establish a three-step acceptance workflow:

1. Visual review: initial composition, hierarchy, and material recognizability.
2. Code review: whether reflections, shadows, animations, and similar effects are mechanism implementations or visual approximations.
3. Parameter verification: translate every quantified prompt constraint into computable assertions and verify them at the source code level.

The corresponding evaluation checklist:

• Single file, zero external dependencies, runs fully offline.
• Main subject occupies a reasonable portion of the initial frame; primary and secondary relationships hold.
• Dynamic effects are mechanism implementations, not textures or approximations.
• Every quantified prompt constraint is satisfied and can be written as an assertion script.
• The scene remains coherent under extreme parameters, such as highest and lowest tide levels.

Prompt Design and Iteration Suggestions for Developers

Based on these findings, four practical suggestions can be applied directly:

1. Front-load and structure constraints: Write “the tide must affect boats, waterline, and submerged state together” as numbered items rather than clauses inside a long sentence. Models show significantly higher adherence to explicit entries.
2. Provide verifiable quantified goals: For example, “at high tide, at least two-thirds of the steps must be below the water level.” This allows automated assertion checks during acceptance instead of subjective judgment.
3. Generate in stages: First have the model complete composition and static materials. After confirming satisfaction, add the dynamic system. This avoids multiple objectives competing for quality budget within a single generation.
4. Treat source code as the final acceptance target: Visuals can deceive; parameters do not. Reading constant tables, height tables, and water level definitions in the code often exposes problems more effectively than screenshots.

It is worth noting that this evaluation approach is not tied to a specific platform. On mainstream LLM service platforms, developers can use APIs to batch-call different model versions with the same prompt and turn the visual-mechanism-parameter workflow described above into an automated evaluation script. This makes version comparison a quantifiable, traceable engineering practice rather than a subjective impression.

Conclusion

The core message of this test can be summarized in one sentence: the ability to shape a strong first impression is improving rapidly, but what determines production readiness lies in the parts you cannot see—mechanism correctness, parameter consistency, and coherence under extreme conditions. For practitioners, the evaluation method itself must evolve accordingly: from viewing screenshots to reading code to calculating parameters. Visual quality determines whether you are willing to use a model for a demo; parameter consistency determines whether it can enter your production pipeline. The real arena for the next stage of model competition lies in these backend details.