Testing LLM 3D Code Generation: Strong First Impressions, Weaker Backend Logic

Why Use a “Single-File 3D Scene” to Test LLMs

Frontend 3D scene generation is one of the few coding tasks that can assess multiple capabilities at once. It requires spatial composition and aesthetic judgment, graphics knowledge such as cameras, lighting, and reflections, animation and state management skills, and the engineering discipline to maintain long-range code consistency within a single file.

For this test, the task was constrained to an “offline single HTML file”: no external images or model assets, no third-party libraries, and everything had to be implemented from scratch with raw WebGL2. This constraint matters for three reasons:

1. It removes the “glue code” illusion. If a mature 3D library is allowed, the model can lean on the library’s default behavior. Writing from scratch exposes the model’s actual ability.
2. It tests long-range consistency. A convincing scene easily reaches over a thousand lines of code. Coordinate systems, units, naming conventions, and geometry constants must remain coherent throughout. Any inconsistency surfaces at runtime.
3. The result is verifiable. Visual output can be compared through screenshots, and geometry and water-level parameters can be extracted from the source code for numerical checks. This makes the task naturally suitable for building a reproducible evaluation baseline.

To control for variables, the comparison was run on the same client and at the same inference tier. The test subjects were two successive generations of models on the same technical lineage, referred to below as “the previous generation” and “the new generation.” This comparison is meaningful because, while much of the industry races toward lightweight small models, the generational improvement of frontier models better reflects how the ceiling of foundational capability is actually moving.

Test Task Design: Making the Tide a Global State

The task required generating three things: a giant ring-shaped tidal gate, a terraced seaside city, and a tide that rises and falls. The prompt included only two hard requirements:

• Establish composition, architecture, and materials first.
• The rising and falling tide must not be limited to moving a blue plane up and down. Boats, waterlines, and plaza submersion must all change together.

The second requirement is the heart of the test. It does not measure “whether the model can animate.” It measures state propagation. The tide level is a global state. Boat buoyancy and rocking, the waterline position, and the number of submerged plaza steps must all derive from that same state, rather than each running its own isolated tween animation. This kind of coupled constraint appears constantly in real-world engineering: one state change must synchronously refresh multiple views. It is therefore a precise probe of the model’s engineering maturity.

Dimension 1: First-Screen Composition — Where First Impressions Come From

Both generations delivered a single-file WebGL2 scene with no external dependencies, but the scene organization under the default camera revealed the first meaningful gap.

The previous generation’s issue: the giant ring gate did sit at the center of the frame, but the mountains on both sides occupied more screen space. Near slopes and distant peaks filled the first screen, while the city huddled deep inside the bay. The building clusters and central plaza had weak visual presence. In essence, it was “a ring-shaped object placed on terrain” rather than “a composed world.”

The new generation’s approach: the ring, bridge openings, beacon towers, and terraced housing were organized into a coherent visual sequence. Building clusters spread along both shores, the central plaza showed clear concentric step layers, and the warm-cool contrast between copper oxide and stone materials was easier to distinguish. The scene no longer felt like disconnected objects placed on a landscape.

First impressions come down to hierarchy management: the position and orientation of the default camera, the choice of visual focus, and the density distribution of elements. This can be operationalized into three checkable criteria:

1. The 5-second recognition test: show the default-view screenshot to a colleague who has not read the prompt for five seconds. Can they identify the protagonist of the scene?
2. Element recognizability: can every named subject in the prompt — the tidal gate, terraced plaza, beacon towers, and boats — be identified on the first screen?
3. Prompt fulfillment: does the composition serve the theme, or is it just filling the frame with terrain?

Dimension 2: Reflection Implementation — From Surface Effects to Real Mechanisms

The water surface is the visual anchor of this type of scene, and the two generations showed a substantive difference in how they implemented reflections.

The previous generation used an analytical approach: procedural sky, sun specular highlights, and light streaks to “simulate” reflections:

// Analytical water surface: sky color and specular highlight approximate reflections.
// The reflection has no true spatial source.
vec3 skyColor = mix(horizonColor, zenithColor, pow(max(viewDir.y, 0.0), 0.6));
float sunGlint = pow(max(dot(reflect(-viewDir, waterNormal), sunDir), 0.0), 256.0);
vec3 water = mix(skyColor, deepWaterColor, 0.45) + sunGlint * sunColor;

The new generation additionally introduced a mirror camera: it renders the scene flipped across the water plane into an offscreen buffer, then samples that texture onto the water surface. Shoreline buildings therefore have spatially correct reflections. Conceptually:

// Planar reflection (illustrative): a mirror camera renders the scene again
// into an offscreen framebuffer.
const mirrorCam = cloneCamera(mainCam);
mirrorCam.position.y = 2 * WATER_LEVEL - mainCam.position.y;
mirrorCam.forward = mirrorAcrossPlane(mainCam.forward, WATER_PLANE);
renderScene(mirrorCam, reflectionFBO); // second render pass
// In the main pass, sample the reflection texture by world coordinates
// and blend with the water base color using a Fresnel term.

Why does the mechanism matter more than the visual output?

• Transferability: an analytical highlight breaks down when the scene layout changes. Planar reflection is a general mechanism that follows any shoreline geometry.
• Engineering signal: proactively introducing a second render pass and an offscreen framebuffer shows that the model understands the rendering pipeline rather than simply patching visuals.
• Maintainability: in a mechanism-based implementation, the water level and mirror plane are parameterized, making later adjustments less fragile.

A corresponding verification method: place a building with a distinctive silhouette near the shore and observe whether its reflection matches the object point by point. Then inspect the source code for the presence of an offscreen framebuffer and a second pass.

Dimension 3: Parameter Consistency — The Real “Backend” Test

What best illustrates that first impressions are not the same as internal rigor is the high-tide parameter. The prompt explicitly required that “most of the stone steps be submerged at high tide.” Acceptance cannot rely on visual inspection. The correct approach is to read the source code, extract the step height and maximum water level constants, and verify them numerically.

The result was somewhat ironic: the new generation, which produced the better first screen, had a nine-level plaza where only the lowest four steps sat below the average water level at high tide. It failed to deliver “most steps submerged.” The previous generation, despite its weaker first screen, extended the maximum water level to cover the outermost steps.

The lesson is direct: visual polish and parameter rigor are two independent capabilities. The former is the easiest way to impress a reviewer — and the easiest way to hide a defect. For practical evaluation, automated checks like this can help:

// Evaluation side: extract key constants from the generated artifact
// and run numerical assertions.
const STEP_HEIGHT = 1.5;  // height of each step
const STEP_COUNT  = 9;    // total number of steps
const MAX_TIDE    = 6.0;  // maximum tide level relative to the plaza base
const submerged = Math.min(STEP_COUNT, Math.floor(MAX_TIDE / STEP_HEIGHT));
console.assert(
  submerged > STEP_COUNT / 2,
  `At high tide, more than half of the steps should be submerged. Actual: ${submerged}`
);

A more robust practice is to require the model to output all geometry constants in a central configuration object so that scripts can extract them easily, and to embed console.assert self-checks in the generated artifact. This turns “the math is correct” into a machine-verifiable assertion instead of a subjective impression.

Turning a One-Off Test into a Reusable Evaluation Framework

The value of a single test is limited. What matters is converting the methodology into a regression evaluation asset for the team:

1. Fixed prompt set: codify this task and several similar ones — constraint-satisfaction, state-coupling, and long-range consistency types — into a reusable test suite.
2. Three-axis scoring card: first-screen composition (recognizability, prompt fulfillment), implementation mechanism (whether a general mechanism is used instead of a local patch), and parameter consistency (pass rate of hard-constraint assertions).
3. Automated acceptance: screenshot baseline comparison plus constant-extraction assertions, fully scripted to reduce noise from subjective scoring.
4. Controlled variables and record keeping: same client, same inference tier; record the model version and evaluation timestamp. The community often jokes that “older models get worse after a new model launches.” Whether or not that can be proven, it is a reminder that evaluation results are time-sensitive and baselines should be re-run periodically.
5. Standardized environment: containerize the evaluation pipeline so that regression tests can run continuously as models iterate rather than depending on a single local machine. Cloud container platforms and GPU instances are a natural fit for this.

Conclusion: Easy to Polish the Facade, Hard to Build the Foundation

The conclusion of this comparison can be summed up in one sentence: the ability to create a strong first impression improved clearly between generations — composition is more organized, and water reflections evolved from surface effects to real mechanisms. But backend parameter consistency, such as tide level versus step height, can still fail. For model developers, state propagation and hard-constraint satisfaction are the real next challenges. For users, acceptance can never stop at demo screenshots. Turning both “looks right” and “computes correctly” into measurable assertions is the only reliable basis for model selection and delivery.