Testing LLMs on 3D Scene Generation: Visuals Are Easy to Shape, Logic Is Hard to Deliver

Why Complex 3D Scenes Make a Good Stress Test for Coding LLMs

Once large language models broadly gained long-context and code generation capabilities, “writing code that runs” stopped being the hard part. What actually separates models is this: given a complex goal coupled together from multiple subsystems, can a model handle visual organization, rendering mechanics, and numerical logic all at once?

Front-end 3D is a typical example of such a task. It demands that the model handle scene composition, lighting design, water physics, and state coupling within a single file of code — any weak link shows up directly in the running result. That is why “offline, single-file 3D scene generation” is becoming a touchstone for the overall capability of coding models: it cannot hide weaknesses behind external assets, and it cannot get by on template reuse. Every capability has to show up in the lines of code the model actually writes.

One notable recent trend: newer-generation models are improving much faster on “first-glance appearance” than on logical correctness. This article uses a concrete side-by-side comparison (outputs from two generations of anonymized models under identical conditions as analysis samples) to unpack the technical reasons behind this uneven progress, and offers a reusable evaluation methodology.

Test Design: A Reproducible Evaluation Task

The test asked the model to build, in a single offline HTML file, a complete scene featuring a giant ring-shaped tidal water gate, a terraced sea city, and a rising-and-falling tide. Constraints included:

• Native WebGL2 only — no external images, model files, or third-party libraries;
• The prompt explicitly required establishing composition, architecture, and materials first, then handling dynamic effects;
• The tide could not simply rise and fall as a flat blue plane — ship draft, waterline position, and plaza flooding state all had to change in sync.

To control variables, both outputs were generated on the same client with the same inference mode, minimizing interference from service-state fluctuations at different times, and the full source code was preserved for item-by-item verification afterward.

The value of this task lies in its naturally layered difficulty: composition and materials test the model’s “aesthetic decisions,” water reflection tests “mechanism design,” and the numeric relationship between tide level and steps tests “logic fulfillment.” Three ascending layers of difficulty that map neatly onto three capability gaps in coding models.

Dimension 1: Scene Composition — Where First Impressions Split

Both generations delivered runnable, native-WebGL2 single files, but the first-screen impression differed sharply.

The typical problem with the earlier version was “the subject drowning in its environment”: the giant water gate sat at the center of the frame, but the mountains on both sides occupied more of the canvas; near hillsides and distant peaks filled most of the first screen, while the city’s buildings shrank deep into the bay with weak recognizability. Viewers had to squint to confirm the “sea city” even existed — a failure in visual storytelling.

The newer version organized the giant ring, bridge arches, beacon towers, and terraced housing into a coherent landscape: buildings spread along both banks, the central plaza showed clear concentric step tiers, and the warm-cold contrast between oxidized copper and stone was easier to distinguish. It no longer reads as a stiff collage of “a ring-shaped object placed on a terrain,” but as a “complete world” with hierarchy, depth, and detail density.

Technically, the essence of this progress is joint optimization of the camera system and scene layout — the newer model understood that “the default camera position decides the first impression” and proactively reserved a visual corridor for the main viewpoint. This capability is relatively easy to acquire through training, because compositional quality has a massive body of human work to reference; it belongs to the category of “imitable abilities.”

Dimension 2: Water Reflections — a Generational Gap in Implementation Mechanism

Water rendering is the part of this task that most exposes “depth of mechanism understanding,” and the two outputs chose starkly different technical routes:

• Analytic approximation: procedural sky colors, sun highlights, and light streaks approximate the look of reflection. Computation is extremely cheap, but the reflections in the water have no real spatial correspondence with the objects on shore — the “light patches” on the surface are just artifacts computed by a shader.
• Mirror-camera reflection: an extra virtual camera, mirrored across the water plane, redraws the city and ships completely, renders them to an offscreen texture, and then samples that texture on the water surface. Reflections of onshore buildings therefore have a genuine spatial source — a substantive upgrade in rendering mechanism.

The core logic of mirror-camera reflection can be simplified as:

// Illustrative: the basic flow of mirror-camera water reflection
function renderReflection(scene, mainCam, waterLevel) {
  const mirrorCam = cloneCamera(mainCam);
  // Mirror the camera position across the water plane
  mirrorCam.position.y = 2 * waterLevel - mainCam.position.y;
  mirrorCam.pitch = -mainCam.pitch;   // Flip pitch
  mirrorCam.roll  = -mainCam.roll;    // Flip roll
  // Render the reflected scene into an offscreen buffer for the water shader to sample
  bindFramebuffer(reflectionFBO);
  renderScene(scene, mirrorCam);
  bindFramebuffer(null);
}

This shift in route marks the model’s evolution from “tuning parameters to fake an effect” to “understanding the causal chain of the rendering pipeline.” It is a typical mechanistic upgrade — harder to acquire than compositional progress, and much closer to real graphics engineering capability.

Dimension 3: Parameter Fulfillment — the Commonly Ignored “Backend Test”

The strictest constraint in the task was: “At high and low tide, most of the stone steps should be submerged.”

This required the model to do something entirely unrelated to visuals: numeric verification. The model had to derive the flooding range from the maximum water-level parameter and the geometric dimensions of the steps, ensuring that at high tide the water really did cover the outer tiers.

The results were ironic: the newer version, superior in both first-screen appearance and reflection mechanism, failed exactly on this item. Checking the step heights and maximum water level in its source code: at high tide, only the lowest four of the nine concentric step tiers sat below the average water level — far short of “most steps submerged.” Meanwhile, the earlier version, weaker in composition, had correctly set its maximum water level to cover the outermost tier.

This result reveals a key fact about current LLM coding ability: visual priors and logical post-verification are separate. Rendering quality has oceans of data to imitate, whereas closed-form numeric reasoning like “the water-level parameter must satisfy a geometric constraint” lacks direct training signals and can only be solved through general-purpose reasoning — which is exactly why it becomes the weak point.

The more insidious risk: this kind of failure is nearly invisible in a demo. In a video, the tide still rises and falls and the scene still looks beautiful; only someone who opens the source code and checks the parameters will spot the problem. That is precisely the deceptiveness of the “first impression” — and why the real test lives in the backend.

Build Your Own Layered Evaluation Framework

Based on the breakdown above, consider evaluating a coding model’s 3D generation ability in three layers, accepting each layer separately:

1. Appearance layer: composition hierarchy from the default camera, material distinctiveness, color atmosphere. Quick check — “can you read the scene’s subject within three seconds of the first screen?”
2. Mechanism layer: is the core effect “faked” or “computed”? Check — read the source for a causal chain: do reflections come from a real mirror render, do lights and shadows follow a single consistent light source, do tide and ship states stay coupled?
3. Logic layer: do numeric parameters satisfy the prompt’s quantitative constraints? Check — manually verify key numbers, or write an assertion script to validate automatically.

The third layer is the one most often skipped, yet it best reflects a model’s reliability in real engineering. Here is a ready-to-use automated acceptance example:

// Illustrative: logic-layer automated acceptance — verify submerged step count at high tide
const STEP_HEIGHT = 0.8;   // Height per step tier (extracted from the generated source)
const STEPS = 9;           // Total number of concentric step tiers
const HIGH_TIDE = 5.0;     // Maximum water level (extracted from the generated source)
const PLAZA_BASE = 1.2;    // Base elevation of the plaza's lowest tier
function submergedSteps(tide) {
  let count = 0;
  for (let i = 0; i < STEPS; i++) {
    const stepTop = PLAZA_BASE + i * STEP_HEIGHT;
    if (stepTop < tide) count++;  // A tier counts as submerged if its top is below the water level
  }
  return count;
}
// Assertion: at high tide, "most stone steps are submerged" (>= 2/3 of tiers)
console.assert(
  submergedSteps(HIGH_TIDE) >= Math.ceil(STEPS * 2 / 3),
  'High-tide submersion count does not meet the prompt requirement'
);

The value of this method is that it fully decouples “looks right” from “computes right”: in production, most code defects are not “look wrong” — they are “compute wrong.”

Closing Thoughts: After the First Impression, the Real Test Begins

Taken together, this round of testing supports one conclusion: newer-generation models have made substantive progress in scene composition and rendering mechanisms — the ability to shape a “first impression” is clearly maturing. But on backend abilities such as fulfilling numeric logic, progress is not in sync, and in some cases the two may even trade off against each other.

The takeaway for developers is direct: when evaluating AI coding ability, do not be bought by the first screen. During technology selection or delivery acceptance, insist on parameter verification and logic assertions, and score “does it look good” and “is it correct” separately — the former determines how the demo plays, the latter determines production reliability.

From a broader perspective, as mainstream training paradigms see shifting marginal returns, gains in “imitable abilities” (visual style, structural organization) will keep getting easier, while gains in “reasoning-required abilities” (constraint satisfaction, numeric consistency, cross-module logical closure) are the real dividing line for the next stage. After the first impression, the test has only just begun.