Evaluating LLMs in 3D Scene Generation: Visual Appeal vs. Constraint Verification

1. Why Use a “3D Web Page” to Evaluate LLMs

As Large Language Models (LLMs) iterate rapidly, the gap between benchmark scores and real-world user experience is widening. Benchmarks may suffer from data contamination, and evaluation metrics do not always cover the capabilities developers genuinely care about. In coding scenarios, one of the hardest abilities to “game” through benchmark tuning is long-range planning and constraint satisfaction.

To test this, I designed a composite task: prompt the model to generate a completely offline, single-file HTML page. The theme is a “Stepped Sea City” — the scene must feature a giant ring-shaped water gate, buildings spreading along both shores, a central plaza with concentric steps, and a functional tidal system. The prompt also specifies hard constraints: the rising and falling tides cannot be simulated merely by moving a blue plane up and down. The buoyancy of boats, the position of watermarks on the shoreline, and the submersion state of the plaza’s steps must all react synchronously.

There are three reasons for choosing this task:

1. Zero Dependencies, Testing Only the Model Itself. The prompt forbids referencing any external images, model files, or third-party libraries. Everything must be implemented using the browser’s native WebGL API. This eliminates the advantage of “package calling,” forcing the model to rely entirely on its own spatial imagination, shader programming, and numerical calculation capabilities.
2. Sufficient Task Length. A complete 3D city involves multiple subsystems: terrain, architecture, water bodies, lighting, and animation. If any part fails, the entire page breaks, naturally testing long-range consistency.
3. Verifiable Constraints. Requirements like “most stone steps should be submerged at high tide” can be translated into numerical assertions against source code constants, shifting evaluation from “visual scoring” to “scripted verification.”

2. Control Variables: Ensuring a Fair Comparison

Two models from the same series (referred to below as the older version and the newer version) were tasked with completing the prompt under the same desktop environment and reasoning effort level. Two easily overlooked engineering details should be noted here:

• Beware of Online Configuration Drift. The inference configurations of online models (quantization strategies, request routing, etc.) may be adjusted over time. Community experience suggests that model performance can fluctuate after release. Therefore, comparative tests should be conducted within adjacent time windows, and multiple samplings of key tasks should be performed to avoid mistaking fluctuations for generational differences.
• The Result is the Final Product. The generated single-file HTML page requires no build process. By uploading it to an object storage service (such as Baidu AI Cloud or similar cloud providers) and enabling static website hosting, you can immediately obtain a shareable demo link, facilitating team reproduction and review.

3. Dimension One: Visual Appeal — Generational Progress in “First Impressions”

Both model versions delivered a compliant native WebGL single file: no external dependencies, complete scenes, and functional tides. What truly separated their visual quality was how the default camera organized the scene’s primary and secondary elements.

• The Older Version’s Flaw: “Unbalanced Composition.” The giant ring was adequately sized, but the mountainous terrain on both sides occupied too much of the frame. The first screen was largely filled with foreground slopes and distant mountains, squeezing the city into the depths of the bay. The辨识度 of the building complex and the plaza was weak — viewers saw a “ring-shaped object placed on terrain” rather than a city.
• The Newer Version: “Landscape Narrative.” It organized the ring gates, bridge arches, beacon towers, and tiered residential buildings into a cohesive landscape. The building complex spreads along both shores, the central plaza features clear concentric step layers, and the cold tones of oxidized copper contrast identifiable against the warm tones of stone materials. This aligns much closer with the prompt’s requirement for a “complete world.”

The conclusion for this dimension is clear: regarding issues that directly impact first impressions — such as camera placement and subject framing — the newer version shows perceptible progress. This aligns with the general trend of increasing investment in post-training — the ability to align with human aesthetic preferences is often the first capability to be reinforced.

4. Dimension Two: Rendering Mechanisms — From “Looks Like” to “Calculated Correctly”

Water surfaces are the best differentiator for assessing implementation levels. The two model versions chose two different technical routes:

• Older Version: Analytical Approximation. Primarily used analytical sky colors, sun specular highlights, and light smears to simulate reflections. This implementation is simple and low-cost, but the reflections have no true spatial correspondence with actual objects in the scene. Essentially, it is “painted to look like” a reflection.
• Newer Version: Mirror Camera Plane Reflection. Introduced an additional reflection camera mirrored across the water plane. It re-renders the city and boats into a texture, which the water shader then samples via projection. The reflections of onshore buildings thus have a true spatial source, representing an upgrade at the implementation mechanism level.

The core process of planar reflection can be summarized in three steps:

// Pseudo-code: Three steps for planar reflection
// 1) Construct a mirrored view matrix relative to the water plane
mat4 reflView = mirrorMatrix(waterPlane) * mainView;
// 2) Render the scene to a framebuffer texture using the reflection camera 
// (with a clipping plane, retaining only the part above the water)
vec4 clipPlane = vec4(waterNormal, -waterDist);
renderToTexture(reflView, clipPlane);
// 3) Perform projected sampling on water fragments, 
// then blend with normal perturbation and Fresnel terms
vec4 pc = texMatrix * worldPos;
vec3 refl = texture2D(reflTex, pc.xy / pc.w).rgb;

Why is the mechanism more important than the visual appeal? Because visuals can be “tuned” via parameters, while the mechanism determines the ceiling and stability of the effect. As soon as the camera moves or a boat sails, the analytical approximation exposes its flaws, whereas planar reflection naturally follows scene changes. When evaluating code generated by models, reviewers should first prioritize “which route it chose” before discussing “how well it was tuned.”

5. Dimension Three: Parameter Fulfillment — The Real Test is in the Backend

The newer version, which dominated the first screen, actually dropped the ball in this dimension. Based on the step height and maximum tide level parameters found in its source code: the plaza has a total of nine stone steps, but at high tide, only the lowest four steps are below the average water level. The prompt’s requirement that “most steps should be submerged” was not fulfilled. Conversely, the older version, which lacked visual appeal, set the maximum water level to cover the outermost steps.

This case reveals the most easily missed component in LLM evaluation: translating every hard constraint from the prompt into executable assertions against the source code. The naked eye cannot determine “how many steps were submerged” from the screen, but reading the code can:

// Constraint verification script example: Extract key constants from generated code and assert
const STEP_COUNT  = 9;     // Total number of steps (from generated source code)
const STEP_HEIGHT = 0.75;  // Height of a single step
const MAX_TIDE    = 3.0;   // Maximum tide level elevation relative to average water level
const submerged = Math.floor(MAX_TIDE / STEP_HEIGHT);
const pass = submerged >= STEP_COUNT * 0.6; // "Most" is quantified as 60%
console.assert(pass,
  `High tide should submerge most steps, but only ${submerged}/${STEP_COUNT} are actually submerged.`);

Taking this further, the entire set of requirements can be compiled into an assertion checklist: single file, zero external requests, WebGL context exists, tidal animation simultaneously covers boats/waterline/steps, maximum water submersion ratio meets the standard, etc. Once the evaluation script runs, it directly outputs a pass rate. This is the closed loop of “evaluation as testing,” and the hard standard for judging whether a model has truly “understood the requirements.”

6. From Case to Framework: A Reusable Evaluation Methodology

Abstracting the practices above, LLM coding evaluation can be broken down into three layers:

1. Visual Layer (First Impression): Composition hierarchy, layered rhythm, material differentiation, and default camera performance. Suitable for human blind review, but requires anonymization and side-by-side comparison to avoid bias.
2. Mechanism Layer (Implementation Route): Whether the correct technical route was chosen (e.g., planar reflection over analytical approximation) and whether edge cases were handled. Requires manual code reading, focusing on auditing the implementation methods of key subsystems.
3. Constraint Layer (Parameter Fulfillment): Translating hard prompt constraints into numerical assertions, automatically judged by scripts, outputting a quantifiable pass rate.

Combined with an environment control checklist — fixing reasoning effort, fixing time windows, multiple samplings, and A/B blind comparisons — you can achieve an evaluation process that is low-cost but yields significant confidence.

Finally, regarding the source of generational differences. As the marginal returns from distillation diminish, next-generation model improvements rely more heavily on post-training and reinforcement learning oriented toward long-range tasks. This means the evaluation focus must also shift: ask less “does it know,” and ask more “can it maintain a dozen constraints simultaneously across hundreds of lines of code.” This empirical test perfectly validates this point — “first impressions” are becoming basic skills, while constraint satisfaction and numerical consistency are the true differentiators for the next generation.

There are only two recommendations for developers: write acceptance criteria into your prompts, and write assertions into your evaluation scripts. The former constrains the model; the latter protects you.