LLM 3D Scene Generation Evaluation: Visual Quality, Implementation Mechanism, and Requirement Compliance

1. Why Use 3D Scene Generation as a Test Case

Over the past few months, the overall landscape of the LLM industry has shifted noticeably. On one hand, flagship models have pushed parameter counts into the trillions and context lengths toward one million tokens, with post-training increasingly focused on high-barrier capabilities such as code generation and 3D creation. On the other hand, substantial resources are flowing toward lightweight, fast-tier models built for high-volume throughput, creating two distinctly different product tracks. As the marginal returns from distillation gradually decline, post-training quality has become the dividing line between leaders and followers.

Against this backdrop, benchmark scores are less informative than a carefully designed real-world task. Different models diverge sharply across different tasks, and only by placing a model in a concrete scenario can its progress and regressions become visible.

“Implement a 3D seaside scene in a single offline HTML file” is exactly such a good task, for three reasons:

1. Highly comprehensive: Spatial layout, visual design, graphics knowledge, and front-end engineering are all required, and any weak point will be exposed;
2. Tightenable constraints: Requiring a single file, browser-native 3D graphics standards (WebGL2), and zero external image/model/third-party library dependencies forces the model to apply real capability where it matters and eliminates the dilution of “library assembly”;
3. Verifiable at multiple layers: Visual quality can be checked via screenshots, mechanism via source code, and compliance via numerical calculation—naturally suited to layered evaluation.

2. Evaluation Method: One Task, Three Dimensions, Fully Reproducible

Controlled variables are the prerequisite: old and new model generations use the same environment, the same interaction mode, the same prompt, and the same scoring criteria. Note that online models are continuously iterated and adjusted over time, and community discussions about “performance fluctuations” are common, so evaluation conclusions must be annotated with time and version; a fixed prompt plus fixed scoring criteria is the minimum requirement for reproducibility.

Prompt design is an overlooked step in many teams, with three key points:

1. Establish composition and materials first: Explicitly require the model to first specify camera organization, building hierarchy, and material tone—front-loading “aesthetic requirements” rather than patching them afterward;
2. Block shortcuts: Directly state that “tide changes cannot rely solely on raising and lowering a single blue plane,” and require boats, waterlines, and plaza submersion states to be linked to tide level; otherwise the model will most likely deliver the lowest-cost implementation;
3. Quantify soft requirements: Expressions like “most stone steps are submerged” must be convertible into numerical relationships among step count, per-step height, and maximum water level; otherwise they cannot be verified afterward.

These clauses can be固化 into a requirements checklist:

[Sample Requirements Checklist]
1. Single-file HTML, zero external image/model/third-party library dependencies
2. Scene elements: giant ring-shaped water gate + stepped sea city + adjustable tide
3. Tide linkage: boats rise/fall with water level, waterline position moves with tide, plaza submersion state stays synchronized
4. At maximum tide, "most stone steps are submerged"
   (Calculable: step count x per-step height vs. maximum water level coverage)

3. Dimension 1: First-Screen Visual Quality—Testing “Scene Organization” Capability

Both implementations delivered acceptable single-file 3D scenes, but what immediately separated them in visual quality was how the default camera organized scene hierarchy.

The weaker version: The ring-shaped主体 was large enough in volume, but was crowded out of frame by mountains on both sides. The first screen was largely filled by nearby slopes and distant mountains, with the city tucked deep into the bay and the building cluster and plaza lacking recognizability—viewers had to “work” to identify the focal point.

The stronger version: It organized the giant ring, bridge arches, beacon towers, and terraced housing into a coherent landscape. Buildings spread along both banks, the central plaza had clear concentric step levels, and the cool/warm contrast between metal and stone was easy to distinguish—closer to the “complete world” required by the prompt rather than the stiff feeling of “placing a single ring-shaped object on terrain.”

Technically, this tests the model’s ability to translate a design brief into layout constraints: camera parameter selection, screen-share ratio of each element, gaze guidance paths, and visual weight distribution. Two improvement practices:

• Have the model “propose a plan before writing code”: First output a composition description (subject, supporting elements, visual flow), confirm it, then generate the implementation—this significantly reduces first-screen failures;
• Give hard constraints in the prompt: For example, “the subject occupies no less than 1/2 of the frame height” and “foreground occlusion does not exceed 1/4 of the frame,” turning vague aesthetic requirements into executable instructions.

4. Dimension 2: Implementation Mechanism—Two Technical Routes for Water Reflection

Water is the “mechanism touchstone” of this task, and the two versions took different routes:

• Route A: Analytic approximation. Uses analytic sky, sun highlights, and light streaks to simulate reflection. Advantages are low cost and stability; the drawback is that reflections have no real geometric source—essentially “painted reflections.”
• Route B: Planar reflection (mirror camera). Introduces an additional mirror camera, redraws the city and boats symmetrically about the water plane, then samples that into the water shader. Reflections of shoreline buildings correspond one-to-one with real geometry—an upgrade at the implementation-mechanism level, not just parameter tuning.

The core steps of planar reflection can be summarized in three stages (illustrative pseudocode):

// Planar reflection sketch (pseudocode)
const mirrorCamera = mainCamera.clone();
mirrorCamera.applyReflection(waterPlane);   // 1. Build a mirror camera symmetric about the water plane
renderer.setRenderTarget(reflectionRT);     // 2. Offscreen render to reflection texture
renderer.render(scene, mirrorCamera);
// 3. Projective sampling in the water fragment shader
// vec2 uv = projCoord.xy / projCoord.w * 0.5 + 0.5;
// vec3 refl = texture(reflectionMap, uv + rippleOffset).rgb;

Engineering cost must be considered: a mirror camera means the entire scene is drawn one extra time, nearly doubling geometry overhead. Common optimizations include clipping the reflection camera frustum by the water plane, lowering reflection texture resolution, and falling back to Route A on mobile. Evaluation should also include “whether these engineering trade-offs were made” as an observation item.

5. Dimension 3: Requirement Compliance—Verify with Source Code, Not by Eye

When it came to the maximum tide level parameter, the visual-quality ranking and the compliance ranking reversed.

Calculating from the step height and maximum water level in the generated source code: in one version, only the lowest four of the nine plaza steps were below mean water level at full tide, failing to deliver “most stone steps are submerged”; the other version, despite weaker first-screen composition, extended maximum water level coverage to the outermost steps.

This case shows: good visual quality does not equal good compliance. When LLMs generate code, visual presentation and numerical correctness are two independent capability lines—the former relies on composition and material accumulation, the latter on constraint satisfaction and geometric computation. Evaluation must turn the requirements checklist into executable assertions:

• Static calculation: Read constants from source code (step count, per-step height, maximum water level) and perform geometric verification;
• Runtime assertions: Inject scripts into the page, read scene object states, and verify that tide linkage actually occurs;
• State archiving: Capture screenshots at key states such as lowest/highest tide for cross-version comparison.

Only by automating these three types of checks can “compliance rate” avoid being skewed by demo effects.

6. Model Evolution Direction Seen Through Progress and Regression

Returning to the industry trends reflected in the old-vs-new comparison, three points deserve elaboration:

1. Benchmark divergence is normal. New-generation model highlights tend to concentrate on long-horizon programming and tool-assisted tasks; other tasks may not improve in sync and may even regress locally. When selecting models, don’t只看 composite leaderboards—look at sub-scores that match your task profile.
2. Track divergence continues. Lightweight fast-tier and frontier flagship models serve different scenarios: the former competes on response speed and cost, the latter on capability ceiling. Evaluators should build separate scorecards rather than measuring both with the same ruler.
3. Post-training becomes the watershed. As base capabilities converge and distillation returns decline, data quality and domain reinforcement (code, 3D, vertical industries, etc.) determine the real increment of each product generation. This also explains why “same task, old vs. new generation” differences are more informative than cross-vendor benchmark runs.

7. Turning Evaluation into an Engineering Process

For engineering teams, one-off “hands-on evaluation” has limited value. The methods above should be consolidated into a sustainable pipeline:

• Layered question bank: Composition, mechanism, and constraint tasks configured in proportion, covering the three capabilities of “looks good,” “is clever,” and “is correct”;
• Automated scoring: A trio of key-state screenshot comparison with structural similarity, static source checks (dependency count, constant calculation), and runtime assertions;
• Version regression: Fixed prompts and scoring criteria, rerun after each model version update, and output incremental reports;
• Infrastructure: The pipeline can be built entirely on general cloud capabilities—using GPU compute instances for batch inference (for example, model services and compute resources offered by cloud platforms such as Baidu AI Cloud), object storage for screenshots and source artifacts, and logging and monitoring services to track task status. The evaluation itself does not depend on any proprietary component.

Conclusion

From the improvement in “first impression,” it is clear that models are indeed evolving in aesthetics and scene organization. But what truly determines whether generated results can be put into use are three “back-end” things: whether constraints are satisfied, whether numbers are correct, and whether long-horizon consistency holds. Only by setting checkpoints at all three gates—visual quality, mechanism, and compliance—can evaluators avoid being misled by a beautiful demo. This is both the落地 of evaluation methodology and the best response to the idea that “the real test lies in the back end.”