Testing 3D Scene Generation in LLMs: Beyond the First Impression

Why Benchmarks Are Not Enough: Version Upgrades Need Structured Testing

For developers, benchmark scores are aggregate metrics. They answer the question “Which model is stronger overall?” but fail to answer two more critical questions: How much did a new version actually improve on the tasks I care about? And are there any hidden regressions? In practice, the same model can perform very differently across different tasks, and aggregate scores tend to mask exactly these divergences.

A more effective approach than chasing leaderboard rankings is controlled same-prompt comparison: fix the prompt, fix the product entry point and inference tier, fix the time window, and have the old and new model versions solve the same problem. Then break down the differences layer by layer. Two engineering details deserve attention:

• Time window: Online model services may change dynamically over time (often referred to informally as “degradation”). Old and new comparisons must be completed within the same time window, ideally with multiple samples to reduce randomness.
• Consistent mode: If a product offers different inference tiers, both sides of the comparison must use the same tier. Otherwise, the experimental variables are not isolated, and the conclusions are unreliable.

Test Case Design: A Multi-Dimensional 3D Challenge

This evaluation used a deliberately broad test case: generate an offline, single-file HTML page using native WebGL2, with no external images, model files, or third-party libraries. The scene must include a massive ring-shaped water gate, a terraced coastal city, and tides that rise and fall. The prompt specified two hard constraints:

1. Establish composition, architecture, and materials first — testing spatial organization and artistic expression.
2. The tide must not be just “a blue plane moving up and down” — ship draft, shoreline waterline, and plaza submersion state must all respond to tide level — testing consistency of parameterized constraints.

This problem was chosen for several reasons:

• Single file + zero dependencies: All complexity is compressed into a single context, testing the model’s overall command of the rendering pipeline, matrix transformations, and shader authoring.
• Clear visual hierarchy: The giant ring serves as the visual anchor, while the coastal city is the narrative focus — testing compositional judgment.
• Tide level is a computable constraint: Step elevations and maximum water levels can be verified directly from the source code. Visual impressions cannot be faked.

A skeleton of the expected structure:

<!-- Constraints: single file, native WebGL2, no external resources or third-party libraries -->
<canvas id="gl"></canvas>
<script>
const TIDE = { min: 0, max: 6.5, periodSec: 120 };   // Tide-driving parameters
const STEPS = Array.from({ length: 9 }, (_, i) => 0.6 * (i + 1)); // Nine-tier step elevations
// Each frame: update ship draft, waterline position, and plaza submersion based on tide level
</script>

Layer 1 Evaluation: Initial Framing and the Model’s First Impression

The default camera position is the most direct expression of how well a model understands the “complete world” it has generated. It is also the first impression users form within 0.5 seconds of opening the result.

Typical problems in the previous version: The giant ring itself was large, but the mountains on both sides occupied even more of the frame. A significant portion of the initial view was filled by nearby slopes and distant peaks, pushing the city deep into the bay. Buildings and plazas had weak recognizability — the main object was established, but the world was not.

Improvements in the new version: The giant ring, bridge openings, beacon towers, and terraced dwellings were organized into a coherent landscape. Buildings spread along both shores, and the central plaza displayed clear concentric step layering. The warm-cool material contrast between oxidized copper and stone was also more distinguishable. The result came closer to the “complete world” the prompt requested, moving beyond the rigid feel of “a single ring-shaped object placed on terrain.”

This layer can be further quantified:

• Does the main subject occupy a reasonable proportion of the frame, and is the visual flow logical?
• Are mid-ground and background layers readable?
• Is material differentiation clear (warm-cool contrast, roughness contrast)?

Layer 2 Evaluation: Look at the Mechanism, Not the Surface Effect

Two implementations that look similar may differ by an order of magnitude in their underlying mechanisms. Take water reflections as an example. The industry commonly uses two approaches:

1. Analytical approximation: Use sky gradients, sun highlights, and light streaks to “fake” the sense of reflection. Implementation cost is low and results are acceptable, but there is no true spatial correspondence between the reflection and the actual object.
2. Mirror camera: Flip the entire scene across the water plane, render it again to a texture, and sample that texture in the water surface fragment shader. Reflections now have a real spatial source, with pixel-level correspondence between buildings, ships, and their reflections.

In testing, the old version relied primarily on analytical techniques, while the new version additionally introduced a mirror camera, rendering the city and ships into a texture sampled by the water surface — a substantive upgrade in implementation mechanism, and more noteworthy than simply “looking brighter and more realistic.”

The core idea of planar reflection:

// Flip the scene across the water plane y = waterY and render to an FBO texture first
mat4 reflect = mat4(
  1.0, 0.0, 0.0, 0.0,
  0.0, -1.0, 0.0, 0.0,
  0.0, 0.0, 1.0, 0.0,
  0.0, 2.0 * waterY, 0.0, 1.0
);
// In the water fragment shader, sample the reflection texture by screen coordinates,
// then apply normal perturbation and the Fresnel term

Proper implementation also requires clipping planes (to prevent underwater objects from appearing in the reflection) and oblique frustum culling. The presence or absence of these details is a strong signal for judging “mechanism depth” during evaluation.

Layer 3 Evaluation: Source-Level Verification of Parameterized Constraints

At the highest tide level, the new version actually revealed a weakness. The prompt required “most of the stone steps to be submerged,” but calculating from the step elevations and maximum water level in the source code: at full tide, only the lowest four tiers of the nine-tier plaza steps fell below mean water level. The “mostly submerged” requirement was not met. Ironically, the old version — despite its weaker initial framing — extended the maximum water level to cover the outermost steps.

This yields two important insights:

1. Visual improvement and constraint fulfillment are two independent capability dimensions. One can improve while the other regresses.
2. Acceptance criteria must be computable. Translate “mostly submerged” into an assertion:

function checkFlooding(steps, maxWater, ratio = 0.5) {
  const submerged = steps.filter(h => h < maxWater).length;
  return submerged / steps.length >= ratio;
}
// checkFlooding(STEPS, TIDE.max) === true means the requirement is met

In regression testing, such executable assertions are more reliable than human visual scoring and are easier to accumulate as reusable evaluation assets.

From Case Study to Methodology: A General Framework for Model Upgrade Evaluation

Abstracting the process above yields a three-layer evaluation model:

• Visual layer: Composition, hierarchy, materials, atmosphere — answering “Is the first impression good?”
• Mechanism layer: Whether the implementation approach has genuinely upgraded (e.g., reflections evolving from analytical approximation to mirror camera) — answering “Is it actually more capable?”
• Constraint layer: Source-level verification of parameterized requirements — answering “Are there hidden regressions?”

Supporting engineering practices:

1. Layered prompts: Establish composition and materials first, then behavioral constraints, then quantifiable acceptance criteria.
2. Fixed variables: Same entry point, same tier, same time window, multiple samples.
3. Assertions first: Any constraint verifiable from source code should be written as an executable check.
4. Build a regression set: Store test cases as cross-version comparable benchmarks, and rerun them with every version upgrade. Cloud-based model services and GPU infrastructure can batch-run these regression suites, forming a continuously comparable capability archive.

From a broader industry perspective, when distillation and other techniques yield diminishing marginal returns, it is not surprising that new versions show “regression” in certain dimensions — shifts in research focus directly manifest as capability divergence. This reminds us that the purpose of evaluation is not to assign a model a single total score, but to draw a capability radar chart: to see where training compute was invested and where gaps remain.

Conclusion

Progress in first impressions (initial framing and visual quality) is the easiest to perceive. Mechanism upgrades require reading the code to confirm. Constraint fulfillment gaps can only be exposed through source-level verification. For developers, “the real test lies in what happens behind the scenes” translates into a practical engineering principle: visual quality determines whether you try it, mechanism quality determines how far you can go, and constraint fulfillment determines whether you can ship to production. Building this three-layer evaluation system has more long-term value than chasing any single version release.