Evaluating LLM 3D Code Generation: Beyond First Impressions

1. Why Benchmark Scores Are Not Enough: The Value of Hands-on Testing

In the current rapid iteration of large language models (LLMs), various evaluation benchmarks emerge frequently. However, benchmark scores reflect statistical average capabilities and rarely answer the question developers care most about: In the specific tasks I need to accomplish, what are the actual strengths and weaknesses of the new generation of models?

Recently, we used a typical “front-end 3D scene generation” task as a sample to conduct a comparative test on older and newer generation models using the exact same prompt. The results show that the newer generation of models has made significant progress in “first impressions” and substantial upgrades in rendering implementation mechanisms. However, a clear divergence appeared in numerical constraints and scene consistency. This conclusion is not entirely consistent with the “comprehensive lead” presented by benchmarks, indicating that evaluating the code generation capabilities of LLMs requires a finer-grained methodology.

Before testing, strict environment control is necessary. Two easily overlooked details are:

• Fixed inference tier and interaction mode: Different inference strengths introduce additional variables. Comparative testing must be conducted at the same tier.
• Beware of version drift in online services: Model services may undergo silent adjustments over time. Important conclusions should be retested at multiple time points. Whether using LLM services provided by platforms like Baidu AI Cloud or a locally deployed inference environment, it is highly recommended to retain version snapshots and complete test records to ensure results are traceable and reproducible.

2. Task Design: A Stress Test That Separates the Best

The design quality of the evaluation task directly determines the credibility of the conclusions. This task had two key settings:

1. Single-File Delivery with Zero Dependencies

The model was required to output an offline HTML file based entirely on the browser’s native 3D rendering capabilities (WebGL2). External images, model files, and third-party libraries were strictly forbidden. The purpose was to eliminate the possibility of “asset搬运” (copying external assets) — all geometries, materials, water surfaces, and lighting had to be implemented by the model writing code on the spot. This approach squeezes the moisture out of the model’s actual capabilities.

2. Layered Prompts + Negative Constraints

The task was to construct an ocean city: a giant环形水闸 (circular sluice), a stepped city layout, and a tidal system that rises and falls. The prompt did not stop at vague descriptions like “make a rising and falling tide.” Instead, it explicitly stated:

The rising and falling tide cannot just rely on raising and lowering a single blue plane. Boats, waterlines, and the submerged state of the plaza must all change accordingly.

This negative + linked constraint is a crucial evaluation technique. It forces the model to answer one question: Are you “faking the visual perception,” or are you “building a system”? Only by understanding the linkage between the water level and multiple objects in the scene can the model deliver a qualified implementation.

3. Dimension One: Initial Composition — The “First Impression” Gap

The gap was apparent the moment the generated results were opened. This dimension evaluates the model’s understanding of scene organization and hierarchy.

• Previous Generation: Although the circular main structure was massive, the mountainous terrain on both sides took up more of the frame. A large area of the initial screen was filled with nearby hillsides and distant mountains. The city was squeezed deep into the bay, and the recognizability of the building groups and plaza was weak — the main subject stood up, but the world felt scattered.
• New Generation: The circular structure, bridge holes, beacon towers, and staggered residential buildings were organized into a coherent landscape. The building groups spread along both banks, and the central plaza featured clear concentric stepped layers. The warm and cold materials of oxidized copper and stone were also easy to distinguish.

This shows that the newer model’s understanding of a “complete world” is improving: evolving from “placing single objects on terrain” to “organizing a layered landscape.” This is backed by a comprehensive upgrade in spatial layout capabilities and prompt semantic coverage — a typical “first impression” capability enhancement.

4. Dimension Two: Rendering Mechanism — Implementation Path Upgrades Behind the Visuals

If you only look at screenshots, the water surfaces look identical. But reading the source code reveals completely different implementation mechanisms.

Scheme A: Analytical Simulation. It uses sky gradients, sun highlights, and light smears to “approximate” reflection effects. The computational cost is low, but the reflections lack a real spatial source. Essentially, it is a visual sleight of hand.

Scheme B: Mirror Camera Redraw. It mirrors the onshore scene across the water plane, redraws it completely, and samples it onto the water surface. The conceptual code is as follows:

// Core concept of mirror reflection (illustrative)
const mirrorCam = mainCamera.clone();
mirrorCam.scale.y = -1;                    // Mirror across the water plane
// Using the water plane as a clipping plane, redraw the scene to a reflection texture
const reflectionMap = renderToTexture(scene, mirrorCam, {
  clipPlane: waterPlane
});
// The main screen samples the reflection texture in screen coordinates and overlays normal distortion
water.material.reflectionMap = reflectionMap;
water.material.distortion = waveNormalMap;

The significance of Scheme B is that: there is a one-to-one spatial relationship between the reflection and the buildings. During high and low tides, the reflection changes naturally with the water surface, rather than being a “texture-based fake reflection.” This is a generational upgrade at the implementation mechanism level.

The takeaway for developers is straightforward: Accepting AI-generated code cannot be based solely on rendered screenshots. Two implementations with similar visuals may differ by an order of magnitude in engineering value — requiring the model to explain its implementation path is a necessary step in code review.

5. Dimension Three: Constraint Adherence — The Most Easily Overlooked “Backend Test”

What truly separates professional capabilities is the fulfillment of quantitative constraints. The prompt required that “most stone steps are submerged at high tide.” This requires the model to align the geometric parameters of the steps with the maximum water level in the source code. The verification method is as follows:

// Constraint verification example (values are illustrative)
const STEP_COUNT = 9;                  // Number of plaza step layers
const STEP_HEIGHT = 0.6;               // Height of a single step
const TERRACE_BASE_Y = 2.0;            // Step base elevation
const MAX_TIDE_Y = TERRACE_BASE_Y + 2 * STEP_HEIGHT; // Highest tide level in the source code
// Number of step layers submerged at high tide
const submerged = Math.floor((MAX_TIDE_Y - TERRACE_BASE_Y) / STEP_HEIGHT);
// If "most steps are submerged" is required, submerged should be greater than half the total
console.assert(submerged > STEP_COUNT / 2, 'High tide level does not cover most steps');

An interesting reversal occurred in the test results: The newer generation version, which had the upper hand in initial visual composition, only had the lowest four layers of steps below the average water level at high tide according to its source code parameters. It did not fulfill the requirement of “most being submerged.” Conversely, the older generation version, which lacked visual advantages, extended the highest water level to cover the outermost steps.

This reveals an important phenomenon: Model capabilities are structurally divergent. Improvements in expressive and organizational capabilities do not mean that numerical calculation and long-range constraint capabilities have improved synchronously.

6. Methodology Distillation: A Reusable Evaluation Framework

By abstracting the above process into a universal workflow, any team can reuse it:

1. Task Isolation: Single-file, zero-dependency to eliminate external resource interference.
2. Layered Prompts: Four-layer description — composition, material, behavior, and constraints. The constraint layer must include negative expressions and quantitative metrics.
3. Three-Layer Acceptance: Initial visual perception (screenshot comparison) → Implementation mechanism (source code review) → Constraint calculation (numerical verification), going deeper layer by layer.
4. Version Comparison & Multi-round Sampling: Run the same prompt across multiple model generations to observe both progress and “regression,” avoiding being misled by a single result.
5. Record Capability Divergence: Benchmark average scores mask structural differences. Recording by dimension is the only way to guide model selection.

7. Conclusion: The Real Test Lies in the “Backend”

From an industry perspective, after the diminishing marginal returns from distillation, mainstream model providers have generally shifted to increasing post-training investments in vertical directions like coding and creative generation. This explains why newer models progress rapidly in “expressive” tasks — composition, atmosphere, and mechanism design are all capabilities “moving toward the frontend.”

For developers, the conclusion is equally clear: “First impressions” are no longer the bottleneck. Backend capabilities like numerical consistency, long-range constraints, and cross-object linkage are the true watershed for the next phase. Treat AI as a talented but junior engineer who needs validation. Inspect every line of its code with engineering standards — this is perhaps the most essential development literacy in the era of large models.