Evaluating LLM Frontend 3D Generation: The Real Test Beyond the First Impression

1. Evaluation Task Design: Why an “Offline Single-File 3D Scene”?

When evaluating the frontend code generation capabilities of Large Language Models (LLMs), task design is half the battle. A highly differentiating test task must satisfy three hard constraints:

1. Only a single offline HTML file is allowed, implemented with native WebGL2.
2. No external images, model files, or third-party libraries can be referenced.
3. All geometry, materials, lighting, and animations must be procedurally generated by the code.

These constraints yield three key benefits:

• Eliminates asset shuffling: Without textures or models to rely on, the model must genuinely “write” the shapes and textures, making it impossible to use image maps to mask deficiencies in geometric construction.
• Comprehensive evaluation dimensions: A single task simultaneously tests spatial composition aesthetics, rendering pipeline comprehension, state management, and performance awareness.
• Horizontal comparability: Under identical constraints, multiple outputs can be placed side-by-side, making performance gaps obvious without the need for subjective justification.

For this test, the prompt specified: a giant annular watergate, a stepped ocean city, and a rising and falling tide. Two critical designs were embedded in the prompt:

1. The model was first required to establish the composition, architecture, and materials.
2. It was explicitly stated that “the rising and falling tide cannot just be a single blue plane moving up and down—boats, waterlines, and the flooded state of the plaza must all change accordingly.”

The first requirement tests the “first impression,” while the second acts as a hook for “hardcore engineering skills.” In practice, almost all compliant outputs satisfied the first requirement. What truly separated them was the second, along with the rendering mechanisms hidden beneath the water.

2. First Impressions: Primary and Secondary Organization in the Default Camera View

Two compliant outputs might pass the technical baseline, but the moment you open the initial viewport, the difference in visual appeal is immediate.

The typical problem with poorer outputs is not “drawing it inaccurately,” but a “disorder of priorities”:

• Although the annular watergate is the largest object in the scene, the mountains on both sides occupy more of the frame.
• Near slopes and distant mountains fill the initial screen, compressing the city deep into the bay.
• The architectural clusters and plaza have weak recognizability. The audience has to strain to find the narrative protagonist—the giant ring stands in the center, but the ocean city is reduced to a background board.

Better outputs use the default camera as a narrative tool:

• The ring, bridge arches, navigation towers, and terraced residential buildings on both banks are organized into a coherent visual flow.
• The central plaza presents clear concentric terraced layers, directly visualizing the “stepped ocean city” concept.
• The cold green of oxidized copper and the warm gray of stone create a distinct contrast in material temperature.
• The overall scene reads like a self-consistent “complete world,” rather than a rigid collage of a single ring object placed on terrain.

From this, we can extract a Composition Checklist to quickly score the “first impression”:

Dimension	Checkpoint
Subject Framing	Does the core structure occupy the visual weight center of the initial screen?
Sightline Guidance	Is there a clear guiding path between the foreground, midground, and background?
Area Recognizability	Are functional zones like plazas, residential areas, and waterways distinguishable at a glance?
Material Distinction	Are contrasts in temperature, roughness, and smoothness established?
Frame Economy	Do background elements serve the main subject, or swallow it?

3. Water Surface Reflections: From “Illusion Stitching” to “Mechanism Upgrade”

Having “bright spots on the water” might look the same at first glance, but the two implementation approaches reflect entirely different capability levels.

Approach 1: Analytical Approximation. This uses the colors of the analytical sky, sun highlights, and light trails to “simulate” a reflection. The advantage is almost zero additional performance overhead. The disadvantage is that there is no real spatial correspondence between the reflection and the scene—if the buildings on the shore change position, the water surface remains static, and the illusion breaks the moment you rotate the camera.

Approach 2: Mirror Camera with Off-screen Rendering. This introduces an additional camera mirrored along the horizontal plane, completely redrawing the city and boats, rendering them to an off-screen texture, which is then sampled by the water surface shader:

// Update reflection every frame: mirror camera + off-screen rendering
function updateReflection(mainCam, waterY) {
  // 1. Mirror the main camera along the water surface
  mirrorCam.position.y = 2.0 * waterY - mainCam.position.y;
  mirrorCam.up.set(0, -1, 0);      // Flip the up vector to avoid an upside-down image
  mirrorCam.lookAt(mirrorTarget(mainCam, waterY));
  // 2. Redraw the entire city and boats to an off-screen frame buffer
  gl.bindFramebuffer(gl.FRAMEBUFFER, reflectionFBO);
  drawScene(mirrorCam);
  gl.bindFramebuffer(gl.FRAMEBUFFER, null);
  // 3. Water fragment shader samples the reflection texture with normal perturbation
  // vec3 refl = texture(uReflection, uv + normalPerturb).rgb;
}

The evaluation criterion is not “is there a reflection,” but “does the reflection of the shoreline buildings have a real corresponding spatial source?” This represents an upgrade at the implementation mechanism level. The more complex the scene and the more camera movement there is, the faster the gap between the two approaches widens.

4. Dynamic Consistency: Tide Level is a Global State, Not Just a Number

The prompt requirement that “the tide cannot just be a rising and falling blue plane” evaluates the model’s understanding of state coupling. It must abstract the tide level as a global state and have all related objects subscribe to it, rather than writing independent animations for each.

// Tide level as a global state, objects respond according to their own rules
function setTide(level) {
  water.setHeight(level);                   // Overall water surface rises and falls
  boats.forEach(b => b.setBuoyancy(level)); // Boat draft lines float accordingly
  shoreline.updateWaterline(level);         // Shoreline water marks sync movement
  plaza.setFlood(level - PLAZA_BASE_Y);     // Plaza steps flood level by level
}

The associated checkpoints for this linkage include:

1. Boats: Does the floating state change with the tide level, without hanging in the air or clipping?
2. Waterline: Does the shoreline water mark move with the water level, and is the flooded edge natural?
3. Plaza: Are the steps flooded level by level according to height, changing color and showing immersion effects?
4. Reflections: Does the height baseline of the mirror camera update with the water surface? Otherwise, the reflection from the previous section will detach from the water surface.

5. Completeness: Pushing Parameters to the Boundary

A notable conclusion from testing is that an overall superior output can still lose points on extreme parameters. For example, when pulling the tide level to the maximum, whether water overflowing building bases handles clipping, whether the flooding order follows geometric logic, and whether flickering occurs under extreme values can all reveal oversights.

This indicates that merely looking at the visual appeal under default parameters is insufficient. Evaluators must proactively push the system to its boundaries:

• Is the scene still self-consistent under the highest and lowest tide levels?
• Does rapidly switching parameters consecutively cause state confusion?
• Does long-term running exhibit memory growth and frame rate decay?

6. Beyond the First Impression: The Real Test is in Backend Engineering

Frontend visual appeal is the easiest part to optimize through short-term intensive effort, and it is the easiest way to create a “stunning moment” in a demo. But for developers looking to use models in real engineering, a different set of capabilities determines usability:

1. Long-range Programming: Maintaining interface consistency in projects spanning thousands of lines, not forgetting earlier constraints, and being able to locate and fix defects introduced by the model itself.
2. Tool-augmented Tasks: Maintaining goal stability in multi-step calling chains, and correctly handling failures and retries.
3. Context Management: The value of ultra-long context lies not in “how much you can stuff in,” but in “whether you can remember and utilize it after it’s stuffed in.”

From an evaluation methodology perspective, two additional points are worth noting:

• Record Versions and Timestamps. Community experience shows that the online performance of the same model might fluctuate over time after server-side policy adjustments. Rigorous evaluations should fix and record the model version, test time, and environmental parameters, and retest if necessary, to avoid mistaking server-side changes for capability changes.
• Generational Comparisons are More Informative than Absolute Scores. Under the current industry backdrop where structural changes in training data (e.g., diminishing marginal returns from high proportions of synthetic data) are prevalent, observing the increments and regressions between adjacent generations of the same model family is often more predictive of future trends than simply looking at leaderboards. When most teams pivot to lightweight, fast models, teams still pushing the frontier demonstrate that the “rate of progress” itself constitutes a critical selection criterion.

7. Distillation: A Reusable Evaluation Process

1. Task Design: Apply hard constraints (single file, no external dependencies, native APIs) and explicitly require dynamic coupling.
2. Layered Prompting: First establish composition, architecture, and materials; then pressure test dynamic behavior and boundary conditions.
3. Initial Assessment: Score based on the composition checklist under the default camera, recording the “first impression.”
4. Mechanism Review: Read the code directly, focusing on verifying the reflection implementation, state coupling, and rendering pipeline approach.
5. Boundary Stress Testing: Test extreme parameters, rapid switching, and long-term running.
6. Hardcore Skill Spot Checks: Add long-range tasks (e.g., continuously iterating new features within the same project) to observe the decay rate of consistency.
7. Engineering Integration: Teams needing to incorporate this into production workflows can deploy the aforementioned use cases on a cloud container platform for continuous regression, utilizing logging services and monitoring alerts to track output drift caused by model version changes.

8. Conclusion

The first impression determines whether someone wants to keep watching; the hardcore engineering backend determines whether you dare to put it into a production environment. For model evaluation and selection, the former is merely an entry ticket, while the latter is the final answer.