How to Evaluate LLMs for Code Generation and 3D Scene Building
As large language models move from answering questions to executing tasks, this article lays out a reproducible way to evaluate their code-generation and 3D-scene-building abilities. It covers how to fix test metadata, define layered metrics (functional correctness, interaction completeness, visual completeness, aesthetic quality, engineering usability), and close the loop with runnable code rather than pseudocode. Two hands-on tests—an interactive vector animation and a code-only lighting scene—show how a preview model performs and where it falls short, backed by parameter-level defect tables, a 1–5 aesthetic scoring rubric, and enterprise pipeline guidance including isolation, failure thresholds, and version regression. It offers a practical framework for teams that need to measure capability, verify stability, and judge final user experience.
1. The Evaluation Focus Is Shifting: From “Answering” to “Executing”
In the past, general-purpose models were judged mainly on text comprehension, answer accuracy, and language quality. At the current stage, a model’s value is shifting from “giving an answer” to “completing a task.” In scenarios such as code generation, 3D scene construction, and agent orchestration, a model must not only produce correct output but also deliver work that is runnable, interactive, reusable, and open to aesthetic judgment. This is the starting point for asking how to evaluate an LLM’s code and 3D capabilities.
One boundary worth stating upfront: the conclusions here are based on specific tasks and specific model versions. Readers should not extrapolate a single case’s performance to all models. Different tasks, models, and versions all require fresh evaluation. Compared with traditional multiple-choice benchmarks, a more sensible approach is to design evaluation around real engineering tasks—have the model generate code, run that code in an environment, and subject the result to both functional and aesthetic checks.
2. A Reproducible Evaluation Method: Metadata, Layered Metrics, and a Closed Execution Loop
Judging whether a model is production-ready cannot rely on a single screenshot. You need a mechanism that is reproducible, comparable, and regression-friendly.
Step 1: Fix the test metadata. Every evaluation should record the model identifier, invocation parameters, runtime/render environment, and acceptance criteria. Below is how one evaluation’s record table is filled in (sample values; actual runs should reflect real execution):
| Field | Example |
|---|---|
| Model identifier | ernie-5.5-flash-preview (preview) |
| Invocation parameters | temperature=0.7, top_p=0.8, max_tokens=8192 |
| Render/runtime environment | Chromium 124, WebGL2, Ubuntu 22.04, 8-core/16GB container |
| Full prompt | See the original prompt text for each task |
| Acceptance criteria | Renders successfully, all controls present, runs offline, no external resources |
| Evidence retained | First-screen screenshot, console logs, interaction recording, HTML artifact |
Step 2: Define layered metrics:
| Metric layer | Focus | Typical question |
|---|---|---|
| Functional correctness | Does it run | Does the code error out, does the page open |
| Interaction completeness | Are all controls present | Are pause, speed, and switching usable |
| Visual completeness | Is the scene formed | Are structure, lighting, and materials clear |
| Aesthetic quality | Does it look natural and good | Light-dark transitions, color layering, detail texture |
| Engineering usability | Is it easy to maintain | Is it modular, does it depend on too many resources |
Step 3: Close the loop with runnable code, not pseudocode. The example below can run in an offline container with Python 3.11+ and headless Chromium:
from evalharness import run_casecases = [{"name": "vector_animation","prompt": "Generate a standalone web vector animation: a pelican cycling along the coastline, with pause/speed/day-night toggle, no external images","checks": ["render_ok", "controls_complete", "offline"]},{"name": "light_scene","prompt": "Using code only, build an afternoon-lit room: window-pane shadows, table and chairs, potted plants, floor reflections","checks": ["geometry_ok", "light_dir", "material_roughness", "reflection"]},]model = {"id": "ernie-5.5-flash-preview","params": {"temperature": 0.7, "max_tokens": 8192}}env = {"browser": "Chromium 124", "renderer": "WebGL2", "offline": True}for case in cases:artifact = run_case(model=model, env=env, case=case)report = artifact.report # returns passed / failed_checks / logs / screenshotsprint(case["name"], report["passed"], report["failed_checks"])
Field meanings: name is the task name, prompt is the full original prompt text, and checks are the acceptance items. The report returns passed items, failed items, logs, and screenshots. Functional items are judged automatically by the script; aesthetic items go to human review.
3. Hands-On Test 1: Interactive Vector Animation Tests Code Organization
The task was to generate a vector animation whose subject is a pelican cycling along the coast. The full prompt: “Please generate a standalone web-style vector animation whose subject is a pelican cycling by the sea. It must include pause, speed adjustment, and day-night switching, must not use external image assets, and should have a clear, maintainable code structure.”
Measured against the metadata table from Section 2, the model showed strong “task reconstruction” ability here. The generated scene included a character, road, sea surface, sky, and a fluttering scarf; the animation was continuous; the pause, speed, and day-night controls were largely complete; it ran offline; and the artifact was relatively self-contained, making deployment and further modification easy.
This capability can be summarized in three points: structured generation (turning natural language into executable code), interaction logic (pause/speed/day-night all involve state management), and runnability (not pseudocode—it renders and interacts for real). Potential shortcomings include: whether complex physical motion looks natural, whether state stays stable under high-frequency interaction, how rich the visual detail is, and whether the code is modular. Such tasks are therefore better suited to a “basic usability” evaluation than an “extreme aesthetics” one.
4. Hands-On Test 2: A Code-Only Lighting Scene Tests 3D and Aesthetics
This task required building a room with an afternoon-lit atmosphere using code only, no external images. The difficulty concentrated in three areas: geometric structure (whether the spatial relationships between objects are reasonable), lighting simulation (direction, intensity, attenuation, shadows, reflections), and aesthetic judgment (softness of shadows, tabletop reflections, light-dark gradient on walls, and the translucency of plants).
There are several implementation routes: a real-time graphics route emphasizes interactivity and runnability; a procedural geometry route generates the room and light paths; an offline physically based render pursues realism at high cost; and a hybrid route trades off performance and image quality.
Replace vague “worse than frontier approaches” with parameter-level defect localization. When recording this preview model’s output, annotate problems parameter by parameter rather than offering only a qualitative comparison:
| Defect | Verifiable metric | Observation this time |
|---|---|---|
| Shadow sampling | shadow map resolution / edge aliasing | visible aliasing on wall-corner edges at 2048 |
| Material roughness | whether roughness varies by region | uniform and single-valued; wood/stone/tile hard to distinguish |
| Reflection intensity | reflection coefficient range | tabletop reflection weak, close to 0 |
| Contact shadow | missing AO at wall-floor junction | AO missing, wall corners look floaty |
| Global illumination consistency | highlight overflow / crushed blacks | shadows lean toward crushed black, harsh transitions |
Conclusion: the model can now “build a scene with code,” but there is still clear room for improvement in physical realism, material detail, and aesthetic atmosphere. These parameters can serve directly as quantitative handles for iteration and regression.
5. Evaluating 3D Capability: Geometry, Materials, Lighting, and Aesthetics
Judging 3D generation by screenshots alone invites subjective bias, so an interpretable framework is needed: geometric completeness (position/perspective/proportion/viewpoint), materials and surface texture (wood, stone, and tile distinguishable; metal, glass, and fabric identifiable), lighting plausibility (consistent main-light direction, shadows that respect occlusion, layered light and dark, no overexposure or crushed blacks), and aesthetic expression.
Aesthetic expression needs a clear standard to avoid “looks fine” conclusions. A recommended approach uses 1–5 anchors, scored independently by 3 reviewers, aggregated by median:
| Score | Anchor description |
|---|---|
| 1 | Structure missing or clearly wrong; lighting/materials basically do not hold up |
| 2 | Structure discernible but crude; harsh light and dark; materials confused |
| 3 | Structure correct and light direction discernible, but transitions harsh and detail lacking |
| 4 | Reasonable structure, natural lighting, distinguishable materials, fairly good detail |
| 5 | Close to photorealistic; soft light and dark, realistic materials, unified atmosphere |
6. Why Aesthetics Fall Short: Insufficient Constraints
Preview models often produce output that is “structurally correct and runnable but not refined aesthetically,” for three reasons. First, models are good at satisfying explicit constraints (“has a window,” “has light”) but weak at implicit aesthetics (“a quiet afternoon,” “interlaced light and shadow”). Second, there is a lack of continuous verification of physical realism—high-quality rendering requires repeated correction of light sources, materials, and shadows. Third, visual detail depends on compositional reasoning (the softness of window-pane shadows, the position of tabletop highlights, the translucency of plants, and so on).
Turning abstract goals into verifiable constraints is the key to narrowing the gap: “a quiet afternoon” can be decomposed into a warm color temperature (5000–6500K), moderate contrast, and no highlight overflow; “interlaced light and shadow” can be decomposed into a fixed main-light azimuth/elevation, shadow softness (penumbra width), and a reflection-ratio range.
7. Engineering Implementation: An Enterprise Evaluation Pipeline
A unified task set has three layers: basic tasks (web pages, buttons, forms), intermediate tasks (interactive animation, data cards, simple 3D), and advanced tasks (complex lighting, agent workflows, multi-module applications). Each task should specify inputs, constraints, acceptance criteria, and failure examples.
Isolated runtime environments: model-generated code carries security risk and must run offline inside containers, with restricted network access, file writes, and compute. Results are archived to object storage, scheduling uses a message queue, and state is written to logs with monitoring and alerting. Example failure thresholds: load timeout > 10s, errors after ≥10 consecutive switches, memory > 1.5GB or frame rate < 24fps, or any external URL appearing (offline requirement)—any of these means failure.
Human aesthetic review: for visual tasks, use a unified scoring sheet (structural completeness / lighting plausibility / material realism / color atmosphere / detail completeness, 20 points each), scored independently by 2 reviewers and averaged to reduce individual variance. Version regression mechanism: after any change to the model, prompt, or render environment, rerun historical tasks and compare pass rate, aesthetic scores, and stability to prevent “local improvement, overall regression.”
8. Implications for Model Development
Strongly constrained prompts improve stability. Multi-turn iteration is more reliable than a single generation; it is recommended to record the changes, failure reasons, and regression results for each round:
| Round | Change | Failure reason | Regression result |
|---|---|---|---|
| R1 | Added window-pane shadows | Shadow direction inconsistent with main light | Passed |
| R2 | Softened the black line at wall corners | Missing contact-shadow AO | To be verified |
Evaluation should not test only “one-shot generation” but also “multi-round correction.” Aesthetics are a core capability of visual tasks and should be built into the evaluation, training, and feedback loop, not treated as an afterthought.
9. Conclusions and Selection Guidance
The new generation of preview models already has decent foundational ability in code generation and 3D scene building. They can turn natural-language requirements into runnable code, produce interactive animations, and attempt lighting scenes—moving from “assisted writing” toward “assisted construction.” But for high-quality lighting images, while they can build the structure, there are still gaps in light-dark transitions, material detail, physical consistency, and aesthetic atmosphere: they can “make it,” but are not yet at “make it well, make it beautiful, make it stable.”
For enterprise selection, what matters is not whether a single generation is impressive, but whether a long-term mechanism can be established: use a task set to measure capability, use an automated pipeline (with failure thresholds) to verify stability, use two-reviewer back-to-back aesthetic scoring to judge the final experience, and use a multi-turn feedback template to drive iteration. For LLMs to truly enter production, stronger generation ability is needed, but so are more rigorous verification, a more stable engineering pipeline, and a more mature aesthetic evaluation system—only then can code generation, 3D construction, and agent applications move from “seemingly feasible” to “truly usable.”
