How to Evaluate Large Language Models' 3D Creation and Spatial Reasoning Capabilities
As large language models move beyond code generation, 3D creation has emerged as a demanding new evaluation dimension that tests geometric understanding, spatial reasoning, aesthetic judgment, and engineering-grade delivery. This article lays out a practical methodology for assessing a model's native 3D modeling, spatial understanding, and engineering output: how to design zero-shot bare prompts, how to score the three common deliverable types, how to verify spatial fidelity against a reproducible element checklist, and how to quantify results with a weighted four-dimension scorecard. It closes with an end-to-end evaluation workflow and a repeatable archiving approach.
Why 3D Creation Has Become a New Focus for LLM Evaluation
Historically, the mainstream way to evaluate large language models was code generation: whether they can write correct algorithms, debug errors, and refactor functions. Code has a single correct answer, so the evaluation boundary is relatively clear. 3D creation opens up a more complex dimension that is closer to real productivity: it requires a model to combine geometric understanding, spatial reasoning, aesthetic judgment, and engineering-grade delivery at once.
These tasks have no single solution. Given the same theme, different models produce vastly different approaches, allowing evaluators to score from multiple angles and more fully expose a model’s true capability across the “understand–plan–execute” chain. Spatial reasoning is especially critical: whether a model can correctly handle relationships such as “the second floor sits above the ground floor,” “the staircase connects the two floors,” or “the storefront sign sits in the most prominent spot on the building’s front” directly determines whether the deliverable is usable.
Intended audience and scope: This article targets developers and researchers who evaluate LLM-based 3D generation, focusing on modeling fidelity, spatial understanding, and engineering delivery. It does not cover final art-quality acceptance, product-usability acceptance, or commercial-grade asset review—those require professional art and product teams.
How to Design Bare Prompts That Reveal Native Capability
Scientific evaluation begins with controlling variables. Two choices are key here:
- Whether to attach skills or external tools. Once a dedicated skill or toolchain is attached, model capability is greatly enhanced, but what you measure is the joint “model + tools” performance. To assess native capability, use bare (zero-shot) prompts without any auxiliary capabilities.
- The information density of the prompt. Too sparse, and the model tends to miss key elements; too detailed, and the creative space is compressed into “fill-in-the-blank.” A fairer approach is to provide enough on theme, style, and deliverable requirements, and leave the specific details for the model to reason about on its own.
Here is a reusable bare-prompt template with configurable parameters:
Create a miniature voxel-style scene with the theme "Detective Agency."[Configurable parameters]- File format: .blend / .glb / .obj (choose one)- Units: meters (scene scale = 1:100)- Object naming: group prefixes, e.g. ARCH_ (architecture), FURN_ (furniture), CHAR_ (characters)- Polygon budget: <= 200k triangles- Camera angle: perspective 35 degrees, front 3/4 view[Content requirements]1. Use a voxel-block language to render the main building, the surrounding street, and characters;2. Reflect a multi-story structure (ground-floor shop, second-floor office space, an external staircase connecting the two);3. Deliver an editable 3D engineering file, plus a rendered image.
The template deliberately omits details such as “put a sign on the wall” or “place characters at the entrance,” precisely to test whether the model can fill in key visual elements through spatial reasoning.
Counter-example (an overly detailed prompt to avoid): “Place a brown wooden desk by the window on the right side of the second floor, with a green desk lamp and three folders on it; make the doormat dark red; write ‘XXX Detective Agency’ on the sign…” This kind of prompt pins down every element, degrading the generation task into fill-in-the-blank—it can neither test spatial reasoning nor compare differences between models.
How to Score Deliverables: Three Deliverable Types and Common Failure Cases
Deliverables from 3D creation tasks usually fall into three categories. Each should have its own criteria rather than a vague “looks good.”
1. Editable engineering file. Focus on: whether the object hierarchy is reasonable (clear grouping of building, furniture, and characters), whether materials and colors are named consistently, whether the mesh can be edited further, and whether the overall topology is sound. Common failure cases: render-only, non-editable (the “engineering file” is actually a baked static mesh) → score 1–2 on the engineering dimension; messy material naming (e.g., Material.001, cube.003) → deduct 0.5 per issue category; incorrect staircase topology (interpenetrating steps, flipped normals, non-manifold edges) → treated as both an engineering and a spatial problem. Judgment guidance: if the file is not editable, fail the engineering dimension outright and trigger a retry.
2. Rendered image. Assess composition, lighting, camera angle, and material expression. A good render should have a prominent theme, clear layering, and volumetric light and shadow—indirectly reflecting the model’s understanding of the “final presentation” rather than merely assembling geometry.
3. Interactive web page (bonus). Some models additionally produce an interactive 3D showcase page. This kind of “delivered feature that the user never explicitly asked for” is an important signal of proactivity; at the same time, watch for over-engineering—does the added feature genuinely improve the experience, or does it merely add complexity? Its evaluation focus: controllability (whether view/zoom/layer switching works) and loading stability (first-screen load, whether the model is missing).
How to Verify Spatial Fidelity: A Reproducible Case and Element Checklist
The most direct test is whether “the elements that should be present are all present.” To ensure reproducibility and verifiability, fix a reference-material checklist before evaluation to serve as a scoring anchor.
Reproducible reference material (self-built by the evaluator, not a specific artwork): Use a descriptive text as the “standard-answer material,” made public and frozen before evaluation. An example key-element checklist:
| ID | Key element | Position constraint | Basis for judgment |
|---|---|---|---|
| E1 | Side-mounted external staircase | Connects the ground floor and second floor | Continuous steps, reachable to the second-floor landing |
| E2 | Second-floor office space | Located above the ground floor | Has an independent floor with doors and windows |
| E3 | Ground-floor café | At the building’s ground floor | Has a storefront, sign, or display window |
| E4 | Character at the entrance | In front of the building entrance | At least one voxel character |
| E5 | Agency sign | Most prominent spot on the building’s front | Front-facing, visible, theme-relevant |
Evaluators should confirm fidelity item by item (in place / partially in place / missing) and archive the checklist together with the results.
Key visual anchors matter more than generic structure. If a model reproduces most of the building structure but omits the most recognizable core symbol (such as the theme sign), it shows the model can handle common spatial relationships like “generic building structure” but tends to overlook “the core symbol of a specific theme”—revealing a weakness in “extracting key elements from a theme.”
Proactive completion of off-script content (e.g., replacing the sign with another reasonable decoration, or adding street details) can be either a strength or a deviation, so evaluators should distinguish: does the addition serve the theme, does it break spatial plausibility, or is it pure fabrication (hallucination)?
Evaluation focus by deliverable: for printable models, check whether the mesh is closed (watertight) and whether wall thickness is reasonable; for interactive web pages, check controllability and loading stability; for static engineering files, check hierarchy and topology.
How to Quantify Scoring with a Scorecard
To make scoring quantifiable, use a weighted four-dimension scorecard. Score each dimension on a 1–5 anchor scale, then take the weighted sum for a total; a total ≥ 3.0 is the pass line.
| Dimension | Weight | 1 point | 3 points | 5 points |
|---|---|---|---|---|
| Engineering file | 35% | Baked mesh only, non-editable, messy naming | Editable, grouping basically reasonable, naming acceptable | Clear hierarchy, standard naming, sound topology, further-editable |
| Spatial fidelity | 30% | More than half of the key elements missing | Main structure in place, some anchors missing | Checklist complete, includes the theme’s core symbol |
| Rendering | 20% | Chaotic composition, missing lighting | Theme readable, lighting average | Prominent theme, clear layering, volumetric light and shadow |
| Proactivity / interactive completion | 15% | No extra completion, or severe over-engineering | Reasonable completion | Completion serves the theme, interaction controllable and stable |
Scoring anchor notes: 2 and 4 points are intermediate states between adjacent tiers. The pass line is set at a weighted total ≥ 3.0; if the engineering-file dimension scores 1, fail immediately and trigger a retry.
Evaluation Workflow and Summary
Chaining the steps together yields a practical evaluation loop:
- Set the proposition and deliverable type (3D modeling, interactive web page, printable model—choose one or combine), and freeze the reference-material checklist.
- Launch the first generation with the parameterized bare prompt, and record the original prompt for reproducibility.
- Score with the four-dimension scorecard: engineering file, rendering, spatial fidelity, and proactivity/interactive completion.
- Check against the key-element checklist: confirm fidelity item by item.
- Iterate with follow-up prompts: add requirements for missing items (e.g., “Please add the agency sign at the most prominent spot on the building’s front”) and observe the model’s multi-turn correction ability. This iterative capability is often closer to real development scenarios than one-shot generation.
One-line conclusion card: bare prompts test native ability, checklists verify fidelity, multi-turn iteration tests correction.
3D creation capability is becoming an important new dimension for evaluating large language models, following code ability. It comprehensively tests spatial reasoning, geometric understanding, engineering delivery, and proactive completion—difficult to evaluate but information-rich. When putting such evaluations into practice, it is advisable to build a stable environment backed by a reproducible compute and archiving setup, and to uniformly archive prompts, deliverables, and scoring results into a continuously improvable evaluation asset. Grasp these three points, and you can move beyond vague judgments like “looks fine” and reach technical conclusions that genuinely guide decisions.
