The Next Opportunity for AI Researchers: Native 3D Capabilities in Large Models

3D creation is emerging as a new testing ground for frontier model capabilities: building a room from a single sentence, driving a mechanical assembly with a piece of code, or directly generating an interactive 3D webpage—these have all moved from speculation to routine demo material. As “demos” evolve toward “real creation,” a key fact surfaces: language models can orchestrate pipelines, but they still cannot build good meshes. This gap defines where the next generation of research opportunities lies.

I. From Conversation to Creation: A New Axis for Capability Evaluation

In the past, evaluating a large model primarily meant looking at text reasoning, code generation, and tool use. Today, 3D creation is becoming a new litmus test. The reasons are straightforward:

• Coding ability has crossed the practicality threshold. Models can now reliably translate natural language into runnable scene code, leveraging general-purpose 3D rendering to build rooms, mechanical assemblies, and even fully interactive pages.
• Tool invocation has matured. Workflows that once required shuttling between multiple professional applications—modeling, texturing, assembly, rendering, export—are being chained into a single natural-language-driven pipeline.
• Output is inherently shareable. The visual impact of “generate an interactive 3D page from one sentence” far exceeds that of a text response, making 3D scenes the preferred showcase for teams demonstrating their capabilities.

Compared with text, 3D is a more demanding comprehensive test: it requires the model to handle semantics, geometry, and spatial relationships simultaneously, and any weak link is immediately exposed on screen.

II. The Real State of Play: Can Build Scenes, Can’t Sculpt Faces

When creation gets serious, the questions become specific: Can a character stay close to a reference image? Can the parts of a mechanical assembly move independently? Can generated assets go directly into a game engine? If the first version isn’t satisfactory, can you modify just a local region without starting over?

Recently, a mainstream large model team candidly acknowledged in their official tutorial that for tasks requiring fine mesh modeling—characters, animals, statues, monsters, and complex props—large language models “still don’t do very well for now.” To ensure a reasonable experience, when users initiate such tasks, the model automatically organizes prompts and reference images, then routes the work to a dedicated 3D generation model.

This disclosure is highly valuable because it reveals the industry’s true waterline: what language models excel at is “organizing the process”—understanding requirements, decomposing tasks, generating code, and scheduling tools. For mesh quality itself, dedicated geometry models remain irreplaceable. This division of labor is not a stopgap measure; it is very likely a system形态 that will persist long-term.

III. The Current Compromise Architecture: Orchestrator and Executor

The industry widely adopts a layered architecture (orchestrator–executor pattern):

User Intent
   │
   ▼
Large Language Model (Orchestration Layer)
   ├─ Parse requirements, decompose subtasks
   ├─ Determine task type: scene building or fine mesh
   ├─ Organize prompts, select reference images
   ├─ Dispatch to execution layer:
   │    ├─ Scene/mechanical → generate code + general 3D rendering
   │    ├─ Character/statue → dedicated 3D generation model
   │    └─ Material/animation/export → standard format conversion & toolchain
   └─ Aggregate results, validate, and support iteration

Pseudocode illustration:

def create_3d_asset(request):
    plan = llm.parse(request)               # 1. Understand requirements, determine task type
    if plan.needs_mesh_modeling:            # Fine modeling for characters, statues, etc.
        prompt, refs = llm.prepare(plan)    # 2. Automatically organize prompts and reference images
        asset = mesh_model.generate(prompt, refs)  # 3. Hand off to dedicated 3D model
    else:                                   # Scene tasks like rooms, mechanical assemblies
        asset = llm.gen_scene_code(plan)    # 2'. Directly generate code to drive the scene
    return validate_and_export(asset)       # 4. Validate topology/UV, export standard format

This architecture delivers quick results, but its ceiling is equally clear: the orchestration layer cannot exceed the capability limit of the execution layer, and the language model lacks deep control over mesh details. In one sentence—it can delegate work on your behalf, but it cannot delegate work it doesn’t know how to do itself.

IV. Why Mesh Modeling Is So Hard: Four Deep Bottlenecks

1. The representation gap. Language is a discrete sequence of symbols, while a mesh is high-dimensional, unstructured geometric data: vertex coordinates are continuous, topological connectivity is irregular, and edge flow must serve downstream rigging and animation. Even if geometry is discretized into tokens for autoregressive generation, having a text model directly “write out” a high-quality mesh is still equivalent to generating tens of thousands of lines of mutually coupled “coordinate code” in one shot—any local error corrupts the whole.

2. Dual scarcity of data and evaluation. High-quality, commercially usable mesh assets with complete topology and UV layouts are far smaller in scale than text and image corpora, and face higher copyright and format barriers. Evaluation is equally immature—“looks similar” can be judged subjectively, but whether edge flow is even or topology is riggable is extremely difficult to measure automatically.

3. Quality standards are industrial-grade. Game and film pipelines impose hard requirements on meshes: predominantly quad faces, even edge flow, low-poly and high-poly with normal baking compatibility, reasonable UV unwrapping, and direct riggability. A visually “acceptable” mesh that cannot enter the engine scores zero for professional users—this is the fundamental reason general-purpose generation results struggle to enter production pipelines directly.

4. Missing local editing and part-level semantics. “Change only the sleeve, leave the body untouched” requires the model to understand part-level semantics of a mesh. Most existing approaches are essentially “reroll the whole thing,” lacking stable part-level control—precisely the most painful point for professional users during iteration.

V. Where the Next Research Opportunities Are

First, native 3D representations and generation. Implicit fields (neural radiance fields, signed distance fields), point clouds, 3D Gaussians, and native mesh diffusion are all evolving rapidly. The real decisive factor is not “which representation looks better” but compatibility between the representation and industrial pipelines: can it directly produce clean-topology, editable, riggable meshes rather than one-off results requiring manual retopology?

Second, spatial intelligence and world models. Occlusion, support, interpenetration, assembly—giving models stable spatial cognition is the key to moving from “looks right” to “built right.” This shares roots with robotics and embodied intelligence research, and the two lines will converge on a unified spatial representation. The 3D consistency implicitly encoded in video generation models may also become another path toward spatial understanding.

Third, part-level controllable generation and editing. Reference image alignment, local regeneration, and parametric assembly are all fundamentally controllability problems. Whoever can turn generation from a “one-shot deal” into an “iterable engineering process” will hold the entry point to professional workflows.

Fourth, 3D-level alignment and evaluation infrastructure. Text–image alignment is relatively mature, but “text/image to 3D structure” alignment remains an open problem, lacking large-scale paired data and公认 evaluation protocols. Building datasets and automated metrics is classic “grunt work,” and it is currently the scarcest contribution.

VI. Engineering Perspective: How to Ship This Today

For developers and enterprise teams, rather than waiting for a “unified” model, it is better to design systems with a layered architecture now:

• The orchestration layer belongs to the language model. Focus on intent understanding, task decomposition, prompt engineering, and result validation—don’t demand it produce meshes directly.
• Decouple hard geometry. Delegate mesh generation to dedicated models or toolchains, connected through standard 3D exchange formats, avoiding tight coupling to any single execution backend.
• Make the full pipeline observable. Use object storage to uniformly manage reference images and generated assets, container platforms to host rendering and evaluation tasks, and logging and monitoring services to track success rates and latency at each stage—so that “which step failed” is measurable, attributable, and optimizable. On platforms like Baidu AI Cloud, the storage, container, logging, and monitoring capabilities mentioned above fall under general-purpose categories, allowing teams to focus their energy on 3D data flow and orchestration logic itself.
• Write “editability” into acceptance criteria. Require outputs to carry standardized UVs and reasonable grouping from the start of the pipeline; otherwise, downstream iteration costs will grow exponentially.

VII. Conclusion

“The more models can advance complex 3D projects, the more specific the demand for native 3D capabilities becomes.” This statement precisely captures the current moment: language models have connected the pipeline, which in turn has thrown the geometry core gap into sharp relief. For researchers, this is an ideal “next hill”: the problem is clearly defined, industrial demand is urgent, and existing solutions are visibly insufficient. The next opportunity is not in larger parameter counts, but in giving models a genuine “sense of space”—whoever can turn topology, parts, and assembly into a model’s native language will hold the key to professional creative workflows.