From Prompt to City: 3D Content Generation Practices with AI Agents

1. From Conversational to Capable: The Shift in Ability Boundaries

Recently, real-world tests of next-generation large language models (LLMs) published by early developers have caused quite a stir in the tech community. These tests share a common characteristic: almost no one is using the models for simple Q&A. Instead, they are connecting them to software, handing them tools, and assigning them long-running tasks. Representative scenarios include:

• Within minutes, a developer used natural language to instruct a model to replicate a classic interactive electronic device in 3D software. The model included menus, scroll wheels, and UI interfaces, with the internal structure carrying the developer’s task threads and project data.
• A user provided a set of property photos, and the model reconstructed the 3D structure of the entire house, filled in interior furnishings and surrounding environments, and conveniently generated a promotional video.
• Another demonstration showed the model “constructing” buildings along a street one by one, completely building out a city-scale scene including central green spaces and surrounding building outlines.

Setting specific products aside, these cases point to a shared trend: the capability focus of large models is shifting from “language understanding” to “long-horizon task execution.” When spatial understanding, tool calling, and autonomous planning overlap, 3D content production—a field long constrained by professional software barriers and high labor costs—becomes the most direct beneficiary.

2. Technical Pipeline: How a 3D Generation Task is Completed

To break down “building a house with a single sentence,” we can abstract a universal agent execution pipeline:

1. Intent Parsing and Task Decomposition: The model translates natural language instructions into a structured list of steps. For example, “build a seaside villa” is decomposed into subtasks like terrain, main structure, courtyard, materials, lighting, and camera, along with their execution order and dependencies.
2. Scene Graph Construction: Before formal modeling, an intermediate representation—the Scene Graph—is maintained. It records objects, attributes (geometry, material, color), and spatial relationships (adjacency, containment, occlusion). This is the foundation for subsequent incremental modifications and consistency checks.
3. Tool Calling and Parametric Modeling: Instead of manipulating meshes vertex by vertex, the model generates scripts or parameters to drive the modeling interfaces of 3D software. This ensures that the generated results can be further edited in professional software and allows for human takeover.
4. Rendering Self-Check and Visual Reflection: After completing each stage, a snapshot is rendered and passed back to the model for visual self-inspection—checking for issues like mesh intersections or scale distortions. If deviations are found, it enters a repair loop.
5. Delivery and Post-Processing: Model assets are exported, rendered into final images, or composited into video, completing the last mile from “generation” to “usable.”

A minimal viable execution framework can be expressed in pseudo-code:

# Pseudo-code: Minimal execution framework for a 3D generation agent
def run(prompt):
    plan = llm.decompose(prompt)            # 1. Task decomposition
    scene = SceneGraph()                    # 2. Scene graph intermediate representation
    for step in plan:
        if step.type == "geometry":
            scene.add(llm.gen_geometry(step))      # Generate modeling script and execute
        elif step.type == "material":
            scene.apply(llm.gen_material(step))
        elif step.type == "layout":
            scene.place(llm.gen_layout(step, scene))
        snapshot = renderer.render(scene)          # 3. Render snapshot
        issues = llm.review(snapshot, step.check)  # 4. Visual self-reflection
        if issues:
            scene.patch(llm.fix(issues))           # 5. Iterative patching
    return scene.export()                   # Export model / rendered output

3. Core Capabilities Deconstructed: Where the Progress Lies

1. Spatial Semantic Understanding: From “Shapes” to “Lived-in Spaces”
Early model-generated scenes often only had “showroom” level skeletons—walls were walls, tables were tables, but lacked details. In recent tests, sofa wrinkles, desktop cups, poolside ornaments, and distant vegetation are all included in the default generation scope. This indicates a significant enhancement in the model’s common-sense understanding of “what a real space should contain,” moving scene completion from “geometrically correct” to “semantically reasonable.”

2. Parametric Modeling and Procedural Generation: Key to City-Scale Scenes
Modeling a single building manually takes days, yet the model in the demonstration could proceed building by building along a street to complete an entire block. This relies on Procedural Generation: using rules and parameters to mass-produce stylistically consistent architectural variants, then incrementally adding street facilities and surface details. Once the rules are established, scaling is merely a matter of computation.

3. Tool Calling and Long-Horizon Task Management
Looking at typical community usage, people are essentially treating the model as an “intern who knows how to use professional software”: granting it software operation permissions, registering custom tools, and assigning tasks that last for hours. This requires the model to possess agent infrastructure capabilities like state maintenance, failure retries, and context memory, rather than just single-turn generation.

4. Multimodal Input and Reconstruction
“Photo to 3D” tests multi-view consistency understanding and generative completion of invisible areas. In tests, a usable output was generated in a single pass (though the author admitted details were still distorted), indicating that the “reconstruction + completion” combined pipeline is already viable for engineering exploration.

4. Typical Application Scenarios and Practical Value

• End-to-End Real Estate Visualization: Property photo reconstruction, interior furnishing completion, and promotional video synthesis. Work that originally requires days of collaboration among modelers, renderers, and editors is compressed into an automated pipeline.
• Virtual Production and Film Previsualization: Script storyboards are directly converted into 3D scene previews. Directors and teams can confirm spatial relationships and camera language before shooting, significantly reducing pre-production communication costs.
• City-Scale Digital Twins: Batch generating buildings and blocks along road networks to serve planning simulations, game world building, and virtual scene operations.
• Rapid Interactive Prototyping: Replicating classic device interfaces and generating mini-game prototypes validates the complete pipeline from “natural language” to “interactive program,” which is highly valuable for rapid trial-and-error during the product prototyping phase.

5. Engineering Challenges and Countermeasures

1. Precision and Realism Gap: Single-pass generation results generally suffer from detail distortion. A “generation-validation-rework” closed loop must be established, and manual sampling for key projects is indispensable.
2. Compute and Rendering Costs: Modeling, rendering, and video synthesis are compute-intensive. In practice, this is typically handled via elastic scaling of cloud GPU compute, combined with object storage for hosting model assets, message queues to decouple rendering tasks, and monitoring/alerting to ensure pipeline stability. Cloud platforms like Baidu AI Cloud offer mature, combinable general infrastructure capabilities for these requirements.
3. Lack of Quality Evaluation Standards: There are no unified quantitative metrics for 3D quality. It is recommended to establish a tiered acceptance checklist based on geometry, materials, layout, and interaction, releasing the project layer by layer.
4. Data and Copyright Compliance: The boundaries of using real architectural data and reference materials, as well as the copyright ownership of generated assets, must be clarified early in the workflow.

6. Practical Advice for Developers

• Have the model generate parametric scripts rather than directly outputting meshes to ensure the results are editable and versionable.
• Establish a scene graph intermediate representation to cache geometry, materials, and lighting in layers, reducing re-rendering costs.
• Make rendering snapshot feedback a standard step for visual self-checking to achieve quality convergence at minimal cost.
• Adopt an incremental asset management strategy to ensure that every rework is rollbackable and comparable.

7. Conclusion

From individual models to entire cities, the paradigm of 3D content production is being rewritten. For developers, what is truly worth noting is not the stunning effects of a single demo, but the reusable engineering paradigm behind it: task decomposition, scene representation, tool protocols, and validation loops. As spatial intelligence and tool calling mature further, “building worlds with language” may become as commonplace as writing a document today. Teams that understand this pipeline early will gain a head start in the next round of content production efficiency competitions.