Full-Process Practice of LLM-Driven 3D Content Generation

    This article explores the full pipeline of generating 3D display works driven by large language models (LLMs). It covers the background, four main technical routes, overall pipeline design, and detailed steps from requirement parsing and procedural geometry generation to material rendering, front-end export, quality validation, and engineering deployment. Readers will gain a comprehensive understanding of how to build a maintainable, hybrid 3D generation pipeline.

    1. Background

    The generation process of 3D display works typically involves natural language requirements, structured parameters, procedural modeling, and rendering output. By breaking down this pipeline, developers can reproduce similar effects in their own projects and quickly identify which steps are the most time-consuming or error-prone.

    This article discusses general technical approaches rather than specific products. The data structures, function names, and parameters mentioned are for illustrative purposes and should be adjusted based on your chosen modeling library, rendering environment, and business requirements during actual implementation.

    2. Four Technical Routes for 3D Content Generation

    1. Manual Modeling
    Artists create meshes and textures one by one in modeling software. The final quality is the highest, but the labor cost is high, making it difficult to produce in bulk.

    2. Procedural Modeling
    Uses code to describe geometric shapes and parameters, such as using variables to control cylinder radius, array count, and rotation angle. This approach is suitable for scenes with regular structures, such as exhibition booths, data visualization graphics, and product displays.

    3. Learning-Based Generation
    Uses neural networks to generate 3D content based on text or images. The creative space is vast, but the results lack stability and usually require manual screening and correction.

    4. Hybrid Approach
    First uses semantic understanding to convert requirements into structured descriptions, then uses procedural modeling to ensure geometric correctness, and finally uses learning models to supplement textures or decorative details. This is a common choice for mass production.

    Route Controllability Production Speed Applicable Scenarios Main Risks
    Manual Modeling High Slow Premium Assets High labor cost
    Procedural Modeling High Fast Regular Structures Complex shapes are hard to parameterize
    Learning-Based Generation Low to Medium Fast Creative Sketches Unstable results
    Hybrid Approach Medium to High Medium Mass Production Complex pipeline

    3. Overall Pipeline Design

    A maintainable generation pipeline usually consists of six stages:

    Requirement Input -> Semantic Parsing -> Structured Scene Description -> Geometry Generation -> Material & Rendering -> Validation & Export

    The output of each stage should be a checkable intermediate result. By serializing or persisting these intermediate results to disk, you can quickly pinpoint the specific stage where an issue occurs without re-running the entire pipeline. Structured descriptions should use a fixed format (e.g., JSON) to facilitate version control and regression testing.

    4. Step 1: Converting Requirements to Structured Descriptions

    Natural language requirements are often vague, such as “make a tech-style display booth.” Developers need to break this down into objects, positions, sizes, materials, lighting, and camera angles.

    Below is an illustrative scene description:

    1. {
    2. "scene_id": "showcase_001",
    3. "units": "meter",
    4. "axis": "z_up",
    5. "camera": {"position": [4, -6, 3], "target": [0, 0, 0], "fov": 45},
    6. "objects": [
    7. {"id": "base", "type": "box", "size": [4, 4, 0.2], "material": "matte_gray"},
    8. {"id": "core", "type": "sphere", "radius": 1.0,
    9. "position": [0, 0, 1.2], "material": "glossy_blue"}
    10. ],
    11. "lighting": {"type": "directional", "intensity": 1.0,
    12. "direction": [-1, -1, -2]}
    13. }

    There are three key design points for this description.

    First, units and coordinate systems must be fixed. If part of the data uses centimeters and another uses meters, the final model will have size mismatches. Teams should explicitly declare units and axis directions in the description file.

    Second, fields must have white-list constraints. Material names, primitive types, and lighting types should be selected from predefined lists to avoid generating unrenderable values.

    Third, semantic parsing can be accomplished using large language models, but the model output must pass schema validation. If validation fails, the system should fall back to default values or prompt the user for missing information, rather than silently guessing.

    5. Step 2: Generating Geometry with Code

    The core of procedural modeling is mapping each primitive type to a constructor function and executing transforms in sequence.

    1. PRIMITIVE_BUILDERS = {
    2. "box": build_box,
    3. "sphere": build_sphere,
    4. "cylinder": build_cylinder,
    5. }
    6. def build_scene(spec):
    7. validate_spec(spec) # Structure and field validation
    8. scene = create_empty_scene()
    9. for obj in spec["objects"]:
    10. builder = PRIMITIVE_BUILDERS.get(obj["type"])
    11. if builder is None:
    12. raise ValueError(f"Unsupported primitive type: {obj['type']}")
    13. mesh = builder(obj) # Generate base mesh
    14. mesh = apply_transform(mesh, obj.get("position", [0, 0, 0]))
    15. scene.add(obj["id"], mesh)
    16. return scene

    Note the following during implementation:

    • White-list dispatch: Only registered primitive types are allowed; unknown types throw an error directly.
    • Transform order: Typically scale first, then rotate, and finally translate. Reversing this order will cause object position offsets.
    • Topology check: After generation, check whether the mesh is closed and whether there are degenerate faces to prevent errors during subsequent rendering or 3D printing.
    • Face count control: Set a face count limit for each object to prevent parameter anomalies from causing an explosion in mesh count.

    6. Step 3: Materials, Lighting, and Rendering Parameters

    Materials determine the appearance of the model under different lighting. The industry-standard Physically Based Rendering (PBR) material model typically includes the following parameters:

    • Base color: The inherent color of the surface.
    • Metallic: Distinguishes metallic from non-metallic surfaces, ranging from 0 to 1.
    • Roughness: Determines the spread of highlights; the higher the value, the softer the highlight.
    • Normal map: Simulates surface details without increasing the polygon count.

    For lighting, three-point lighting is a common beginner-friendly setup: the key light establishes the main light and shadow relationship, the fill light softens overly dark shadow areas, and the rim light highlights the edges of the object.

    Rendering configurations can be divided into two tiers: preview and final:

    1. preview:
    2. resolution: [960, 540]
    3. samples: 16
    4. denoise: true
    5. final:
    6. resolution: [3840, 2160]
    7. samples: 256
    8. denoise: true

    The preview tier is used for quickly checking composition and materials, while the final tier is for the ultimate output. This approach saves computational resources for the rendering tasks that truly need them, preventing high-quality renders from running on every single modification.

    7. Step 4: Export and Front-end Display

    After geometry and materials are generated, they need to be exported into a universal 3D exchange format and handed over to the web front-end’s 3D rendering capabilities for display.

    The following preprocessing should be completed before export:

    1. Texture compression: Lower texture resolution to reduce transfer size.
    2. Level of Detail (LOD): Prepare low-poly versions for distant objects and high-poly versions for close-up objects.
    3. Bounding box calculation: Pre-calculate bounding boxes for front-end frustum culling and initial camera positioning.
    4. Metadata writing: Write the scene description and version number into the metadata section of the model file for traceability.

    On the front-end, it is recommended to use on-demand loading. Load the low-poly version first to ensure fast first-screen rendering, and then replace it with the high-quality model in the background. For mobile devices, further limit the polygon count and texture size.

    8. Step 5: Quality Validation and Iteration

    Automated validation can intercept most obvious errors. Below is a simplified validation example:

    1. def quality_check(stats, limits):
    2. issues = []
    3. if stats["triangles"] > limits["max_triangles"]:
    4. issues.append("Triangle count exceeds limit")
    5. if stats["bbox_size"] > limits["max_size"]:
    6. issues.append("Bounding box exceeds scene range")
    7. if stats["missing_materials"]:
    8. issues.append(f"Missing materials: {stats['missing_materials']}")
    9. return issues

    In addition to automated validation, a regression mechanism should be established:

    • Fixed random seed: The same description should yield the same result under the same seed, making it easier to compare the impact of changes.
    • Screenshot comparison: After each modification, render screenshots from fixed angles and compare pixel differences against a baseline image.
    • Versioned descriptions: Include scene descriptions in version control so that every render can be traced back to its corresponding input.
    • Candidate retention: For learning-based generation, multiple candidate results can be retained simultaneously, with the best one selected either manually or via a scoring rule.

    9. Engineering Deployment Recommendations

    Since the computational overhead of generation and rendering is high, it is recommended to split them into independent services.

    • Task queue: After the front-end submits a requirement, the queue asynchronously distributes it to generation and rendering worker nodes to prevent request timeouts.
    • Containerized deployment: Rendering nodes are suitable for container platforms, allowing elastic scaling based on load.
    • Caching strategy: Use the hash value of the scene description as the cache key. If the description hasn’t changed, return the existing artifact directly.
    • Artifact archiving: Model files, preview images, and scene descriptions should be archived together, along with generation time and resource consumption.
    • Cost control: Generate low-resolution previews first, and only execute high-quality rendering after user confirmation.

    If deployed on Baidu AI Cloud, model files and preview images can be stored in object storage, rendering tasks can run on the container platform, and failure rates and latency can be continuously tracked through logging and monitoring services.

    10. Risks and Boundaries

    • Copyright and asset authorization: The textures, models, and fonts used in generated content must have their authorization scopes confirmed. Compliance reviews should be completed before commercial use.
    • Result instability: Multiple generations of the same requirement may yield significant differences. Key projects should retain a manual review process and use fixed seeds to reduce volatility.
    • Semantic ambiguity: When requirements are vague, the system should prompt for key parameters (e.g., size, color scheme, and viewing angle) rather than relying on default values.
    • Performance boundaries: Complex scenes may experience lag on the web front-end. In such cases, you should reduce the number of objects and merge meshes.