Building Interactive 3D Digital Exhibitions with LLM-Driven Development: A Practical Guide to Workflows and Pitfalls

Introduction: From AI-Assisted Coding to Creative Engineering

As large language models become increasingly capable of generating production-quality code, the developer’s role is shifting from writing every line to orchestrating architecture and delegating tasks. However, in a comprehensive project involving 3D rendering, complex user interactions, and multi-modal generation, relying solely on a single prompt rarely yields a deliverable result. This article uses a case study—an interactive 3D website that recreates the creation of a classic sculpture—to examine how LLMs can be applied across the entire development pipeline, from concept narrative and visual design to final deployment. It also addresses the engineering approaches needed when tasks are interrupted or truncated.

Requirements Abstraction and Task Decomposition

The core goal of this project was not simply to display a 3D model. Instead, the experience needed to communicate the artistic concept of subtractive sculpture through direct user interaction. To achieve this, the high-level goal was broken down into the following technical subtasks:

1. Narrative and Visual Design: Construct an immersive darkroom environment that uses lighting and textual cues—such as “Marble has no undo button”—to guide users toward understanding the irreversibility of carving.
2. 3D Data Processing and Rendering: Integrate real, high-precision 3D scanned model data and ensure smooth loading and rendering performance in the browser.
3. Interactive Logic: Allow users to simulate the chiseling process through a specific action, such as holding down a button, and observe the dynamic transition as stone layers are removed to reveal the sculpture.
4. Multi-Modal Asset Generation: Beyond the interactive website, generate a 9:16 promotional video that includes controlled camera movements, pacing, and synthesized audio.

Core 3D Interaction: Technical Approaches for Subtractive Rendering

To achieve the visual effect of stone peeling away layer by layer to reveal the sculpture, the choice of technical approach is critical. Performing real-time boolean operations directly in the front-end is extremely performance-intensive and usually leads to page stuttering. The following two engineering practices are more efficient:

Approach 1: Precomputed Level-of-Detail (LOD) Model Switching

Preprocess the original scanned data in 3D modeling software or through backend scripts to generate multiple model files representing different stages of the carving process—for instance, from the raw block to the final sculpture across 10 distinct stages. The front-end then dynamically swaps the Mesh object in the scene based on user interaction progress.

// Pseudo-code example: Model swapping based on interaction progress
const sculptingStages = [
  'raw_block.glb', 
  'stage_1.glb', 
  // ...intermediate stages
  'final_sculpture.glb'
];
let currentProgress = 0;
function onUserChisel() {
  currentProgress = Math.min(currentProgress + 0.1, 1.0);
  const targetIndex = Math.floor(currentProgress * (sculptingStages.length - 1));
  loadModel(sculptingStages[targetIndex]).then(mesh => {
    scene.replaceCurrentMesh(mesh);
  });
}

Approach 2: Dynamic Culling via Vertex Shader

Merge the raw block and the final sculpture into a single mesh. Control the visibility of vertices by modifying their transparency or positional attributes. When the user interacts, adjust a Uniform variable in the shader. The GPU’s parallel processing power then executes an extremely smooth peeling animation, avoiding frequent CPU-GPU memory communication.

LLM Scheduling Strategies and Fault Tolerance

In practice, orchestrating LLMs for large development tasks presents significant challenges. A complete interactive project often comprises dozens of files across HTML, CSS, and JavaScript, along with various configuration settings. When handling long-context tasks, models are prone to generating truncated outputs or “forgetting” the early architectural decisions.

1. Phased Execution for Long Tasks

When dealing with complex engineering, expecting the model to output the entire codebase at once is unrealistic. The recommended practice is to adopt a “project folder” concept, allowing the model to progress step-by-step:

• Phase 1: Generate the foundational HTML skeleton and global styles.
• Phase 2: Integrate the 3D rendering library (such as Three.js), initialize the scene, and implement model loading.
• Phase 3: Bind user interaction events and implement the core subtractive logic.
• Phase 4: Integrate narrative text and fine-tune visuals, including intercepting any “undo” button presses.

2. Seamless Handoff Across Tools

When the primary long-context model encounters interruptions due to context window limits or resource scheduling issues, developers need a strategy for quickly switching to a fallback solution. One efficient method is to package the “project folder” structure—which has been partially completed—and switch to a lighter-weight model with faster inference speed and lower cost. By providing this model with the complete directory tree as context, it can seamlessly continue and complete the remaining finishing work.

Throughout this process, object storage serves as a key bridge. Preprocessed 3D models and multi-stage project files can be hosted in a cloud-based object storage service. This ensures that different tools and models can access the latest code and resources via a unified URL path, enabling a smooth and continuous handoff.

Multi-Modal Project Delivery and Best Practices

For comprehensive tasks that include video output, building an automated and streamlined engineering pipeline is particularly valuable.

1. Camera Control and Narrative Sequencing

To produce a video with an exhibition-quality feel, simply recording the screen is insufficient. Developers should control the 3D camera’s motion path programmatically using easing functions to define keyframes, enabling smooth transitions from wide shots to close-ups. Additionally, adjusting ambient and spot light parameters helps highlight the texture of the marble.

2. Video Composition and Cloud-Based Processing

After generating frame-by-frame images or a video without audio, it is often necessary to use a container platform or function computing service on the backend to integrate a multimedia processing tool like FFmpeg for the final composition. Below is a generic command for video synthesis:

# Generic video composition command: Render an image sequence into a 9:16 video with audio
ffmpeg -framerate 30 -i frames/frame_%04d.png \
       -i audio/background_music.mp3 \
       -c:v libx264 -preset slow -crf 18 \
       -c:a aac -b:a 192k \
       -vf "scale=1080:1920:force_original_aspect_ratio=decrease,pad=1080:1920:(ow-iw)/2:(oh-ih)/2" \
       output_final.mp4

At this stage, implementing logging services and monitoring and alerting mechanisms can significantly improve delivery stability. Should a rendering node encounter an out-of-memory error on the GPU or an abnormal frame rate, developers can promptly retrieve the stack trace and initiate a rollback or retry, preventing unforeseen failures at the final delivery stage.

Conclusion

Utilizing large language models to drive complex 3D interactions and multi-modal content generation is moving from conceptual exploration to practical engineering. Developers should focus on three key areas: abstractly understanding the underlying rendering technologies, reasonably decomposing and scheduling long-running model tasks, and effectively leveraging cloud-based infrastructure components such as object storage and container computing. By establishing a robust fault-tolerance and handoff mechanism, it is possible to efficiently deliver digital products that offer deep narratives and a fluid user experience, even under resource-constrained conditions.