From Interactive 3D Storytelling to Sculptural Modeling: A Web 3D Practice Using Generative AI

Recently, the capabilities of generative AI in 3D content creation have been widely discussed. Rather than stopping at the question of “can AI generate a 3D rendering,” I wanted to explore something deeper: given a relatively complex and complete creative goal, can AI autonomously drive the development of a runnable, interactive Web 3D project?

To test this, I conducted an experiment. The task was not to have AI output simple code snippets, but to complete a 3D website featuring narrative, interaction, and real model data. The theme was the creation of Michelangelo’s David statue from a block of stone.

While this appears to be an art history topic, it actually encompasses many typical challenges in 3D web development: real model loading, mesh simplification, subtraction operations, material rendering, interactive state management, animation pacing, performance optimization, and final deployment. These are exactly the areas where generative AI tends to fall short in “engineering delivery” scenarios.

1. Task Design: Why Use “Sculpting” to Test 3D Capabilities

Sculpting is essentially a subtractive process. Starting from a complete block of stone, unnecessary parts are gradually removed until the form is revealed. This logic aligns perfectly with 3D Boolean operations: starting with an initial geometry, repeatedly subtracting certain areas to approach the target shape.

Simulating a real sculpting process on the web is impractical. However, we can use an existing 3D scan model of David for a reverse expression: first present a rough block of stone, then let the user “chisel” through interaction to gradually reveal the pre-existing statue inside.

This presents several technical challenges:

1. Model Source and Licensing
   We need a legally usable 3D scan of Michelangelo’s David. Public domain scan data released under the CC BY-SA license is available and suitable for experimental and educational purposes.

2. Model Denoising and Simplification
   Scan models typically contain a massive number of triangles, which will severely impact web performance if loaded directly. Mesh simplification, format conversion, and generating lightweight versions optimized for browser rendering are required.

3. Stone Shell Representation
   We cannot simply wrap the statue in a cube. The outer shell needs a stone texture and must be “peeled off piece by piece.” This requires dividing the stone geometry or implementing dynamic occlusion.

4. Interaction and Narrative Pacing
   Instead of clicking a play button, users participate in the sculpting process by holding buttons or dragging sliders. The interaction must coordinate with text, camera movements, and lighting to create a complete viewing experience.

2. Model Preparation: From Scan Data to Usable Assets

1. Model Format and Mesh Processing

Common scan models use formats like STL, OBJ, or PLY. STL files only contain geometric information without UVs, materials, or normal maps, requiring extra processing.

In practice, the following pipeline is typically followed:

Original Scan Model
    │
    ▼
Mesh Repair and Retopology
    │
    ▼
Mesh Simplification (Decimation)
    │
    ▼
UV Unwrapping and Normal Reconstruction
    │
    ▼
Export as GLB / glTF

For web rendering, glTF/GLB is the optimal choice. It is compact, loads quickly, and is well-supported by major 3D libraries.

2. Stone Shell Generation

One feasible approach is to enclose the David model within a slightly enlarged stone mesh, which is then divided into chunks. Each chunk acts as an independent mesh; when the user triggers “sculpting,” the corresponding chunks are hidden or removed sequentially.

The advantage of this method is its low computational cost. Instead of performing real-time Boolean operations, the subtraction process is pre-decomposed into discrete chunks. The downside is that chunk boundaries may not look natural, requiring particles, debris, or dynamic lighting changes to enhance realism.

Another approach is using clipping planes or shader masks. By determining the relative position of pixels to the statue’s surface in the fragment shader, the “stone shell” fragments are dynamically discarded. This method provides smoother visuals but demands more from the GPU and increases implementation complexity.

3. Post-Simplification Performance Targets

Both mobile and desktop platforms must be considered. Typical targets include:

• Triangle count: No more than 500,000 for the main statue, and no more than 300,000 for the stone chunks.
• Texture size: No larger than 2048×2048.
• Initial page load size: Keep combined models and textures under 10MB, ideally with progressive loading.

Currently, generative AI can offer directional advice for this phase, but specific decimation parameters, UV repairs, and texture baking still require manual inspection and adjustment.

3. Implementing Interactive Narrative: From “Clicking” to “Viewing”

1. Structure Design

The project is structured into several stages:

• Dark Room Opening: Establishes the spatial atmosphere, drawing attention to the central stone block.
• Sculpting Phase: The user triggers stone peeling by holding a button or dragging a slider.
• Key Text Nodes: Text prompts appear at different stages to help users grasp the concept of “subtractive sculpting.”
• Completion Phase: The camera pulls back, revealing the full statue and prompting the user to observe the artwork.

An effective interactive design choice here is the “Undo” button. Seeing an undo button, users instinctively assume they can step back. However, the system displays: “Marble has no undo key.” This design serves both as interactive feedback and narrative expression.

From an implementation perspective, this prompt is simple—just a state check. However, it demonstrates that generative AI can provide high-quality creative ideas for interactive narratives. The key for developers is reliably translating these ideas into state machines, events, and UI logic.

2. State Management

As the sculpting progresses, the system must maintain multiple states:

{
  currentStage: "intro" | "carving" | "reveal" | "complete",
  removedChunks: number[],
  cameraPose: { position: Vector3, target: Vector3 },
  lightIntensity: number,
  dialogueIndex: number,
  hasTriedUndo: boolean
}

These states affect:

• Which stone chunks are hidden;
• Whether the camera moves;
• Whether the text switches;
• Whether interaction is currently allowed.

State management itself is not complex; the real challenge is ensuring temporal consistency among interaction triggers, animation playback, and text display. For instance, rapid clicks should not skip crucial camera shots, and the system should not advance to the next stage before an animation completes.

3. Visual Effects and Camera

The visual credibility of the sculpting process relies heavily on camera and lighting coordination.

• Dark Room Lighting: Use low ambient light plus a key light to create high contrast on the stone surface.
• Peeling Moment: Simultaneously hide stone chunks and trigger subtle particle or dust effects.
• Camera Push: Slowly move the camera closer to the statue after key sculpting milestones to focus on details.
• Completion Phase: Increase ambient light and reduce shadows to clarify the statue’s silhouette.

These effects can be achieved purely with frontend 3D capabilities, without relying on complex backend services.

4. Real-World Engineering Delivery Challenges

Generative AI often excels in the creative phase, offering narrative structures, visual style suggestions, and even core page code. However, bridging the gap from a generated page to a runnable, deployable web experience involves several engineering hurdles.

1. Model and Code Integration

AI-generated code usually assumes an ideal environment. When integrating real models, issues frequently arise:

• The model’s center point is not at the origin;
• Model scale does not match the scene;
• Material textures are missing or incorrectly oriented;
• Abnormal shadow calculations result in a completely black or overexposed scene.

These issues require step-by-step debugging in a real environment. Center point issues can be fixed by calculating the bounding box and re-centering; scale issues require unifying model units; material issues require checking glTF export parameters.

2. Performance and Asset Loading

3D web pages often suffer from slow first-screen loading and low runtime frame rates.

Common optimization techniques include:

• Using DRACO compression for mesh data;
• Using KTX2 compression for textures;
• Applying LOD (Level of Detail) for non-critical models;
• Dynamically adjusting rendering quality based on device performance;
• Lazy loading off-screen assets.

These optimizations are not default options in AI-generated code and must be actively implemented by developers.

3. Mobile Compatibility

Real users access the web via various screen sizes and devices with different performance levels. Mobile development requires special attention to:

• Touch events and button states;
• Canvas size adaptation;
• Degradation strategies for low-end devices;
• Portrait and landscape orientation.

A common issue: while the desktop uses mouse-down to trigger sculpting, the mobile end must simultaneously handle touchstart, touchend, and touchcancel, otherwise a finger lift might be misinterpreted as continuous pressing.

4. Deployment and Access

After running locally, you need to consider static asset hosting, transmission compression, caching strategies, and access link generation. Deployment can be accomplished using standard object storage, Content Delivery Networks (CDNs), and static site hosting. The core goal is ensuring model assets are reliably and quickly distributed to end users.

5. The Gap Between AI’s “Ideas” and “Delivery”

Based on this experiment, generative AI’s performance in 3D interactive projects can be summarized as follows:

Creative and Structural Level: It provides solid narrative angles, interactive concepts, and overall page structures. Copy like “Marble has no undo key” effectively elevates the expressive depth of the work.

Code Generation Level: It can generate a runnable foundational version, especially for standard page layouts, 3D scene initialization, and lighting/camera setups. However, when handling real model data, performance tuning, and mobile compatibility, the generated code is generally not robust enough.

Engineering Delivery Level: Human intervention is required. Steps like model processing, asset compression, state machine refinement, exception handling, and deployment configurations still demand strong engineering capabilities from the developer.

The takeaway is that for developers, generative AI is best utilized as a “creative accelerator” and “code draft generator” rather than a complete replacement for engineering implementation. The real value is unlocked by embedding AI capabilities into existing development workflows: using AI to rapidly prototype narrative structures and generate base code frameworks, while developers handle model integration, performance optimization, and final delivery.

6. Practical Recommendations

If you want to attempt a similar 3D interactive project, the following tips may be useful:

1. Establish a Runnable Asset Pipeline First
   Before adding narrative and effects, ensure the model loads, displays, and scales correctly, and that the frame rate is acceptable.

2. Decompose “Sculpting” into State Changes
   Do not make users wait for real-time Boolean operations. Pre-chunk the geometry or use shader masks to simulate a continuous process using discrete states.

3. Control Initial Asset Size
   Compress models and downgrade textures, prioritizing load speed and first-paint timing.

4. Set Clear Start and End Conditions for Each Interactive Stage
   Prevent users from interrupting state transitions during animations, which can cause visual and narrative misalignment.

5. Use Standard Cloud Services for Deployment
   Place static assets in object storage and accelerate access via a CDN to reduce user loading latency.

Conclusion

Using the theme of “sculpting David” to test generative AI’s 3D capabilities is a highly effective approach. It evaluates not just model display, but also interaction, narrative, performance, compatibility, and engineering delivery. The experiment shows that while generative AI provides an excellent creative starting point and base code, there remains a gap before it can directly deliver a production-ready project.

For developers, the realistic approach is to treat generative AI as a collaborative partner capable of quickly generating prototypes and exploring design directions, while keeping model processing and engineering capabilities firmly in their own hands. This leverages AI to reduce creative iteration costs while ensuring the final output meets the engineering standards required to be runnable, interactive, and deployable.