Building Interactive 3D Narratives with Multimodal Models: A Practical Guide

1. Starting with an Ambitious Requirement

Having a multimodal model with code generation capabilities independently complete an interactive project with a clear expressive intent is an effective way to test its comprehensive skills. The task documented in this article was to create an interactive website that explains to a general audience how Michelangelo’s David was born from a block of stone. The goal was to give viewers a new understanding of “carving” itself, rather than just having them memorize dates and names.

The key to this task is that it has no standard answer. The narrative angle, visual style, interaction method, and technical route were all left to the model’s discretion. The prompt provided only goals and constraints, not a checklist. This “open-ended delegation” reflects the reality of actual development environments: stakeholders describe intent, not implementation plans.

From an engineering perspective, this task simultaneously tests four types of capabilities: domain research and filtering, narrative structure design, frontend and 3D rendering implementation, and final delivery integrity verification. If any link breaks, the output remains a “draft proposal” rather than an experience.

2. Prompt Structure: Constraining Intent Over Feature Checklists

The structure of the original prompt is worth deconstructing. It consists of four layers:

1. Role and Goal Layer: Clarifies that the audience is “general users” and defines the success criteria as “generating a new understanding and prompting them to pause and observe,” rather than “covering specific knowledge points.”
2. Authorization Layer: Explicitly states that narrative, visuals, interaction, and technical routes are decided by the model, encouraging it to “use your judgment.”
3. Resource Layer: Allows searching for materials, writing code, and using 3D tools, granting permissions for tool invocation and external retrieval.
4. Delivery Layer: Requires evaluating environment capabilities first, then autonomously proceeding until a runnable, experiential product is produced; explicitly forbids stopping at the proposal or mockup stage.

This four-layer structure can be abstracted into a reusable prompt template:

[Goal] Explain <Topic> to <Audience> so that they gain a new understanding of <Core Concept> after the experience.
[Constraints] Language/duration/medium limits; must not deviate from the topic.
[Authorization] Narrative angle, visual style, interaction form, and tech stack are determined by you.
[Resources] You can search for materials, write code, and use 3D or graphics tools.
[Process] Assess environment capabilities -> Propose route -> Implement -> Self-check and fix.
[Delivery] A runnable, interactive finished product, not a document or mockup.

The value of this structure lies in separating “what to do” from “how to do it,” delegating judgment to the model while using delivery standards as a safety net. Compared to listing page elements line by line, open-ended delegation is more likely to inspire non-templated expression. However, it also demands higher self-convergence capabilities from the model—without a checklist, the model must define the boundaries of “done” itself.

3. Narrative Design: Abstracting “Carving” as a Subtraction Process

Different implementations yielded different narrative approaches, but the core remained identical: the essence of carving is subtraction.

Approach 1: What Remains. This solution defined David as “the part of the stone left behind.” The opening scene is a dark room with a block of marble in the center. The user holds down a button to chisel, and the stone peels away block by block, gradually revealing the form. The most clever detail in this design is providing an “Undo” button, which, when clicked, displays the prompt, “Marble has no undo key.” This interaction conveys irreversibility at a very low cost—the fundamental difference between carving, painting, and digital modeling.

Approach 2: Exhibition Reader. Another solution adopted an exhibition language with a paper-colored background, progressing in four acts. Users drag a slider to reduce the stone from a cube layer by layer until the form is complete. The slider compresses the “timeline” into a “spatial axis,” allowing viewers to perceive the overall subtraction process in a single drag, making it suitable for quickly building global understanding.

These two approaches represent two directions of interactive narrative: Process Experience (heavy on participation and emotion) and Global Cognition (heavy on structure and efficiency). In real projects, they are not mutually exclusive. A common practice is to use a slider to establish a global framework, then insert “press and hold” micro-interactions at key nodes to create pauses.

From an information architecture perspective, the entire experience can be broken down into four states:

Phase	Stone State	User Action	Information Conveyed
Prologue	Complete cube	Observe, click start	Material and scale
Rough Carving	Heavy reduction	Press and hold to chisel	Subtraction and irreversibility
Fine Carving	Form emerges	Drag slider	Proportional judgment
Completion	Complete form	Orbit and observe	Change in viewing perspective

This state machine forms the backbone of the frontend implementation; all subsequent 3D and interaction code is organized around it.

4. Technical Route: Assets, Rendering, and State Management

4.1 3D Asset Acquisition and Compliance

High-quality sculpture models do not need to be built from scratch. Public scanning projects dedicated to cultural heritage provide numerous 3D scan files of museum collections, typically released in STL or GLB formats with explicit Creative Commons licenses. The test used a public scan asset of David, licensed under Attribution-ShareAlike.

Engineering-wise, three points require attention: First, STL files usually contain only geometric information without materials or UV maps. Direct web rendering will result in a “white model” effect, requiring supplementary materials or stylized shading. Second, scanned models often have excessively high polygon counts; mesh decimation and Draco compression must be applied before mobile loading. Third, the license requires attribution. Retaining source and license information on the page is a compliance baseline, not an optional feature.

4.2 Rendering Pipeline

The mainstream choice for web-based 3D rendering is a WebGL-based graphics library. The core workflow is: load compressed geometry -> establish material -> configure lighting and camera -> render loop. Below is the key skeleton (illustrative):

// Scene initialization (illustrative)
const scene = new Scene();
const camera = new PerspectiveCamera(45, w / h, 0.1, 100);
const renderer = new WebGLRenderer({ antialias: true });
// Load compressed geometry
const loader = new STLLoader();
loader.load('david.compressed.stl', (geometry) => {
  geometry.computeVertexNormals();
  const material = new MeshStandardMaterial({
    color: 0xd8d2c8,      // Marble color
    roughness: 0.85,
    metalness: 0.0
  });
  const mesh = new Mesh(geometry, material);
  scene.add(mesh);
});
// Frame-by-frame rendering
function tick() {
  renderer.render(scene, camera);
  requestAnimationFrame(tick);
}

4.3 Implementing the “Peeling” Effect

The “stone peeling away block by block” segment is the most expressive part of this project. There are three feasible engineering paths, with increasing complexity:

• Shader Clipping: Uses a vertex shader with clipping planes or noise masks to discard fragments based on progress, visually achieving a “chiseled away” effect. Low implementation cost, requires no pre-cut geometry, suitable for rapid prototyping.
• Pre-cut Fragments: Offline splitting of the stone into several fragments, dynamically hiding them and playing fall animations at runtime. High visual realism, but requires an additional asset preprocessing pipeline.
• Real-time Boolean Operations: Uses CSG for runtime subtraction. Most intuitive expression, but high performance overhead; not recommended for mobile.

In the test scenario, shader clipping combined with particle debris was the optimal balance: it preserves the semantic meaning of “subtraction” without letting first-screen load times spiral out of control.

4.4 State Management and Interaction Feedback

The core of the interaction is mapping the continuous input of “press and hold to chisel” to a progress value, which then drives the clipping parameters:

let progress = 0;               // 0 -> 1
const clamp01 = v => Math.min(1, Math.max(0, v));
function onPointerDown() {
  pressing = true;
}
function onPointerUp() {
  pressing = false;
}
function update(dt) {
  if (pressing) {
    progress = clamp01(progress + dt * 0.35);
    material.uniforms.uProgress.value = progress; // Drive clipping
    spawnChips();                                 // Debris particles
  }
}

The handling of the “Undo” button is also worth mentioning: it does not change any state; it only pops up a text prompt. This “fake interaction” is not deception, but rather uses interaction inertia to create a cognitive gap—the user expects the button to work, but the system informs them of the rules of the physical world. This is one of the lowest-cost, highest-impact techniques in narrative design, provided the copy is restrained and not repeatedly triggered.

5. Video Adaptation: From Web Page to Short Video

The same theme also needed to be presented as a vertical short video. This task similarly used open-ended delegation: based on the completed website and research, the model was to decide how the story fit video expression. It could reuse website assets and scenes or reorganize the visuals. Narrative, camera, pacing, sound, and visuals were all decided by the model. The deliverables included a playable final cut, an editable project file, and a source asset list, requiring actual review and fixing of obvious issues.

This exposed a universal rule: web experience and video narrative are two different information structures. Websites rely on active user exploration, allowing for layered information and delayed reveals. Video is a linear timeline; it must establish suspense in the first few seconds and complete a full emotional arc within a limited duration. Therefore, a “website intro video” should not be a screen recording. The visual language must be redesigned—for example, opening with a close-up of the stone, using chisel strikes to establish rhythm, and using the “cannot undo” prompt as a narrative turning point.

The “editable project + source assets” in the delivery checklist is an easily overlooked engineering requirement. It means the output must be maintainable and traceable, not a one-off render. This aligns with the software delivery principle of “buildable and auditable.”

6. Practical Findings: Capability Boundaries and Cost Constraints

The testing process yielded several observations worth recording:

1. Great Ideas, But Must Push to Delivery. The model can provide structurally complete narratives with novel entry points, but pushing from a mid-stage proposal to a runnable product still requires a continuous self-verification loop. Under open-ended delegation, models tend to linger too long in the “design” phase, compressing the implementation phase.
2. Version and Cost are Real Variables. Different tiered models within the same series show significant differences in latency and unit cost. Lowering single-call costs and improving response speeds directly change the feasibility of “running multiple iterations.” In trial-and-error tasks, cost structure often impacts the final outcome more than single-peak capability.
3. Quota Management Affects Delivery Integrity. The video segment failed to iterate fully due to exhausted compute quotas, proving that resource planning should occur alongside task decomposition: schedule high-consumption tasks during periods with budget surplus, or downgrade to lighter implementation paths in advance.
4. Self-Checks are Mandatory. A page opening does not equal a successful experience. Loading failures, camera clipping, mobile touch event conflicts, and long-press triggering system menus must all be verified on real devices.

7. Reusable Engineering Recommendations

These experiences can be distilled into several actionable recommendations:

• Anchor Openness with Delivery Standards. Open-ended prompts must be paired with a clear “definition of done,” such as “runnable, interactive, passes real-device verification.” Otherwise, they easily produce beautiful but unusable draft proposals.
• Assess Environment Before Defining Tech Route. Before writing the first line of code, confirm: available asset formats, target device performance limits, and whether offline preprocessing is needed. This step prevents massive rework.
• Abstract Narrative into a State Machine. Any interactive narrative can be broken down into finite states and transition conditions. Draw the state table before writing rendering logic, and the code structure will be much clearer.
• Prioritize Asset Compliance. When using public scan assets, simultaneously record the source, license, and attribution requirements, and display them in the UI.
• Distinguish Experiential from Cognitive Interaction. Use “press and hold” continuous input for segments requiring emotional impact; use sliders or timelines for building global cognition. Mix them cautiously.
• Design Information Structures for Video and Web Separately. Do not use screen recordings as promotional videos. Re-orchestrate suspense and pacing along a timeline.
• Reserve Compute Budgets. Multi-round iterations and video rendering are high-consumption segments. Plan ahead and downgrade implementations if necessary.

Conclusion

The significance of this test lies not in how pretty a webpage turned out, but in demonstrating a complete path: using open-ended delegation with clear intent to let the model autonomously complete the entire process from narrative design to engineering delivery. The judgment the model exhibited was impressive, but what truly determines success or failure is engineering discipline—abstracting state machines, asset compliance and compression, the habit of real-device verification, and upfront cost planning. Doing these things solidly ensures that open-ended delegation reliably translates into an experiential product, rather than just another good-looking proposal.