Agent-Driven 3D Interactive Storytelling: Development Practice and Delivery Lessons

1. From a One-Sentence Brief to a Working Artifact

Asking an agent with code generation and tool-calling capabilities to deliver a runnable, explorable 3D interactive website from a single paragraph of natural language is a comprehensive test of its planning ability, tool use, and engineering execution. This article uses a real task as its case study: build a Chinese-language interactive site that explains to a general audience how a classic sculpture is gradually released from a block of raw stone, and that leaves viewers not only remembering a few historical facts but also forming a new understanding of the act of carving itself.

The brief was deliberately left open: narrative angle, visual style, interaction model, and technical route were all left to the agent’s judgment, with only three hard goals—it must run, it must be explorable, and it must not stop at the proposal stage. This “few constraints, high demands” setup is exactly what exposes an agent’s real capability boundaries on long-chain tasks. The following sections break the work down across five layers: task decomposition, technical route, interaction design, derivative content production, and delivery retrospective.

2. Task Decomposition: Turning an Open Brief into Executable Milestones

Given an open-ended prompt, jumping straight into coding is the easiest way to lose control. A safer approach is to assess capabilities first, then define milestones. A reusable decomposition framework looks like this:

Phase 1: Environment assessment
  - Runtime and dependency availability (package management, build tools, browser rendering capability)
  - Network and asset acquisition channels (public 3D assets, textures, fonts)
Phase 2: Content research
  - Subject background facts (creator, period, material, process)
  - Narrative spine (the "subtraction" process of a block of stone)
Phase 3: Technical selection
  - Rendering layer: WebGL / 3D engine
  - Asset layer: model formats and compression strategy
  - Interaction layer: state machine and event model
Phase 4: Implementation and integration
  - Scene setup → geometry-stripping interaction → copy and guidance → mobile adaptation
Phase 5: Delivery acceptance
  - Runtime check, performance check, accessibility check

The key to this decomposition is putting assessment first. The agent must answer “what can the current environment do”—for example, whether it can install 3D engine dependencies, whether it can reach public model repositories, and whether the browser supports the required shader features—before committing to a technical route. Skipping assessment and generating code directly tends to produce repeated failures at dependency installation or asset loading, eventually exhausting the execution budget.

3. 3D Assets: Real Scan Models Are the Foundation of Narrative Credibility

To make viewers “willing to stop and observe,” the realism of geometric detail is essential. Procedurally generated approximations cannot carry the anatomical detail of a sculpture’s surface, so the plan selected a high-precision sculpture scan from a public 3D scanning community, released under an open license that permits legitimate use for demonstration.

Integrating the asset involves several engineering issues that are easy to underestimate:

1. Format conversion and size control. Raw scan models have extremely high polygon counts, and loading them directly causes excessively long first-screen waits. Mesh simplification is needed while preserving silhouette and muscle relief, followed by conversion to a browser-friendly format.
2. Coordinate system and scale normalization. Scan assets differ in origin, orientation, and units, and must be unified into the scene coordinate system—otherwise camera orbit controls and lighting calculations will be distorted.
3. Normal and material reconstruction. Scan models often lack well-formed normal information, so tangent-space normals must be recomputed; for materials, a neutral stone-colored PBR material is preferable, avoiding the extra bandwidth cost of high-dynamic-range environment maps.

// Illustrative asset loading and simplification strategy
async function loadSculpture(url) {
  const gltf = await loader.loadAsync(url);
  const mesh = gltf.scene.children[0];
  // Progressive simplification toward a target face count, preserving silhouette features
  const simplified = simplifyMesh(mesh, { targetRatio: 0.18, preserveBoundary: true });
  simplified.geometry.computeVertexNormals();
  return simplified;
}

4. Interactive Narrative: Replacing Knowledge Transfer with a “Subtraction” Metaphor

4.1 Narrative Angle

Conventional popular-science pages tend to list facts such as date, dimensions, and material, leaving the audience’s cognitive gain at the memorization level. This plan adopts “subtraction” as its core metaphor: a sculpture is not added into existence but gradually released as the residual form of the stone. This angle maps naturally onto 3D interaction—each “chisel strike” the viewer performs removes a layer of geometry, so visual change and conceptual understanding happen in sync.

4.2 Scene and Visuals

The page opens into a low-light exhibition space with a complete block of stone at the center, its volume outlined by a single side light. The low-light environment suits the “studio” context and reduces distraction from irrelevant visual elements. As more layers are stripped away, ambient light gradually increases, and the final finished form receives full illumination—creating a visual rhythm of “from hidden to revealed.”

4.3 Interaction Mechanism

The core interaction is a sustained “press-and-hold to chisel” action rather than a single click. The intent behind continuous pressing is to let viewers experience the repetitiveness and physicality of the process, rather than reducing it to a single button operation. Geometry stripping is implemented via pre-segmented layered meshes: the model is pre-cut into several shells from outside to inside, each corresponding to accumulated chisel progress; once progress crosses a threshold, that layer fades out and reveals the inner layer.

// Illustrative state advance for layered stripping
const state = { layer: 0, progress: 0, threshold: 0.34 };
function onChiselTick(delta) {
  state.progress += delta;
  if (state.progress >= state.threshold) {
    state.progress = 0;
    state.layer += 1;
    revealLayer(state.layer);      // Fade out the outer layer, show the inner one
    updateNarrative(state.layer);  // Advance copy and ambient light in sync
  }
}

4.4 Copy and the Negative Design of “Undo”

The page includes an “Undo” button which, when clicked, does not roll back the geometry state but instead displays a message: once stone is chiseled away, it cannot be restored. This is a deliberate negative interaction design, using the feedback itself to convey the irreversibility of the material. Compared with stating this fact in the body text, having viewers trigger a failed operation firsthand produces markedly stronger retention. Such designs require restraint: negative feedback should appear only once and be worded with restraint, so it does not repeatedly interrupt the experience rhythm.

5. An Alternative Route: Linear Exhibition-Style Narrative

The same brief can also lead to a completely different expression. The second route adopts an exhibition-reader style with a light paper background, dividing content into four consecutive acts and using a single slider to control the stone’s reduction from a complete block down to the finished form. Compared with the immersive route above, this one is closer to a graphic exhibition:

• Controllable pacing: the slider hands control to the viewer, suiting self-directed reading rhythms;
• Lower implementation cost: no complex lighting or particle effects, so performance pressure is low;
• Higher information density: process notes and historical annotations can sit alongside each act.

Both routes share the same scan asset, which shows that the asset layer and the narrative layer should be decoupled: the model, materials, and the intermediate representation of layers can be reused, while camera, lighting, interaction, and copy are implemented independently per narrative need. This layering is the key to improving reuse rates.

6. Video Derivative Content: Reorganizing from Website to Short Video

After the website was delivered, the agent was further asked to produce, based on the completed site and research materials, a 60–90 second vertical Chinese short video aimed at a general audience that conveys the site’s core thesis and sparks interest in learning more.

Turning this into video is not simple screen recording; it requires re-judging what media expression suits the story:

1. Camera language: in vertical framing, the subject should sit centered and slightly high, chisel actions shown in close-up, and the stripping process compressed with time-lapse;
2. Rhythm design: the first 3 seconds juxtapose the raw stone against the finished form to create suspense, the middle section uses three stripping beats to map to three cognitive turning points, and the ending returns to the theme of irreversibility;
3. Sound design: chisel sounds form the rhythmic skeleton, narration stays in short sentences with low information density so it does not compete with the visuals;
4. Asset reuse: reuse the site’s layered models and lighting configuration to render key frames, ensuring visual consistency;
5. Deliverables: a playable vertical video file, an editable project file, and an asset source list—plus actual review of the final cut and fixes for obvious issues (such as subtitles running out of frame, audio-video desync, or a black first frame).

This stage exposed the agent’s weaknesses in cross-modal production: its ability to organize visual material is acceptable, but rhythm control and audio-visual alignment require multiple rounds of human review. If the prompt explicitly provides a shot-duration table and a checklist, the pass rate of the final cut improves noticeably.

7. Delivery Retrospective: Capability, Cost, and Engineering Constraints

7.1 Planning Ability Exceeds Execution Endurance

The most typical failure mode observed is not “couldn’t think of it” but “didn’t finish it.” The agent can produce a clearly structured technical route and narrative plan, but after repeated consumption on long-tail steps such as dependency installation, asset downloads, and build troubleshooting, it tends to stall just short of a runnable state. The countermeasure is to split delivery into multiple independently verifiable milestones, each accepted on the criterion “it opens and it interacts” rather than “the code is written.”

7.2 Tradeoffs Between Cost and Speed

The same task performs noticeably differently under different inference configurations: heavier configurations yield higher creative quality but significantly higher time and cost, while lighter configurations offer advantages in speed and unit price and are sufficient to support medium-complexity page generation. In practice, resources should be allocated by task stage—use more capable configurations for solution design and narrative ideation, and switch to more cost-effective configurations for bulk code generation and troubleshooting, maintaining quality while keeping total cost under control.

7.3 Runnable-Acceptance Checklist

Whichever technical route is taken, a mechanical checklist should be passed before delivery:

• The first screen renders within a reasonable time, with no prolonged blank screen;
• Core interactions (chisel / slider) can be triggered on both desktop and mobile;
• Layered stripping shows no geometry interpenetration or flipped normals;
• Copy and interaction state stay strictly in sync, with no misalignment;
• Asset license information is fully retained;
• A fallback message appears when offline or when asset loading fails.

8. Conclusion

This case shows that an agent with tool-calling and code-generation capabilities can already independently complete the full chain of research—selection—implementation—delivery under an open brief, and produce a 3D interactive work with narrative ingenuity. However, its reliability remains constrained by long-chain execution endurance and by rhythm control in cross-modal content. Practical engineering improvements include: front-loading environment assessment, constraining progress with runnable milestones, decoupling the asset layer from the narrative layer, allocating inference resources by stage, and closing out acceptance with a mechanical checklist. For teams hoping to bring this capability into content production workflows, the recommendation is to first run a single narrative theme end-to-end, then gradually expand to batch production across multiple themes and media.