AI Agent in Practice: Building an Interactive 3D Sculpture Experience from Scratch

A while ago, I set myself an experiment: can today’s AI coding agents independently deliver an ambitious creative web project — not a form page or landing page, but something people actually want to stop and experience? The brief landed on a concrete subject: build a Chinese-language interactive website that helps ordinary people understand how Michelangelo’s David was born from a block of stone.

I defined two soft success criteria. After viewing, the audience should not just memorize a few historical facts but gain a fresh understanding of what sculpting actually involves — and be willing to linger on the work. Narrative angle, visual style, interaction design, and technical stack were all left to the agent. The only rule: first assess what the current environment can do, then autonomously push forward until a runnable, experienceable product is delivered — no stopping at proposals or mockups.

The results exceeded expectations. Below is a full retrospective organized as: task design, solution observation, technical breakdown, video extension, and hands-on recommendations.

1. Task Design: Define “Success,” Not “Steps”

When handing an open-ended brief to an agent, prompt structure matters more than length. A well-structured task description typically has three layers:

• Goal layer: State clearly “for whom, solving what.” Example: for audiences with no art-history background, help them understand how the statue came to be and create an emotional response. This layer sets the quality ceiling.
• Constraint layer: List non-negotiable hard conditions. Example: Chinese UI, interactive, must deliver a runnable product, assets must comply with open licenses. This layer guards the floor.
• Open layer: Explicitly hand over judgment. Narrative structure, visual style, interaction metaphor, and technology choices are all the agent’s call. This layer is where real capability gets tested — if every step is prescribed, you’re only measuring execution; handing over judgment tests “product sense.”

One easily overlooked detail: instructing the agent to “assess environment capabilities first, then proceed autonomously.” This single line significantly reduces two common failure modes — picking a heavyweight solution the environment cannot run, and endlessly outputting design documents without writing code.

2. One Brief, Two Radically Different Answers

Running the same core brief produced two finished products on very different tracks, which in itself is a valuable observation.

Solution A took an “immersive installation” route: the site opens on a dark room with a marble block in the center. The viewer holds a button to chisel, stone chips fall away piece by piece, and the statue’s silhouette gradually emerges. The narrative core is one sentence — the statue is what’s “left” inside the stone. A standout interaction detail: an “undo” button is provided, but clicking it triggers the message “Marble has no undo key.” This line binds the interaction mechanic to the theme — a classic consistency bonus.

Solution B took an “exhibition booklet” route: a paper-toned background unfolds in four acts. The viewer drags a slider to watch the stone block get reduced layer by layer until the statue appears. It reads more like an operable exhibition catalog with a clear information structure, suited to casual browsing.

Both solutions used the same publicly available 3D scan data of the statue (under a Creative Commons Attribution-ShareAlike license) and properly credited the source. This shows the agent is reliable on asset sourcing and compliance. More importantly, when a task leaves enough decision space, different agents form their own “creative stance,” and solutions naturally diverge. When evaluating this kind of capability, don’t just tick off a feature list — check whether narrative, visuals, and interaction all serve a single core expression.

3. Technical Breakdown: How a “Sculpting” 3D Web Experience Works

3.1 Asset Acquisition and Format Selection

Open scan data of museum-grade sculptures is not hard to find; common formats are STL or OBJ. STL contains only geometry, produces large files with no PBR materials, and is unsuitable for direct web use. The standard practice is to convert to a web-oriented format like glTF, then enter an optimization pipeline.

3.2 Model Optimization: From Scan to Loadable Asset

Raw scans can run into millions of polygons — loading them directly will stall a browser. A practical optimization pipeline looks like this:

Raw scan mesh (millions of faces)
 ├─ Mesh cleanup: fill holes, remove non-manifold edges
 ├─ Decimation: preserve silhouette features by visual importance
 ├─ UV unwrap + texture baking (color, ambient occlusion)
 ├─ Mesh compression + multi-level LOD generation
 └─ Export to web format, load on demand

Decimation must be restrained: the statue’s silhouette is the narrative protagonist. Over-aggressive reduction loses the “human form” feel that matters most. The rule of thumb is to spend most of the polygon budget on the contour and let baked textures carry the detail.

3.3 Two Routes to the “Carving” Feel

Route 1: Boolean subtraction (CSG). Each chisel strike uses a random convex polyhedron to subtract from the stone mesh, physically modifying geometry. The visual result is the most grounded:

// Pseudocode: each chisel strike actually cuts the mesh
async function chisel(stone, hitPoint, toolRadius) {
  const chunk = randomConvexChunk(hitPoint, toolRadius); // generate a random convex fragment
  const next = await csg.subtract(stone.geometry, chunk); // boolean difference
  spawnDebris(hitPoint, chunk.volume);                    // emit stone-chip particles
  return next;
}

The trade-off is heavy computation; rapid consecutive clicks cause frame drops. Mitigation: queue boolean operations asynchronously and instance-reuse fragment geometry.

Route 2: Clipping plane / progressive reveal. The mesh stays intact; a progress-driven clipping plane controls how much of the statue is “exposed,” while a particle system supplies the chipping effect:

// Pseudocode: progress-driven reveal
function updateReveal(progress) {
  const plane = buildClipPlane(progress);      // plane position driven by progress
  statueMaterial.clippingPlanes = [plane];
  debrisEmitter.intensity = pulse(progress);   // spike debris on chisel moments
}

This route is lightweight and mobile-friendly, though the physical realism of “stone removal” is weaker. The two finished products happened to land on these two routes respectively, and the interaction feel differs noticeably: the press-and-hold style feels more like actual chiseling, while the slider style better controls narrative pacing.

3.4 Performance Budget

Rough performance red lines for this kind of project: keep initial critical resources small, maintain runtime triangle counts and draw calls at a level mid-range mobile devices can render smoothly, and stay within texture memory limits. Common techniques include geometry and texture compression, reusing a single geometry for debris particles, lazy-loading heavy assets like audio, and automatically downgrading non-essential post-processing on mobile.

4. From Website to Video: Reusing Assets as a Pipeline

After the website was done, I extended the task: based on the finished site and its research process, generate a 60–90 second Chinese-language vertical short video for a general audience — communicating the website’s essence and sparking further interest in the statue. Narrative, shots, pacing, audio, and visuals were the agent’s call. Deliverables: a playable 9:16 final cut, an editable project file, a source list — and actually inspect the cut and fix obvious issues.

The key lesson here is asset reuse: scenes, models, and animation code already built for the website can directly become the video’s visual assets, slashing production cost. The requirement to “actually inspect the final cut” is critical — video tasks often fail not because the agent can’t produce, but because it never looks back. Problems like occlusion, misaligned subtitles, or sluggish pacing only get fixed when acceptance criteria are written into the brief, prompting the agent to self-check.

5. Hands-On Findings and Engineering Advice

1. Long tasks tend to “fade out” in the final stretch. A typical pattern: the agent starts with clear thinking and interesting ideas but gradually loses focus during implementation and integration, failing to reach full delivery. The fix is to break big tasks into milestones — concept approval, runnable prototype, content fill, performance tuning, deployment sign-off — each requiring a runnable artifact, not just a verbal report.

2. Balance capability, cost, and speed. Stronger models are noticeably more creative on open-ended tasks, but they are slower and more expensive. Follow-up iterations on smaller versions often keep quality while cutting cost and boosting speed. A smarter approach is “task-tiered routing”: use a strong model for creative definition and architecture, then hand iterative edits and style tweaks to faster, cheaper models. Heavy continuous use of a top-tier model burns through quota fast; tiering flattens the cost curve.

3. Static projects need proper hosting and distribution. These deliverables are typically pure static sites. Host the pages and model files on object storage with CDN acceleration, and you’re set for a good experience. Avoid hot-linking external asset URLs directly — a broken link kills the work. Open licenses usually require attribution and share-alike; self-hosting while preserving attribution is the safer path.

4. Delivery checklist. In retrospect, “creative agent tasks” of this kind can be validated against five criteria: runnable (open and experience, no manual fixes), performant (smooth on mobile), compliant (clear sources, permitted licenses), narratively consistent (every detail serves the core message), and maintainable (clean code structure, editable project).

Conclusion

The biggest takeaway from this experiment is a confirmation: today’s AI coding agents can deliver end-to-end creative 3D work — provided the task is designed right. Define “success” clearly, genuinely hand over judgment, and bake inspection into the workflow. The statue is just the vehicle; what’s truly worth preserving is the method: open-ended brief + structured prompt + hard acceptance criteria. Next time you want to test an agent’s full capability, try a similarly culturally rich concrete subject. It will show you the model’s real level more clearly than any benchmark leaderboard.