LLM-Driven 3D Interaction: Engineering Practices from Technical Prompts to Artistic Reproduction

As artificial intelligence evolves from “text conversation” toward “multimodal content generation” and “autonomous agent execution,” the challenge facing developers is no longer simply calling an API. It is about getting a large language model (LLM) to understand complex business intent and autonomously chain together search, coding, rendering, and other parts of the tech stack to deliver a runnable engineering artifact. This article uses a project in which an LLM recreated the birth of a classical sculpture as a 3D interactive website to examine how large models perform in complex front-end engineering, how they orchestrate multimodal tasks, and what the experience teaches about putting such systems into production.

1. Complex Intent Understanding and End-to-End Task Orchestration

In a traditional development model, building a website with 3D elements, specific interaction logic, and a narrative rhythm requires close collaboration between a product manager, front-end engineers, a 3D modeler, and a UI designer. In this project, end-to-end decision-making was handed to the LLM, constrained only by a single high-level instruction.

1.1 Defining Requirements by Goal, Not by Steps

The prompt given to the model had to specify the final deliverable rather than concrete execution steps. The core instruction was ambitious: build a Chinese-language interactive website that explains to ordinary people how a classical sculpture is born from raw stone. The instruction specifically emphasized that “the narrative angle, visual style, interaction design, and technical approach are up to you,” and required the model to “assess what the current environment makes possible, then proceed autonomously until you deliver a runnable, experienceable work.”

The essence of this prompting strategy is a goal-driven agent pattern. It demands that the model not only generate code, but also decompose tasks, evaluate its environment, retrieve assets, and check quality.

1.2 Narrative Reconstruction and Technical Decision-Making

Upon receiving the instruction, the LLM demonstrated a deep ability to reinterpret the subject. Rather than piling up facts, it distilled the sculpture-making process into the philosophical concept of “what remains in the stone.”

On the technical side, the model made the following decisions autonomously:

• Asset acquisition: It retrieved and used public-domain, high-precision 3D scan data (open-source STL models released under CC BY-SA 4.0), ensuring the authenticity and accuracy of the underlying geometry.
• Visual and interaction design: It abandoned the conventional showcase-style UI in favor of a dark-room environment that simulates the carving process. By listening for “press” events from the mouse or touch input, it drove the mesh in the front-end 3D engine to “flake away” according to preset logic, gradually revealing the inner form.
• Narrative touches: The model even embedded a counterintuitive detail into the interaction logic—providing an “Undo” button whose event callback intercepts the action and throws the message “Marble has no undo button.” Using code logic to reinforce the theme marks the LLM’s shift from “code porter” to “digital content creator.”

2. Breaking Down the 3D Rendering and Front-End Architecture

Whether simulating stone flaking with a particle dissolve effect or letting a slider peel the model down layer by layer, the underlying implementation relies on a mature Web 3D stack. Code generated by large models typically follows this common architecture:

2.1 Data Loading and Scene Initialization

High-precision STL models can contain hundreds of thousands of triangles. Loading such files directly can exhaust browser memory or stall the first render. Sound engineering practice is to parse asynchronously in a Web Worker during loading, or to convert the STL to a compressed glTF/glb format on the server in advance.

// Illustrative code: WebGL-based 3D scene and STL model loading logic
import * as THREE from 'three';
import { STLLoader } from 'three/examples/jsm/loaders/STLLoader';
const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(45, window.innerWidth / window.innerHeight, 0.1, 1000);
const renderer = new THREE.WebGLRenderer({ antialias: true });
const loader = new STLLoader();
loader.load('model/sculpture_base.stl', (geometry) => {
    // Optimize normal computation to ensure correct lighting
    geometry.computeVertexNormals(); 
    const material = new THREE.MeshStandardMaterial({ color: 0xaaaaaa, roughness: 0.8 });
    const mesh = new THREE.Mesh(geometry, material);
    scene.add(mesh);
    // Trigger scene rendering
    animate();
});

2.2 Implementing the Interactive Carving Effect

To achieve the visual effect of “stone breaking away piece by piece,” large models generally choose between two technical routes:

• Geometry cutting based on Boolean operations: Using Constructive Solid Geometry (CSG), a tiny cutting volume is generated at the mouse-click position and subtracted from the main stone model. This approach is physically realistic but computationally expensive, making it unsuitable for repeated real-time operations in the browser.
• Material dissolution based on shaders: The option better aligned with modern web performance standards. A custom GLSL shader uses noise functions (such as Perlin Noise) together with uniform variables passed from mouse interaction to drive transparency (alpha) transitions on the model’s surface pixels, while a particle system simulates flying stone chips—creating the illusion of flaking.

When generating the actual code, the model typically evaluates the available compute budget and picks the second approach to keep the frame rate (FPS) stable.

3. Multimodal Extension and the Content Delivery Loop

Delivering the 3D website did not end the task. To further test the model’s multimodal orchestration, we asked it to generate a short video introducing the work to a general audience, based on the finished site.

3.1 Cross-Modal Asset Reuse

The instruction allowed the model to reuse the website’s 3D assets, scenes, and code. This meant the model had to understand how to turn a WebGL canvas stream into a video stream. Technically, this can be done by capturing canvas frames with the MediaRecorder API, or by migrating the 3D scene code directly into a server-side rendering pipeline (such as a headless browser render farm) to produce high-frame-rate frame sequences that are then composited into an MP4 file.

3.2 Automated Video Narrative Generation

The model had to make its own calls on camera language, pacing, and audiovisual treatment. This requires video engineering capability: invoking low-level multimedia processing libraries to time-align 3D keyframe animation, TTS voiceover narration, and background music, and finally exporting a playable 9:16 MP4 file.

This process reveals a broader trend in content production: digital assets whose core logic lives in code can, under unified LLM orchestration, be seamlessly transformed into different delivery formats—web pages, videos, even VR experiences.

4. Hands-On Engineering Findings and Architecture Tuning

From watching the LLM complete these complex tasks, we distilled several practical engineering lessons for developers. These findings are valuable references for enterprises building AI agents.

4.1 Balancing Model Capability Against Latency and Cost

Testing showed that while top-tier flagship models (with very large parameter counts) offer extremely strong reasoning and code generation—enough to support highly ambitious architectural designs—their inference latency is high and token costs often exceed enterprise budgets. In real production of complex tasks, optimized iterative versions that run faster and cost nearly half as much (the cost-effective professional-tier models common in the industry) often strike a better balance between “creative quality” and “engineering delivery.”

4.2 Bridging the Gap Between “Producing a Plan” and “Delivering a Product”

The most common failure point when LLMs handle complex 3D and multimodal tasks is not “running out of ideas” but “never making it to delivery.” The model performs perfectly at the solution-design stage, but when writing hundreds of lines of complex front-end 3D code, it easily runs into dependency version conflicts, cross-origin resource loading failures, and memory leaks.

The key to closing this gap is a closed-loop engineering sandbox. Enterprise AI development platforms should integrate containerized execution environments and logging services. Once the model generates code, it must be executed immediately in the sandbox. If a runtime error occurs, the platform should automatically feed the error log back to the model as context, requiring it to debug and iterate until the output is truly “runnable and experienceable”—not just a static code file.

4.3 Ensuring Stable Delivery with Cloud Infrastructure

Applications involving heavy 3D rendering, video transcoding, and high-frequency LLM calls place high demands on compute and storage resources. In enterprise practice, this is typically supported by a standardized cloud-native architecture:

• Object storage: Hosts the high-precision 3D scan STL files and generated MP4 videos, ensuring high-concurrency access performance backed by CDN acceleration.
• Container platforms and microservices: Decouples LLM task orchestration, code sandbox execution, and multimedia rendering into independent microservices, enabling elastic scaling of resources.
• Monitoring and alerting: Tracks the agent’s token consumption, API call failure rates, and the CPU/memory load of sandbox containers in real time, keeping the automated delivery pipeline stable.

In summary, having an LLM recreate the birth of a classical sculpture is, at its core, a deep stress test of a modern AI agent’s end-to-end engineering capability. It demonstrates that, given well-crafted prompt constraints and a solid cloud foundation, LLMs can already deliver across the full lifecycle—from concept design and asset retrieval to low-level coding and multimodal content distribution. What developers need to do is establish sound engineering boundaries and fault-tolerance mechanisms, so that technology’s raw power gallops steadily along its intended track.