Analyzing LLM Code Generation: From Creative Prototypes to Engineering Implementation

1. The Shift in Evaluation Paradigms: From Benchmarks to Tangible Artifacts

In the past, evaluating the coding capabilities of Large Language Models (LLMs) relied heavily on benchmark tests: algorithm problem pass rates, bug fix rates, and repository-level task completion. While valuable, these metrics remain abstract and distant for most developers. Recently, the community has witnessed a shift: developers are increasingly showcasing actual artifacts built with LLMs. These include open-world style game scenes, interactive 3D models of foldable phones in browsers, and mechanical arm demonstrations broken down into over 2,000 components. These artifacts can be clicked and manipulated, shifting the evaluation scale from “scores” to “user experience.”

This shift is significant for two reasons. First, an artifact serves as a comprehensive end-to-end capability test, covering the entire pipeline from requirement understanding and architectural design to code generation, debugging, and修复. A single benchmark struggles to cover all these aspects. Second, artifacts inherently carry time and cost metadata, transforming the “capability-to-cost” ratio into a discussable engineering problem rather than a marketing talking point.

2. Technical Implementation Paths of Three Typical Artifacts

2.1 Open-World Style Game Prototypes

Community developers have showcased open-world style city roaming scenes complete with streets, vehicles, and controllable characters, utilizing commands to adjust graphical enhancements and demonstrate before-and-after comparisons. From a technical perspective, these projects are typically built on browser-based game technology stacks: rendering pipelines based on WebGL/WebGPU handle graphical output, entity and scene graphs organize various objects within the city, and simple pathfinding, collision detection, and state machines drive character behavior.

The true value of the LLM lies not in “inventing” these technologies, but in assembling scattered knowledge into a runnable whole—initializing the renderer, setting up the main loop, generating placeholder assets, handling resource loading sequences, and fixing specific issues like “characters clipping through walls” or “sudden frame rate drops” through multi-turn dialogue.

According to the developer, this project took about ten hours, consumed millions of tokens, and cost tens of US dollars. If a more expensive model had been used for the same task, the cost might have been an order of magnitude higher. It is important to emphasize that this is a specific case; actual costs correlate strongly with scene complexity, iteration rounds, and context management strategies, and cannot be directly extrapolated.

2.2 Interactive 3D Web Pages: From Modeling to Browser

Another representative case is an interactive web page of a foldable phone: the model is first generated in 3D modeling software and then exported to a web page where users can drag to rotate, toggle the folding state, and watch the screen content change accordingly. This pipeline consists of three stages:

• Asset Stage: Describe geometry using parametric methods and export to a standard 3D asset format.
• Rendering Stage: Load the asset in the browser and complete lighting, materials, and post-processing based on WebGL.
• Interaction Stage: Listen for pointer events, use easing functions to drive the folding animation, and synchronously update the screen texture.

For the LLM, the challenge lies in cross-tool format integration and state synchronization—the folding angle is a continuous variable, while screen content needs to switch discretely based on state. This coupling of “continuous animation + discrete logic” is highly error-prone and best tests the model’s grasp of interaction details. Such capabilities offer direct reference value for product presentations, creative proposals, and marketing page development.

2.3 Complex Structure Disassembly and Physical Simulation

The third direction focuses on “whether it moves naturally”: cloth fluttering in the wind, or a mechanical arm disassembled layer by layer into over 2,000 parts. The former relies on vertex shaders or cloth simulation; the latter tests hierarchical scene management—each part is an independent node, and the disassembly process is essentially applying displacement animations to nodes in a predefined order. According to developers, such cases can now be completed in under ten minutes per artifact, indicating that when a task can be decomposed into a clear sequence of steps, the model’s first-pass success rate is quite impressive.

3. Key Capability Factors: Why These Artifacts Are Possible Now

1. Long Context and Large-Scale Output: The consumption of millions of tokens implies the task involves thousands of lines of code and multiple rounds of corrections. The model must maintain architectural consistency under long-range dependencies to avoid “fixing A and breaking B.”
2. Multi-Turn Self-Correction: Artifact-level delivery relies on a “generate-run-feedback-repair” closed loop. The model must locate and fix issues based on error messages or runtime screenshots.
3. Toolchain Collaboration: Calling build tools, executing tests, and reading documentation upgrades the model from a “code completer” to a “task-delivering executor.”
4. Domain Knowledge Density: Knowledge of WebGL, animation easing, and physics simulation is sufficiently represented in public corpora, forming the foundation for generation quality.

4. Calculating Costs: Introduction to Token Economics

Tokens are the unit of measure for how models process text and code. In Chinese, one character typically corresponds to one or two tokens; in English, roughly one word corresponds to one or more tokens. The total cost of AI-assisted development can be approximated as:

Total Cost ≈ Σ per-turn dialogue ( Input Tokens × Input Price + Output Tokens × Output Price )

Three key variables impact total cost:

• Iteration Rounds: Each dialogue round resends the historical context to the model. The more rounds, the faster input tokens accumulate.
• Context Management: Timely pruning of irrelevant history and replacing full text with summaries can significantly compress input overhead.
• Model Tier: Using high-capability tiers for complex architectural design and economy tiers for simple boilerplate code can obviously reduce overall expenses through mixed scheduling.

Referencing public community cases, the cost of an interactive 3D web page can be as low as single-digit US dollars, while a game prototype consuming millions of tokens falls in the tens-of-dollars range. For “validating whether an idea is worth investing in,” this cost is already far lower than traditional prototype development. Currently, mainstream cloud platforms (such as Baidu AI Cloud and other LLM service providers) generally adopt a token-based billing model. Developers can conduct a small-scale trial calculation first to decide on their investment.

5. Engineering Best Practices

A verified prompt structure template:

[Role] Senior Web Full-Stack Engineer, proficient in WebGL rendering and performance optimization
[Objective] Implement an interactive 3D product showcase page
[Constraints]
- Deliver a single HTML file that runs immediately in a browser
- Support mouse drag rotation, scroll wheel zoom, and button toggle for folding state
- Frame rate no less than 30 FPS, use easing functions for animation transitions
[Output] Complete code + execution instructions + list of known limitations

Accompanied by a few practical tips:

• Small Steps, Fast Iterations: Request the minimum runnable version first, then incrementally add features to avoid generating massive, all-inclusive code at once.
• Explicit Acceptance Criteria: Write frame rates, interaction points, and compatibility requirements into the prompt to let the model self-check.
• Version Control: Commit the generated code from each round to a repository so you can roll back easily if issues arise.
• Manual Review: Generated code must undergo security and performance reviews, especially concerning dependency imports and key management.

6. A Sober Perspective: Capability Boundaries Still Exist

Most showcased artifacts are “demo-level” rather than “production-level”: art assets rely on procedural generation or placeholders, gameplay depth is limited, and physics simulations prioritize visual plausibility over precise accuracy. Furthermore, architectural decisions for long tasks, performance bottleneck tuning, and complex data structure design still require human judgment. Positioning LLMs as accelerators that “quickly turn ideas into runnable drafts” rather than silver bullets replacing engineers is a more pragmatic understanding at present.

7. Conclusion

As evaluation shifts from benchmarks to artifacts, the discussion around AI-assisted development moves from “can it be done” to “is it worth it, and how to use it.” For developers, rather than just observing cases, a more valuable approach is to pick a small idea, use the template and cost framework above, and replicate it hands-on. Through real iterations and bills, you can establish your own independent judgment regarding capability boundaries and cost structures.