Evaluating and Practicing Long-Horizon Code Generation with Large Models

Recently, with the continued evolution of next-generation large language models in code generation and multimodal tasks, a series of application cases worthy of deeper analysis has emerged. These cases share common characteristics: long task chains, involvement across multiple types of tools and runtime environments, and high demands on the model’s spatial reasoning and state maintenance capabilities. This article abstracts and decomposes these capabilities from an engineering practice perspective, and offers reusable evaluation approaches and implementation recommendations.

1. Changes in Task Form: From Single-File Generation to Long-Horizon Engineering Orchestration

Traditional code generation evaluations mostly focus on function-level or single-file-level tasks with clear input-output boundaries. However, a notable category of current tasks has a distinctly different form:

• Multi-stage dependencies: A task must first complete scene modeling, then generate interactive logic, and finally package everything into a runnable artifact;
• Cross-runtime collaboration: A single task may simultaneously involve 3D rendering, physics simulation, frontend interaction, and resource management;
• Implicit state maintenance: The model must maintain consistency in variable naming, coordinate systems, and unit conventions across hundreds of dialogue turns or tens of thousands of lines of code.

The evaluation focus for such tasks is no longer “can it write a correct piece of code” but “can it maintain engineering consistency over long-horizon execution.” Based on publicly shared practices, a complete 3D open-scene prototype often consumes millions of tokens and requires hours of iteration to converge—which itself indicates that long-horizon orchestration has become a core bottleneck.

2. Decomposition of Typical Capability Dimensions

Based on currently public application cases, the relevant capabilities can be summarized into the following dimensions.

2.1 Scene-Level Code Synthesis

In game prototype tasks, the model needs to generate complete scenes including terrain, vehicles, and character movement logic. The difficulties of such tasks lie in:

1. Coordinate system and scale consistency: Terrain height, vehicle dimensions, and camera frustum must match each other; otherwise, clipping or proportion distortion occurs;
2. State machine design: Character movement, vehicle driving, and collision detection require clear state transition logic;
3. Rendering pipeline adaptation: Feature toggles for different rendering backends (such as supersampling and anti-aliasing) should be injected via configuration rather than hardcoded.

A reusable practice pattern is to have the model first output a “scene contract” (coordinate origin, units, hierarchy structure), then generate specific modules based on that contract. This can significantly reduce later rework rates.

Scene contract example (pseudocode description):
origin = (0, 0, 0)
unit = meter
up_axis = +Y
layers = [terrain, vehicle, character, camera]

2.2 Bridging 3D Content and Web Interaction

Another high-frequency task category is “model first, then interact.” A typical workflow is: generate an assembly in a 3D modeling environment, export it in a standard format, then drive its motion via scripts in the browser.

The key to this chain lies in the design of an intermediate layer for data formats. If the model directly generates final web code while skipping the intermediate representation, later modification costs become extremely high. A more robust approach is:

• Stage one: Generate a structured description (part tree, materials, joint constraints);
• Stage two: Use the description to drive modeling scripts;
• Stage three: Export to a common 3D format and bind interaction events on the web side.

The foldable device display case exemplifies this pattern: unfolding and folding are essentially interpolation animations of joint angles, while screen content changes are state-driven texture switches. Decoupling the two allows the model to handle “geometry” and “logic” separately, reducing the complexity of any single generation.

2.3 Physical Plausibility and Detail Density

In tests such as cloth simulation and mechanical assembly, the differences in model capability are mainly reflected in two aspects:

• Whether the physical approximation is reasonable: Whether an appropriate constraint-solving approach was chosen rather than simple keyframe animation;
• Whether part granularity is controllable: Whether the assembly can be decomposed into thousands of independent parts while maintaining readable hierarchical naming.

The number of parts itself is not the goal; a readable hierarchical structure is. An assembly with over two thousand parts, if poorly named and flat in hierarchy, is nearly impossible to maintain later. Therefore, evaluation should focus on naming convention compliance and hierarchy depth rather than simple counting.

3. Methods for Evaluating Cost and Efficiency

The token consumption and cost data mentioned in public cases provide a reference for cost evaluation. From an engineering perspective, however, the following metrics deserve more attention:

Metric	Meaning	Optimization Direction
Token consumption	Total processing volume to complete the task	Reduce无效 retries, reuse context
Iteration rounds	Number of rounds needed to reach a usable state	Front-load contract design, staged validation
Per-round output rate	Proportion of runnable code generated per round	Clarify interface boundaries, provide examples
Rework cost	Cost of modifying already-generated modules	Module decoupling, retain intermediate representations

It should be emphasized that cost varies greatly across tasks. A promotional short-video task that can be completed in five minutes and an open-scene prototype requiring hours of iteration have completely different cost structures. Evaluation should be stratified by task type rather than simple horizontal comparison.

4. Reusable Practice Path

Based on the above analysis, the following phased path is recommended:

4.1 Task Decomposition and Contract-First

Before formal generation, have the model output a task decomposition diagram and interface contract. The output of this step does not need to be runnable—only structurally clear. Once the contract is determined, all subsequent generation uses it as the baseline.

4.2 Staged Validation

Validate immediately after each stage’s output rather than waiting until everything is complete. For example:

• After scene generation, first validate coordinates and scale;
• After interaction logic generation, first validate state transitions;
• After resource packaging, first validate loading paths.

4.3 Retention of Intermediate Representations

Preserve intermediate artifacts such as structured descriptions, modeling scripts, and export files. When the final artifact needs adjustment, it can be regenerated from the intermediate layer rather than starting from scratch.

4.4 Cost Monitoring and Circuit-Breaking

Set token budgets and round limits for long-horizon tasks. When approaching the budget, trigger human intervention or degradation strategies to avoid unlimited retries.

5. Extended Considerations for Enterprise Scenarios

For enterprises that need to bring such capabilities into production environments, the following must also be considered:

• Reproducibility: Whether the same input stably produces similar results, which directly affects delivery quality;
• Auditability: Whether the generation process leaves sufficient logs for issue tracing;
• Security boundaries: Whether model-generated code undergoes static checks and sandbox validation;
• Team collaboration: How artifacts generated by multiple people are merged and how naming conflicts are avoided.

In cloud-native environments, generated artifacts can be incorporated into continuous integration pipelines, achieving an engineering closed loop through automated testing and artifact management. General-purpose capabilities such as object storage, container platforms, log services, and monitoring/alerting can all serve as hosting foundations, but the specific integration approach should be designed according to the actual architecture.

6. Summary

The performance of next-generation models in long-horizon code generation and multimodal tasks has moved from “can write code” toward “can orchestrate engineering.” The focus of evaluation and practice should therefore shift from single-point correctness to long-horizon consistency, cost controllability, and engineering maintainability. Through strategies such as contract-first design, staged validation, retention of intermediate representations, and budget circuit-breaking, teams can enjoy capability improvements while keeping delivery quality and cost predictable. For teams hoping to land these capabilities in actual business scenarios, it is recommended to start with medium-complexity tasks with clear boundaries, gradually accumulate patterns and toolchains, and then advance to more complex scenarios.