From Long Context to Long-Horizon Tasks: LLM Evaluation Focus Is Shifting

From Long Context to Long-Horizon Tasks: LLM Evaluation Focus Is Shifting

Over the past year, the narrative around LLM iteration has been shifting. Early competition centered on parameter scale and context window length; today, what truly distinguishes a model’s engineering value is whether it can maintain goal consistency in long-horizon tasks. A long-horizon task here means a complex generation task that requires multi-step planning, state maintenance, and cross-module coordination—for example, producing an interactive 3D scene, a complete front-end project, or multi-stage business logic code in a single pass.

Recently, a frontier model released a new version that increased post-training investment on top of the existing base, focusing on coding, 3D creation, and financial research, with support for up to million-token context. An interesting observation from the discussion around it: the new version shows clear progress at the “first impression” level, but on hard metrics of task completion, it doesn’t necessarily lead the previous generation across the board. This offers a useful observation window—after the marginal gains from distillation change, in which dimensions do model capabilities improve, and in which do they regress?

1. Evaluation Design: Why an Offline, Single-File 3D Scene

This comparison used a deliberately demanding task: build a giant ring-shaped tidal gate, a terraced sea city, and a risable tide system in a single offline HTML file. The prompt explicitly required the composition, architecture, and materials to be established first, and specifically stressed that the rising and falling tide could not be simulated by simply raising and lowering a blue plane—ships, waterlines, and the flooding state of the plaza all had to change in sync.

The task is discriminating because it tests three capabilities at once:

• Spatial organization: Can the giant ring, bridge arches, beacon towers, and stacked residences be organized into a coherent landscape, rather than placing individual objects on terrain?
• Mechanism implementation: Do water reflection and tide-level changes have a real computational basis, rather than being visual approximations?
• Constraint adherence: Are quantified requirements in the prompt, such as “most of the stone steps should be submerged,” actually fulfilled?

Both model versions delivered a native WebGL2 single file with no external images, models, or third-party library dependencies. That means the difference is not “can it be done,” but “how complete and self-consistent the result is.”

2. Visual Gap: How Scene Hierarchy Is Organized

What really separates the two at a glance is the scene hierarchy under the default camera.

The previous generation’s giant ring was large, but the mountains on both sides took up more of the frame; the first screen was largely filled with nearby slopes and distant peaks, the city shrank deep into the bay, and the buildings and plaza were hard to make out. The ring stood in the center of the frame while the sea city required effort to identify—a classic case of “one focal point, everything else out of focus.”

The new version organizes the giant ring, bridge arches, beacon towers, and stacked residences into a coherent landscape: the buildings spread along both banks, the central plaza has clear concentric terraced levels, and the warm and cool textures of oxidized copper and stone are easier to distinguish. It comes closer to the “complete world” the prompt asked for, shedding the stiffness of placing a single ring-shaped object on terrain.

The essence of this difference is whether the model maintains a global spatial relationship graph during generation. The former tends to generate objects one by one and stitch them together; the latter is closer to planning the layout first and then filling in details.

3. Mechanism Difference: How Water Reflection Is Implemented

The water rendering mechanisms differ substantially.

The previous generation mainly simulated reflection through an analytic sky, sun specular highlights, and light trails—a screen-space visual approximation. The new version additionally introduces a mirror camera that re-renders the city and ships and samples the result onto the water surface, giving the reflections of shore buildings a real spatial source.

Pseudocode for the two approaches:

// Approach A: visual approximation
color = skyColor + sunSpecular + lightTrail
// Approach B: mirror camera re-rendering
mirrorCamera.reflect(waterPlane)
mirrorCamera.render(scene)   // re-render the city and ships
color = sample(mirrorCamera.texture, uv)

Approach B costs more compute, but the correspondence between reflections and objects is correct, and the “reflection detached from object” problem does not appear when the camera moves. This is an upgrade in implementation mechanism, not parameter tuning.

4. Constraint Adherence: How Well the Tide-Level Parameters Were Honored

Leading on visuals does not mean leading on task completion. On the hard metric of maximum tide level, the new version actually regressed.

Based on the step heights and maximum water level in the source code, the new version’s nine-level plaza terraces at high tide had only the lowest four levels below the mean water level—failing to fulfill the “most stone steps submerged” requirement. It was the previous generation, with the weaker first screen, that covered the outermost steps with its maximum water level range.

This reveals a key issue: as the model’s visual organization improves, its tracking of numeric constraints may weaken. When generation relies more on global aesthetic priors, local quantified conditions tend to be “averaged out.”

5. Implications for Engineering Practice

Several conclusions from this comparison are directly useful for developers:

1. Separate the “display layer” from the “contract layer.” Visuals, layout, and materials belong to the display layer, where models are improving fast; but dimensions, counts, thresholds, and linkage rules belong to the contract layer and need separate verification. List quantified constraints explicitly in the prompt and check them programmatically after generation.

2. Validate long-horizon tasks in segments. When generating a complex scene in one pass, a model may lose early constraints in later steps. Use a segmented strategy—“generate the structural skeleton first, then fill in module by module”—and re-check constraints after each segment.

3. Don’t judge models by visual impression alone. First-screen effects and default camera composition are highly misleading. Evaluations should cover boundary conditions such as maximum and minimum water levels, extreme camera angles, and concurrent state transitions.

4. Watch the direction of post-training investment. When post-training resources tilt toward coding, 3D creation, and similar areas, the model’s “expressiveness” in those domains improves noticeably, but general constraint adherence does not necessarily grow in step. Evaluate models by task type, not by overall leaderboard rankings.

6. Conclusion

The focus of LLM evaluation is shifting from “can it generate” to “can it hold its goals through long-horizon tasks.” The improvement in first impressions is real—it reflects progress in spatial organization and mechanism implementation. But the real test lies in the backend: the unglamorous numeric constraints, state linkages, and boundary conditions are what determine whether a model can enter production workflows.

For developers, the more practical approach is to build your own task-level evaluation set: use long-horizon tasks from real business scenarios, record pass rates for the display layer and the contract layer separately, and combine those with dimensions like context length and tool-calling stability to make an overall judgment. Only then can you pick the model that genuinely fits your scenario amid rapid iteration.