From Code Completion to Terminal Execution: How Coding Evaluation Is Shifting

Over the past year, the focus of LLM coding benchmarks has shifted noticeably. Early evaluations centered on function-level completion and single-file algorithm problems — tasks with clear boundaries and short context windows, where model performance quickly saturated. What draws more attention now is terminal task execution in realistic engineering environments: the model must plan, execute, correct errors, and deliver runnable results across multi-step interactions involving file systems, command lines, and dependency environments.

This type of evaluation puts three distinct pressures on a model:

• Long-range planning: maintaining goal consistency across dozens of steps without drifting off course;
• Tool invocation and feedback interpretation: command success or failure output becomes the basis for the next decision, so the model must parse it accurately;
• Error recovery: autonomously locating and fixing issues when faced with missing dependencies, incorrect paths, or syntax errors.

A recent post-trained version of an open-weight Chinese model has made meaningful progress in this direction. Building on post-training optimizations over its predecessor, it achieved a score of 65.7 on terminal-based coding tasks, placing it in the top tier of frontier models — behind only a leading closed-source model and ahead of several other frontier models in selected comparisons. The significance is not the single score itself, but what it signals: open-weight models can now compete head-to-head with closed-source frontier models on complex, agent-style coding tasks.

Cost Control: The Underrated Metric

Beyond raw performance, the metric engineering teams should pay closer attention to is cost. Terminal-style tasks involve heavy token round-trips and multiple tool calls, which can multiply the cost of a single task by an order of magnitude. According to official statistics, this model ranks as the most cost-efficient among comparable frontier models on terminal tasks.

It is important to distinguish two concepts:

Dimension	Meaning	Engineering relevance
Lowest absolute cost	Lowest per-token price	Suitable for simple, short-context tasks
Lowest unit task cost	Lowest total spend to complete a full task	Suitable for agentic, long-horizon tasks

Some lightweight models have lower unit prices, but because they require more trial-and-error rounds and consume more tokens on complex tasks, their total cost to complete the same job ends up higher. The optimization target that actually matters is unit task cost, not the number on a pricing page. This is especially critical for teams planning to integrate models into CI/CD pipelines, automated operations, or batch code repair — where task volume is high and each task involves many steps, cost differences compound quickly.

Engineering Adoption Recommendations

If you are evaluating this type of model for your development workflow, start by validating it in the following directions:

# Example: a framework for tracking cost and success rate in terminal-task-style invocations
task_stats = {
    "task_id": None,
    "steps": 0,          # Interaction rounds
    "input_tokens": 0,
    "output_tokens": 0,
    "success": False,    # Whether the final result is runnable
    "retry_count": 0     # Number of self-correction attempts
}
def evaluate(tasks):
    total_cost = 0
    success = 0
    for t in tasks:
        r = run_agent(t)
        total_cost += estimate_cost(r.input_tokens, r.output_tokens)
        success += int(r.success)
    return {
        "unit_task_cost": total_cost / len(tasks),
        "success_rate": success / len(tasks)
    }

Do not look only at latency and unit price for a single call. Instead, track unit task cost and first-pass success rate. These two metrics together reveal whether a model is economically viable in a real production pipeline.

3D Creation: From Tool Orchestration to Direct Script Generation

3D content creation has long faced a structural pain point: professional modeling software has a steep learning curve, while natural-language-driven generation struggles to precisely control geometry, materials, and layout. The previous common approach was to invoke modeling tools through an intermediary layer such as MCP, where the model output structured instructions that the middleware translated into software operations. The problem with this pipeline is that the intermediary layer itself becomes a capability bottleneck — it limits the range of operations the model can express and adds more points of failure.

Recent capability advances offer an alternative path: the model directly writes native scripts for modeling software, bypassing the middleware entirely to handle asset construction, detailing, layout adjustment, and material and geometry modifications. The value of this approach lies in three areas:

• Higher expressive ceiling: script APIs are generally more complete than wrapped tool interfaces, enabling finer-grained geometry and material operations;
• Shorter pipeline: fewer information losses and compatibility issues caused by middleware translation;
• Better reproducibility: the script itself is the artifact, making version control and batch reuse straightforward.

In standardized evaluations focused on modeling capability, this model achieved a score of 92.7 in tool-dependent scenarios, a narrow gap from the leading closed-source model’s range around 96. It currently ranks among the best open-weight models in this category. This gap is unlikely to affect usability in most practical creation scenarios — particularly in batch asset generation, scene layout adjustment, and parametric modeling, where the scripted approach offers clear efficiency advantages.

A Typical Script-Based Modeling Workflow

A practical script-based modeling pipeline usually involves four stages:

1. Intent parsing: decomposing natural language descriptions into structured elements such as geometry, dimensions, positions, materials, and lighting;
2. Script generation: outputting executable script code for the target software, including object creation, transforms, and material assignment;
3. Execution and validation: running the script and checking whether object counts, bounding boxes, and material references match expectations;
4. Iterative refinement: adjusting script parameters based on render previews or validation results.

Illustrative pseudocode:

# Example: structure of a parametric scene-building script
def build_scene(spec):
    scene = create_scene()
    for obj in spec["objects"]:
        mesh = create_mesh(obj["primitive"], obj["dimensions"])
        apply_transform(mesh, obj["location"], obj["rotation"], obj["scale"])
        assign_material(mesh, obj["material"])
        scene.add(mesh)
    setup_lighting(scene, spec["lighting"])
    setup_camera(scene, spec["camera"])
    return scene
spec = {
    "objects": [
        {"primitive": "cube", "dimensions": [1, 1, 2],
         "location": [0, 0, 1], "rotation": [0, 0, 0],
         "scale": [1, 1, 1], "material": "matte_gray"},
        {"primitive": "plane", "dimensions": [10, 10, 0],
         "location": [0, 0, 0], "rotation": [0, 0, 0],
         "scale": [1, 1, 1], "material": "floor_tile"}
    ],
    "lighting": {"type": "area", "intensity": 800},
    "camera": {"location": [5, -5, 5], "target": [0, 0, 1]}
}

The advantage of this pattern is that the script is a readable, editable engineering artifact that fits into version control, rather than a one-shot black-box result. Teams can build their own scene template libraries, turning common layouts and material combinations into parameterized scripts that only require parameter tweaks for reuse.

Why Spatial Reasoning Matters

The spatial reasoning demands of 3D creation are often underestimated. A model must not only recognize “a cube,” but also understand whether its position relative to other objects, occlusion relationships, and scale proportions are sensible. High scores on modeling-focused evaluations reflect precisely this improvement in structured spatial understanding. For creators, this means they can express qualitative requirements in natural language — “place the main object at the left third of the frame, leave depth in the background, use a matte material” — and the model can translate them into concrete coordinates, rotations, and material parameters.

Model Selection and Deployment: How to Evaluate These Upgrades

When facing model capability updates, engineering teams tend to fall into two traps: looking only at leaderboard scores, or looking only at unit price. A more sensible evaluation framework should include:

• Task fit: terminal tasks, script generation, and spatial reasoning correspond to different capability profiles and should be selected based on your use case;
• Unit task cost: calculated with retry rates and interaction rounds factored in;
• Controllability and reproducibility: scripted output is preferable to free-form natural language output, as it supports auditing and rollback;
• Integration cost: whether the model supports standardized invocation patterns and fits into existing pipelines and monitoring systems.

At the infrastructure level, long-horizon task-oriented workloads place high demands on containerized execution environments, log collection, and invocation chain monitoring. A practical approach is to wrap model calls as a standalone service, run it in isolated containers via a container platform, persist input and output at each step through a logging service, and track success rates and cost fluctuations with monitoring and alerting. This turns raw model capability into stable engineering throughput.

Overall, the signal from this round of capability upgrades is clear: open-weight models are moving from “able to converse and write snippets” to “able to execute complete engineering tasks and directly produce reusable scripts.” For developers and enterprises, the real opportunity is not chasing the latest leaderboard ranking, but embedding these capabilities into their own toolchains as early as possible and validating their return on investment in real business scenarios with measurable methods.