From Code Agents to Direct 3D Scripting: Evaluating Vertical Capabilities of Large Language Models

1. The Shift in Evaluation Focus: From Knowledge QA to Real-World Execution

In recent years, the primary metric for measuring the capability of large models has been accuracy on knowledge-based QA benchmarks, focusing on parameter scale and pre-training data quality. However, as technology evolves, simply “knowing” is no longer enough to create differentiation; “doing” has become the new watershed. The evaluation focus is clearly shifting toward three types of tasks: multi-turn interactive terminal coding tasks, 3D content creation relying on spatial reasoning, and professional domain analysis requiring cross-document verification and numerical reasoning.

The technical driver behind this shift is post-training. Base models first learn instruction following through supervised fine-tuning, and then master tool usage, long-range planning, and error recovery through reinforcement learning based on verifiable rewards. Notably, after undergoing the same post-training process, the gap between open-source models and closed-source flagships on frontier tasks has significantly narrowed, and in some dimensions, they can compete head-to-head. This means that when enterprises select models, “open-source or closed-source” is no longer primarily a capability issue, but a comprehensive trade-off of cost, compliance, and deployment form.

2. Terminal Coding Tasks: The Litmus Test for Agent Capabilities

2.1 Why Terminal Tasks Are Difficult

Coding tasks in a terminal environment are regarded as a hard metric for testing agent capabilities because they are fundamentally different from “generating a piece of code in a single turn”:

• Long-range dependencies: A task is often broken down into dozens of steps. Any deviation in an early step will propagate backward.
• State management: File systems, processes, and environment variables constantly change. The model must maintain an accurate perception of the environmental state.
• Non-determinism: Real outputs from compilation errors, dependency conflicts, and test failures vary wildly. The model needs to read errors, locate causes, fix them, and retry.

2.2 Evaluation: Completion Rate vs. Cost

When observing these evaluations, it is recommended to focus on three metrics simultaneously: task completion rate, average interaction steps, and cost per task. The cost dimension is often overlooked but directly determines whether a model can be used in production. A simplified formula can be used for estimation:

# Expected cost per task (illustrative)
expected_cost = (tokens_in * price_in + tokens_out * price_out) \
                * (1 + retry_rate) + human_review_cost * review_rate

For a cheap but less capable model, the retry rate and manual review rate will significantly increase the total cost. A capable frontier model, even with a higher unit price, often yields a greater total cost advantage due to its high first-pass success rate. In recent public evaluations, open-source models have achieved completion rates in terminal tasks on par with top-tier closed-source models, while keeping unit task costs at a highly competitive level. This is a strong signal that open-source models are becoming production-ready.

3. Direct 3D Scripting: A Creative Paradigm Bypassing the Middle Layer

3.1 Comparison of Two Technical Routes

For involving large models in 3D creation, the previous mainstream path was “model + tool calling protocol”: the model outputs structured instructions, which are translated by a middle layer to drive the modeling software. The cost is high adaptation effort, potential protocol translation distortion, and task interruption if any single环节 fails. The recently emerged “direct scripting” route is more straightforward: the model bypasses the middle layer and directly writes script interface code for mainstream modeling software. A sandbox executes the code to build and refine assets, adjusting layout, materials, geometry, and other elements.

Dimension	Tool Calling Protocol	Direct Scripting
Pipeline Length	Model -> Protocol -> Adapter -> Software	Model -> Script -> Sandbox -> Software
Debuggability	Relies on middle layer logs	Scripts can be versioned and replayed
Error Recovery	Retries at the protocol layer	Model reads errors and self-repairs
Coverage	Limited by preset tool sets	Theoretically covers the full script API set

3.2 Why It Is Feasible Now

Direct scripting is essentially the “spillover” of code generation capabilities into vertical domains. Modeling scripts are, first and foremost, code that follows strict syntax and API conventions. The massive amount of modeling scripts accumulated by the community over the years naturally forms training data. The enhancement of spatial reasoning allows models to correctly handle constraints among coordinate transformations, geometric boolean operations, hierarchical structures, and material parameters. The combination of these three factors makes it possible for models to independently complete tasks like “building an indoor forest scene in a terminal” or “parametrically constructing a vortex-structured engine model.”

3.3 Design Essentials for the Execution Loop

For production implementation, it is recommended to organize the workflow with the following loop:

# Direct scripting execution loop (pseudo-code)
plan = llm.plan("Build an indoor forest scene, including lighting and materials")
for step in plan:
    script = llm.emit_script(step)                 # Directly output executable scripts for modeling software
    result = sandbox.run(script, timeout=60, mem_limit="4g")
    if result.failed:
        script = llm.repair(script, result.stderr) # Self-repair based on error output
    scene.merge(result.artifacts)
scene.export("scene.glb")                          # Export to a universal format

Key engineering guardrails include: scripts must be executed in a resource-constrained sandbox; each step’s artifact must undergo geometric validity checks (normal orientation, non-manifold edges, intersection detection); and complex scenes should retain checkpoints for rollback.

3.4 Parametric Geometry: A Minimal Sample of Direct Scripting

The core of direct scripting is translating natural language descriptions into precise parametric geometric calculations. Below is an illustrative implementation for generating grid vertices of a spiral structure, achievable with a standard library:

import math
def helix_surface(u_steps=120, v_steps=40, radius=1.0, pitch=0.5):
    """Parametric helix surface: u is circumferential angle, v is radial gradient"""
    verts = []
    for i in range(u_steps):
        theta = 2 * math.pi * i / u_steps
        z = pitch * theta / (2 * math.pi)          # Spiral lift
        for j in range(v_steps):
            t = j / (v_steps - 1)
            r = radius * (0.4 + 0.6 * t)           # Radius gradient along radial direction
            verts.append((r * math.cos(theta), r * math.sin(theta), z))
    return verts

In real-world tasks, the model must also perform triangulation, boolean union, beveling, and UV unwrapping on this basis. Each step is a dual test of spatial reasoning and API proficiency.

3.5 From Model Files to Deliverables

Models generated via direct scripting can be exported as universal exchange formats (such as OBJ, glTF/GLB, STL) to connect to three common delivery pipelines: browser-based online preview and editing, slicing output for 3D printing, and rendering export for sharing as videos or dynamic images. Setting up this pipeline on the cloud requires standard combinations of general cloud capabilities: object storage for model files, container or function compute for running sandboxes and rendering, and monitoring/alerting for stable task queues. For instance, Baidu AI Cloud offers product support for these categories.

4. Financial Document Analysis: The Hard Battle of Retrieval, Verification, and Multi-Step Numerical Reasoning

4.1 Why the Task Is Difficult

Real-world financial document analysis demands far more from models than general QA: scanned documents have complex layouts and nested tables; the statistical scope of the same metric may vary across documents; and questions often require cross-document data extraction, unit conversion, and multi-step numerical reasoning. A mistake in any single step will distort the final conclusion.

4.2 Capability Composition

The industry generally breaks this down into three layers of capability: layout and table understanding (parsing multi-level headers and merged cells), retrieval and cross-document verification (locating evidence and maintaining citation consistency), and numerical reasoning (performing precise calculations within the reasoning chain rather than “mental math”). According to public evaluations, top-tier models have achieved leading scores on financial document QA benchmarks and are approaching optimal performance in evaluations covering the complete investment workflow from “target screening to profitability analysis.” This indicates that the combined capability of “retrieval + verification + reasoning” is maturing rapidly.

4.3 Hallucination Suppression: From Double-Digit to Single-Digit Error Rates

Financial scenarios demand near-zero tolerance for hallucinations. Effective engineering suppression methods include: forcing answers to include original text citations and performing consistency checks; connecting independent calculators or code interpreters to re-calculate numerical conclusions; and building high-difficulty stress test sets for continuous regression. From an industry trend perspective, the error rate of frontier models in hallucination stress tests has been reduced from the previous generation’s double-digit percentage to single digits. This level of improvement is sufficient to make a “model initial screening + manual review” workflow viable in risk-sensitive industries.

5. A Rational Approach to Benchmark Scores: A Four-Step Evaluation Methodology

Faced with densely released leaderboards, enterprises are advised to filter them through four steps:

1. Task Proximity: Does the benchmark’s task distribution align with your own business? High scores in terminal evaluation do not equate to high scores in frontend generation.
2. Condition Sensitivity: The same model may have vastly different scores under “tool-dependent” versus “bare-run” conditions. Collect data based on real usage conditions.
3. Cost Caliber: Verify the token consumption and retry counts corresponding to the scores, and convert them into unit task costs for comparison.
4. Reproducibility: Prioritize reproducible public evaluations, build your own business evaluation sets, and verify actual gains through A/B testing.

6. Implementation Recommendations and Outlook

For enterprise developers, the currently actionable path is: cut into 3D asset production and terminal automation via direct scripting; enter financial document scenarios using “citation tracing + numerical re-calculation”; build engineering guardrails with sandboxes, validation, and rollbacks; and continuously calibrate model selection using proprietary evaluation sets. It is expected that as post-training methods continue to iterate, the capability curves of code, 3D, and professional domain analysis will trend sharply upward. The competition between open-source and closed-source will further drive down the cost per unit of intelligence—making this the optimal window for practitioners to start implementing these technologies.