LLM Evaluation in Practice: Coding, 3D Modeling, and Financial Analysis

1. The Evaluation Shift: From Answering Questions to Getting Work Done

In the past, the most common way to evaluate a large language model was to run it through a batch of standardized tests: multiple-choice questions, reading comprehension, and math competition problems. These static benchmarks can reflect basic knowledge, but they struggle to answer the question developers actually care about: Can the model finish real work in a production environment?

As a result, the focus of evaluation is moving from static question answering to agentic tasks. You give the model a near-real production environment—an operational terminal, a scripting interface for modeling software, or a document library full of financial reports—and ask it to complete a task end to end through multi-turn interaction. The result is then scored automatically. Over the past two years, open-source and domestic models have rapidly closed the gap with leading closed-source models on this type of evaluation, and the competitive focus has shifted from knowledge coverage to task execution capability.

Regardless of which model is being evaluated, a rigorous agentic evaluation typically involves three components:

• Task environment: an isolated, reproducible execution environment such as a container, with the initial state preset.
• Automated scoring: objective signals such as whether tests pass or whether the output artifact is correct, rather than subjective human judgment.
• Cost measurement: tracking token consumption, interaction turns, and elapsed time for completing the task, then deriving the cost per completed task.

The third component is often overlooked but is critical for engineering decisions. When two models have similar scores but differ severalfold in unit cost, they may lead to completely different selection conclusions.

2. Terminal Coding: The First Proving Ground for Agentic Capability

The terminal environment is considered one of the most demanding scenarios for evaluating coding agents because it is realistic and unforgiving:

• Real tasks: fixing a bug, getting a build to pass, or adding a set of unit tests—these are things engineers do every day.
• Direct feedback: whether a command succeeds or a test passes is immediately visible, with no ambiguous gray areas.
• High fault tolerance requirements: the model must read error messages after a failed command, locate the cause, make changes, and retry. This tests long-horizon planning and self-correction.

The core evaluation loop can be abstracted into a minimal cycle:

# Minimal loop for terminal coding evaluation (pseudocode)
def run_case(case):
    env = BootEnv(case.setup_script)        # Start isolated environment and restore initial state
    agent = Agent(model, tools=["shell"])   # Expose only the terminal as a tool
    for step in range(MAX_STEPS):
        action = agent.act(env.observe())
        if action.type == "submit":
            break
        result = env.exec(action)
        agent.feedback(result)              # Feed stdout/stderr back to the model
    return grade(env.run_tests())           # Automatic scoring based on test results

In this loop, differences in model capability are amplified significantly. Weak models tend to get stuck in a cycle of changing one line and breaking something else, while strong models can converge to a working solution within a small number of turns.

Cost is the other side of the same coin. The industry now commonly treats the average cost required to achieve the same completion rate as a core metric alongside the score. A model with a slightly lower score but a fraction of the cost can be more valuable in engineering scenarios where tasks are run in bulk. Conversely, a model that is cheap but has a clearly insufficient completion rate will generate hidden costs through repeated retries that quickly erode its price advantage. You should evaluate the score and cost together rather than looking at either one in isolation.

3. 3D Modeling and Spatial Reasoning: Script-Driven Creation

If terminal coding tests the logical space, 3D modeling tests a model’s understanding of physical space: geometric relationships, proportions and scale, materials and lighting, and layout structure. This type of task places extremely high demands on spatial reasoning and long-chain tool use, which is why it has become an important proving ground for agentic capability.

There are two common engineering approaches:

1. Dedicated connector layers: building middleware or protocol adapters for each piece of software, with the model calling wrapped high-level interfaces.
2. Direct scripting: the model writes scripts directly against the target software’s scripting interface. Mainstream modeling software generally exposes Python-like scripting APIs, and the model operates the software the way an engineer would.

The second approach has clear and growing advantages:

• Broad applicability: any software that exposes a scripting interface can be covered without building individual adapter layers.
• Debuggability: script errors are directly visible, and the model can read them and make corrections on its own.
• Easy migration: the same loop of writing a script, executing it, observing the result, and refining it can be applied to different tools such as CAD software and renderers.

A typical script-based modeling workflow looks like this: build a rough model → refine geometry and materials → adjust layout and lighting → export the artifact.

# Script-driven modeling of a desk lamp prototype (pseudocode)
scene = Scene()
scene.add_cylinder(r=6, h=1)                     # Base
scene.add_cylinder(r=0.5, h=30).translate(z=1)   # Stem
shade = scene.add_cone(r=8, h=8).translate(z=32) # Lamp shade
shade.material.set(color="#F5E6C8", roughness=0.6)
scene.add_point_light(pos=(0, 0, 30), intensity=20)
scene.export("lamp.stl")                         # Export to a general-purpose 3D printing format

In terms of product experience, online preview, online editing, and export are becoming the standard pipeline for 3D generation capabilities. Users can view and adjust the model directly in a web page, and the finished product can be exported to a common printing format for 3D printing or rendered into a video or animated image for sharing. Behind this lies the challenge of stable integration between model output and the modeling software’s data structures, as well as the engineering maturity of the server-side rendering and file processing pipeline.

4. Financial Analysis: Retrieval, Verification, and Multi-Step Numerical Reasoning

Financial documents are widely recognized as difficult material: they are dense with tables, use mixed units (ten thousands vs. hundred millions, RMB vs. USD), require cross-document alignment of definitions, and are full of numerical relationships that require multi-step derivation. A competent financial analysis model must pass at least three tests:

• Retrieval: accurately locating relevant passages and tables across hundreds of pages of documents.
• Verification: cross-checking the consistency of the same metric across documents and correctly handling unit and currency conversions.
• Reasoning: completing multi-step numerical calculations, such as deriving profitability layer by layer from revenue, gross margin, and expense ratios.

For evaluation, the industry typically uses two types of tasks to characterize capability boundaries. The first is document question answering, which tests the combined accuracy of retrieval, cross-document verification, and multi-step numerical reasoning. The second is workflow-based tasks, which test how well a model supports a complete investment analysis process end to end, from initial screening and financial comparison to earnings estimation. The latter is closer to real business and is more effective at exposing breakdowns in long tasks.

Hallucination is a matter of survival in financial scenarios. A fabricated number or a misattributed source can cause real losses. Mitigation approaches that have proven effective in engineering practice include:

1. Citation constraints: requiring every key figure to be attributed to a source such as a document, page number, or table, with no citation meaning no use.
2. Numerical recomputation: independently recalculating the model’s computed results with a separate script and comparing.
3. Structured intermediate representation: first extracting tables into structured data, then reasoning over the data to avoid doing mental arithmetic directly from raw text.
4. Self-check and reflection: having the model review its key assumptions and number sources after reasoning.

Notably, the error rates of newer-generation models on hallucination stress tests have generally dropped significantly compared with previous generations. This is the combined result of reinforcement on verifiable answering during training and citation and verification mechanisms applied during inference. The industry broadly treats reducing the error rate to single-digit percentages as one of the key thresholds for commercial viability in high-risk scenarios like finance.

5. Why Post-Training Can Deliver Cross-Domain Gains

One phenomenon worth developer attention is that model version iterations—post-training on an existing base model—often bring simultaneous improvements across seemingly unrelated areas such as coding, 3D, and finance. This is not a coincidence. There is clear technical logic behind it:

• High-quality agentic trajectory data: supervised fine-tuning on multi-turn action sequences from real tasks that were completed successfully teaches the model how to get work done, not just how to talk.
• Reinforcement learning with verifiable rewards: using objective signals such as test pass rates and numerical accuracy as rewards to directly optimize end-to-end task success.
• Spillover of general reasoning capabilities: meta-capabilities such as planning, decomposition, and reflection naturally transfer to a wide range of vertical tasks once strengthened.

This also explains why you should not look only at individual benchmark scores when evaluating models. A model that stands out on long-horizon tasks often owes its advantage to underlying general capabilities that will continue to materialize across more scenarios over time.

6. Advice for Developers: Do Not Rely Only on Leaderboards—Build Your Own Evaluation Set

Public leaderboards are useful for a rough initial screening, but the real basis for model selection should come from your own business tasks. A minimal practical process:

1. Define tasks: pick 10 to 30 representative real tasks from your business, such as fixing an actual bug or analyzing a real financial report.
2. Set up scoring: prioritize signals that can be scored automatically, such as test passes, numerical comparisons, and key entity validation.
3. Run a small sample: fix the randomness and record completion rate, interaction turns, token consumption, and elapsed time.
4. Calculate unit cost: measure the average cost of completing a task and factor in the retry rate to get the true cost.
5. Measure hallucination: sample factual statements from the output and trace them back to sources for verification.
6. Roll out gradually: validate in low-risk scenarios first, then expand the scope step by step.

At the infrastructure level, all of these steps can be built from general-purpose components. Use a container platform to isolate task environments, object storage to manage corpora and artifacts, and logging, monitoring, and alerting services to track the details of each evaluation run. Major cloud platforms such as Baidu AI Cloud provide these general capabilities, and developers can combine them as needed rather than building everything from scratch.

7. Conclusion

From terminal coding and 3D modeling to financial analysis, the evolution of evaluation points in the same direction: use more realistic tasks, more objective scoring, and a more complete cost perspective to approximate a model’s true productivity. For developers, rather than chasing individual scores, it is more valuable to establish your own evaluation system as early as possible. It will not only help you choose the right model but also help you know where to focus your efforts next in an era of rapid model iteration.