Domestic LLM Terminal Agent Benchmark: A 3-Trillion-Parameter Model Nears International Leaders at a Fraction of the Cost

1. Why Terminal Environments Are a Critical Test for Agent Capabilities

In the evaluation landscape for coding agents, terminal environments remain one of the most demanding scenarios. Unlike constrained tasks such as code completion or single-file generation, terminal-based work requires an agent to perform a sequence of continuous operations in real command-line interactions: interpreting natural language instructions, planning multi-step execution paths, invoking system tools, parsing outputs, and self-correcting when errors occur.

The core difficulties fall into three areas:

1. Real-time environment perception: Terminal output is often streaming and unstructured. The agent must extract key signals from large volumes of text to determine whether a command succeeded or whether its strategy needs adjustment.
2. Open-ended action space: Unlike closed API calls, the number of executable command combinations in a terminal is nearly infinite. The agent needs sound pruning and decision-making capabilities.
3. Robust error recovery: Real environments are full of uncertainty—permission denials, incorrect paths, missing dependencies, network timeouts. The agent must be able to diagnose problems and roll back when needed.

For these reasons, benchmarks built around real terminal environments have become an important measure of how production-ready a coding agent is. A recent evaluation of terminal coding agents showed that a domestic 3-trillion-parameter model achieved a 65.7% task completion rate, closing the gap with a current international leader to less than one percentage point, while its inference cost was roughly one-fifth of the latter. This result deserves a closer technical breakdown.

2. Interpreting the Benchmark: What 65.7% Actually Means

In terminal-agent evaluation, task completion rate is the most direct core metric. It measures whether the agent can correctly complete a target operation within a limited number of steps after receiving an instruction. In real terminal scenarios, a 65.7% completion rate means:

• More than six out of ten tasks can be completed end-to-end without human intervention. For repetitive daily development work—environment initialization, dependency installation, log retrieval, batch file processing—this level already has practical value.
• The remaining roughly one-third of failures are concentrated in multi-step coupled tasks, scenarios requiring external state judgment, and operations involving complex permission management. These are precisely the boundaries that all current agent solutions face.

Comparing this result with peer solutions clarifies its position: the model trails an international leading model by approximately 0.7 percentage points, while outperforming a mainstream cloud-based solution by nearly 7.5 percentage points. It is important to emphasize that absolute benchmark values vary with task-set versions, environment configurations, and scoring standards. The more meaningful observation is therefore the relative ranking and cost-effectiveness ratio, rather than any single number in isolation.

3. The Engineering Significance of the 3-Trillion-Parameter Scale

Moving from hundreds of billions to three trillion parameters brings more than a numerical increase. It involves a series of engineering trade-offs.

3.1 Higher Capability Density

Terminal-agent tasks require a model to simultaneously handle code understanding, command generation, error diagnosis, and multi-turn planning. A larger parameter scale allows a single model to internalize more task patterns without relying heavily on extensive external tool orchestration. This reduces the complexity of the agent framework and lowers the latency and error accumulation that come with multi-component collaboration.

3.2 Structural Optimization of Inference Cost

Conventional wisdom holds that larger models cost more. But in practice, deployment cost depends on a combination of variables: parameter scale, activated parameter count, inference precision, batching strategy, and hardware utilization. This solution achieves near-leading completion rates while reducing per-task cost to about one-fifth, indicating systematic work on architectural efficiency and inference optimization.

For enterprise users, this cost difference is significantly amplified in large-scale automation scenarios. Assuming 100,000 terminal tasks executed daily, an 80% reduction in per-task cost directly changes the feasibility boundary of a project on an annual basis.

3.3 Improved Deployment Flexibility

A 3-trillion-parameter model still demands considerable memory and compute resources. However, compared with larger dense models, it offers more operational room in quantized deployment and distributed inference. This means enterprises can find a better balance among latency, throughput, and cost based on their own infrastructure conditions.

4. Typical Architecture of a Terminal Agent

To understand the benchmark results, it helps to first understand the basic workflow of a terminal agent. A typical system consists of the following stages:

User instruction → Task parsing → Planner → Action generation → Terminal execution → Output parsing
                ↑                                                    |
                └────────────── Feedback and correction ←────────────┘

Task parsing: Converts natural language instructions into structured task descriptions, clarifying goals, constraints, and available tools.

Planner: Decomposes complex tasks into an executable sequence of sub-steps. Planning quality directly determines the success rate of subsequent execution.

Action generation: Generates specific terminal commands or scripts for each sub-step. This stage places the highest demands on the model’s coding ability and domain knowledge.

Execution and parsing: Executes commands in a real or sandboxed environment, captures standard output and error output, and extracts key information.

Feedback and correction: When execution fails or output does not match expectations, the model needs to diagnose the cause and adjust subsequent actions. This is where agent capability differences become most apparent.

The solution that achieved a 65.7% completion rate in this benchmark applies targeted optimizations at multiple stages of the chain. For example, in the output-parsing stage, it combines structured extraction with semantic understanding to reduce reliance on fixed output formats. In the correction stage, it introduces a context backtracking mechanism based on historical actions to avoid repeating ineffective operations.

5. Cost-Effectiveness Analysis: The Logic Behind One-Fifth the Price

The cost advantage does not stem solely from the difference in parameter scale. It is the combined result of optimization at multiple levels:

Inference efficiency: Techniques such as sparse activation, quantized inference, and KV-cache optimization reduce the actual compute consumed per inference.

Task scheduling: Lighter inference paths are used for simple tasks, while the full planning pipeline is reserved for complex tasks, avoiding disproportionate resource use.

Caching and reuse: Caching mechanisms for high-frequency operation patterns reduce redundant inference.

Hardware adaptation: Operator-level optimization for different deployment environments improves hardware utilization.

For developers, this means that when selecting an agent solution, it is not enough to look at per-task completion rates alone. The more meaningful metric is effective cost—the total cost of completing a task, including inference fees, retry overhead, and manual fallback costs. Under this calculation, the cost advantage becomes even more pronounced.

6. Recommendations for Enterprise Adoption

Based on the analysis above, enterprises considering terminal-agent capabilities may find the following practical path useful:

First, define task boundaries clearly. Start with high-frequency, low-risk terminal tasks with clear steps, such as environment checks, log collection, and batch file operations. These tasks typically have above-average success rates, and the consequences of failure are manageable.

Second, establish sandboxing and rollback mechanisms. The open-ended nature of terminal environments means that incorrect operations can have real impact. It is advisable to execute agent-generated commands in containerized sandboxes and retain operation logs and rollback capabilities.

Third, set up human confirmation checkpoints. For tasks involving data deletion, permission changes, or production-environment operations, keep a human approval step and position the agent as an “operation suggester” rather than the “final executor.”

Fourth, continuously evaluate and iterate. Agent performance fluctuates as task distributions shift. Establish a regular evaluation mechanism to track completion rates, average step counts, and error-type distributions.

Fifth, pay attention to total cost of ownership. Include inference costs, operational costs, and human intervention costs in a unified accounting framework, rather than making decisions based solely on per-call pricing.

7. Conclusion

Coding agents in terminal environments are moving from experimentation toward practical use. A 3-trillion-parameter model achieved a 65.7% task completion rate in real terminal evaluation, narrowing the gap with an international leading solution to 0.7 percentage points while maintaining roughly one-fifth of the cost. This combination has clear engineering significance.

For developers and enterprise users, the key is not chasing a single benchmark number, but understanding capability boundaries, cost structures, and deployment constraints to find the right balance for their specific scenarios. As model efficiency continues to improve and agent frameworks mature, terminal automation is positioned to become a more reliable and economical foundational capability in development workflows.