Chinese Large Language Models: Agent Capability Upgrades and Falling Inference Costs
This article examines the shift of Chinese large language models from conversational assistants to agents capable of executing tasks, explains how their benchmark performance is approaching leading international models in terminal-operation and engineering-assistance evaluations, and analyzes the technical drivers behind lower inference prices. It also provides practical guidance on evaluating benchmarks, designing cost-aware model routing, building tool and sandbox infrastructure, measuring agent trajectories, and maintaining safety and compliance.
Introduction: From Answering Questions to Completing Work
Over the past period, Chinese large language models have entered a dense release cycle. Model portfolios are becoming more segmented, capability positioning is clearer, and inference prices continue to decline. The reductions are especially pronounced for smaller models targeting high-frequency, lightweight scenarios. Alongside lower costs, agent capabilities have advanced significantly: models no longer merely generate text. They can understand goals, call tools, read environmental feedback, adjust strategies, and, when necessary, acknowledge failure or request human intervention.
Some vendors report that on terminal-operation benchmarks, certain Chinese models score above 60%, roughly on par with top international models and, in individual tasks, ahead of some international flagship models. The gap with leading models has narrowed to a few percentage points. Similar trends appear in engineering-assistance benchmarks, such as CAD. For enterprises, the practical implication is that scenarios previously requiring budgeting and evaluation against top international models can now often be delivered at lower cost with Chinese models and comparable quality. However, benchmark scores are only one dimension. Stable production deployment depends on understanding model capability boundaries, cost structures, and engineering constraints. This article follows that thread.
1. Rethinking Benchmarks: What Terminal-Operation and Engineering-Assistance Benchmarks Show
1.1 Core capabilities measured by the two benchmark categories
Traditional language model benchmarks mainly measure single-turn accuracy, perplexity, knowledge coverage, and language expression. Agent benchmarks change the unit of evaluation from a single answer to a complete task execution. A common setup places the model in an isolated environment with a real file system, command line, or engineering software, then requires multi-turn interaction until the task is completed or explicitly fails. These evaluations usually require automatic verification, such as all tests passing, a successful build, or artifacts meeting predefined constraints. The two representative categories assess the following capabilities:
- Terminal-operation benchmarks: The model performs environment configuration, dependency troubleshooting, script writing, and debugging in command-line and file-system environments. Key assessment points include:
- Long-horizon planning;
- Understanding and diagnosing real error messages;
- Accuracy of tool use;
- State preservation across steps.
- Engineering-assistance benchmarks: The model completes tasks in CAD-like software, such as modeling, assembly, parameter adjustment, or change recognition. These benchmarks assess both 3D spatial understanding and domain knowledge, as well as interface operation or multimodal input parsing, usually with tool assistance.
1.2 Three key observations
First, the leading tier has formed, and capability gaps are narrowing. In the above benchmarks, top Chinese models and top international models show mixed results, with differences mostly within a few percentage points. For most enterprise applications, gaps of this magnitude can be mitigated through prompt design, tool wrapping, workflow orchestration, retry mechanisms, and validation. They are no longer decisive factors.
Second, tools and environments often matter more than the model itself. In terminal-operation evaluations, models with tool calling, sandbox execution, and result-feedback mechanisms score significantly higher than models without tool support. Engineering-assistance scores are also explicitly reported as results obtained with tool assistance. This indicates that agent success depends heavily on the coordinated system of model, tools, and environment. A model that can call tools and understand feedback, combined with well-designed tool interfaces and clear error messages, often improves end-to-end success more than simply increasing model size.
Third, benchmark scores have inherent comparability limits. Several caveats apply:
- Public leaderboards are often disclosed by model vendors themselves and may differ in sample selection and prompting strategies, so scores should be treated as reference points;
- Evaluation scaffolding includes prompt templates, tool definitions, retry policies, and maximum step counts. Different scaffolds can produce several percentage points of variation for the same model;
- Evaluation environments differ from real business environments, creating a gap between benchmark performance and production behavior.
Therefore, selection should treat public leaderboards as a starting point and use proprietary business evaluation sets as the basis for decisions.
2. Changes in Cost Structure: Technical Drivers Behind Mid- and Low-Tier Price Reductions
Falling inference prices are not merely commercial promotions; they are supported by solid technical and engineering advances.
2.1 Sparse activation and lightweight architectures
Most mainstream models now adopt mixture-of-experts (MoE) architectures. Each layer contains multiple expert subnetworks, but only a few are activated for each token. Compared with dense models of similar scale, this substantially reduces the number of parameters participating in computation per token, lowering compute consumption and memory-access pressure during decoding. When combined with attention designs such as grouped-query attention (GQA), which reduce KV cache size, and lightweight small models optimized for serving workloads, the per-token service cost can be reduced further.
2.2 Inference engines and system-level optimization
- Low-bit quantization: Compressing weights and activations from FP16 to INT8/INT4 reduces memory usage and memory-bandwidth pressure. Because decoding is often constrained by memory bandwidth, this optimization can be especially effective.
- Speculative decoding: A lightweight draft model or shallow structures within the model itself generate several candidate tokens continuously, and the target model verifies them in parallel. This reduces the number of sequential decoding steps and trades a small amount of extra computation for substantially lower latency.
- Continuous batching and paged KV cache management: These improve GPU utilization, allow more requests to share the same compute resources, and reduce per-request costs.
2.3 Context cache reuse
In agent tasks, system prompts, tool definitions, and prior conversation prefixes are often repeated across consecutive calls. Prefix-based KV cache reuse, commonly referred to as context caching, can reuse the computed intermediate states for these prefixes, reducing redundant computation and lowering first-token latency and cost in long-context scenarios. In agent loops involving dozens of interactions, the cumulative benefit can be substantial.
2.4 Scale effects and ecosystem maturity
The open-source model ecosystem is becoming more complete. Inference frameworks and hardware adaptation continue to improve, and the utilization of training and inference infrastructure is increasing. A mature supply chain and competition among vendors are jointly pushing prices downward. This is also the fundamental reason enterprises no longer need to assume that only flagship models are suitable.
3. Agent Capability Upgrades: From Model Capability to System Engineering
Model-level progress is mainly reflected in the following areas:
- Instruction following and structured output: Models can reliably produce structured results that conform to JSON Schema, making downstream parsing easier.
- Tool calling: Models can call multiple tools in parallel within a single turn and can correct their behavior based on error feedback rather than mechanically repeating the same command.
- Long context and attention efficiency: Long context windows can accommodate code repository fragments, long documents, and long histories while preserving key information from being diluted.
- Inference-time scaling: On complex problems, models can generate longer internal reasoning processes, trading controllable additional inference cost for higher accuracy.
- Multimodal understanding: Models can read screenshots, interfaces, and drawings, providing input for GUI operation and engineering-assistance tasks.
However, running agents in production requires more than model capability. It also requires a set of engineering mechanisms:
- Planning strategies: ReAct-style “thought-action-observation” loops are simple and flexible, suitable for exploratory tasks. Plan-and-Execute first generates a complete plan and then executes it step by step, making it easier to review and retry after failures, suitable for clearly defined, multi-step processes. The two can also be combined.
- Tool gateway: All external tools should be uniformly registered, validated, and audited, as described in the tool-design section below.
- Sandbox isolation: Code execution, file operations, and command-line operations should occur in isolated environments with resource and time limits.
- Memory management: Distinguish working memory from long-term memory. Long-term memory should be recalled on demand through vector retrieval rather than inserting all history into the context.
- Observability and evaluation loops: Record traces, token consumption, latency, and cost, and continuously perform regression validation.
4. Selection and Architecture Practices
4.1 Model tier routing: using the right model for each task
Classify tasks by complexity and try them in ascending order, from lighter to heavier. Simple tasks can be handled directly by smaller models, while more difficult tasks are escalated to more capable models. The following is a simplified example:
from dataclasses import dataclass@dataclassclass Task:prompt: strschema: dict | None = None # Expected structured output (JSON Schema)async def run_with_cascade(task: Task, models: list[str], retries: int = 1):"""Call models in increasing capability order; return an error and log traces if all fail."""for model in models: # e.g., small model -> general model -> reasoning-enhanced modelfor attempt in range(retries + 1):resp = await llm.complete(model, task.prompt, schema=task.schema)if resp.is_valid and resp.confidence >= 0.9:log_trace(task, model, attempt, resp.tokens, resp.cost)return resplog_trace(task, model, retries + 1, resp.tokens, resp.cost)raise TaskFailed("All tiers failed validation") # Escalate to manual handling or alerting
When implementing this approach, pay attention to the following:
- Validation first: Each tier’s output should first pass format validation and business-rule validation before deciding whether to escalate.
- Budget control: Set limits for the maximum number of tiers, maximum steps, maximum tokens, and timeout to avoid runaway cost and latency caused by repeated retries.
- Metric linkage: Continuously monitor escalation rates. A rising escalation rate may indicate that prompts for smaller models need optimization or that task difficulty has changed.
4.2 Context compression and tool design
- Context budgeting: Define token limits for the system prompt, tool definitions, retrieved fragments, and historical traces. When limits are exceeded, summarize earlier traces first.
- Retrieval instead of full injection: Load code repository and knowledge base content on demand through vector or keyword retrieval rather than inserting large sections into the prompt.
- Tool interfaces should be readable, actionable, and fallback-friendly: Describe parameters with structured schemas; include status codes, key information summaries, and human-readable error messages in return values; require explicit confirmation for destructive operations and provide rollback or compensation mechanisms.
- Batch and asynchronous execution: For batch processing, choose models based on priority and cost; tool calls should be executed in parallel whenever possible to reduce serial waiting.
TOOL_SCHEMAS = [{"name": "search_docs","description": "Search internal documents by keyword and return snippets with sources","parameters": {"type": "object","properties": {"query": {"type": "string"}, "top_k": {"type": "integer"}},"required": ["query"],},},# Other tools: create tickets, query metrics, submit builds...]async def agent_step(state, tools: dict, max_steps: int = 20):for step in range(max_steps):action = await llm.decide(state, TOOL_SCHEMAS) # Structured output: tool name + parametersif action.type == "final":return action.resultif action.tool not in tools:feedback = f"Unknown tool: {action.tool}. Available tools: {list(tools)}"else:try:feedback = await tools[action.tool].invoke(**action.args, trace_id=state.trace_id)except ToolError as e:feedback = f"Tool execution failed: {e}. Adjust the parameters or choose another tool."state = state.append(step, action, feedback) # Append the action-observation traceraise StepLimitExceeded()
4.3 Trajectory-level evaluation: turning evaluation sets into engineering assets
Enterprises should build business evaluation sets containing representative tasks, boundary conditions, and adversarial samples. Evaluation should be scored by trajectory rather than by a single answer:
TASKS = [...] # Business tasks: normal cases, boundary conditions, adversarial samplesdef evaluate(task, agent):trace = agent.run(task)score = {"success": trace.final_state == task.expected_state, # End-to-end result is correct"steps": trace.n_steps, # Interaction steps; reflects efficiency"tokens": trace.total_tokens, # Token usage"cost": sum(u["tokens"] * u["price"] for u in trace.usage),"tool_errors": trace.tool_errors, # Number of tool failures"latency_ms": trace.latency_ms,}score["efficiency"] = (score["success"] * 100 - score["steps"]) / score["steps"]return score
Based on these metrics, teams can answer three key questions:
- How much does success rate decline after switching to a lower-cost small model?
- Can the success-rate improvement from additional retries or longer context cover the extra cost?
- Which failures are concentrated in specific tools or specific task types, and should teams fix the tools first or switch models first?
4.4 Security and compliance
Agents can operate real files, code repositories, and business systems, so their risks are far greater than those of pure conversational scenarios:
- Code execution, production system operations, and actions that delete or overwrite data should occur in isolated sandboxes with restricted file-system and network access;
- Credentials for tools should be assigned according to the principle of least privilege, with secondary confirmation and human approval introduced for sensitive operations;
- Untrusted data from external web pages, files, ticket contents, and similar sources should be treated as potential prompt-injection sources and handled separately from system instructions;
- Traces and operation logs should be retained completely to support auditing, accountability, and incident review;
- Regression evaluation must be completed before model upgrades or switches to avoid situations where metrics improve while production incidents increase.
4.5 Cloud infrastructure support
Agent systems impose new requirements on infrastructure: high-concurrency, low-latency inference services, stable model version management, and fine-grained usage metering. When building on cloud infrastructure, the following general capabilities can be considered:
- Model services: Provide a unified calling entry for models of different sizes, supporting tiered routing and gradual rollout;
- Container platforms: Flexibly host agent services and sandbox runtimes;
- Message queues: Decouple asynchronous tasks and smooth traffic peaks;
- Object storage and vector retrieval services: Store traces, knowledge, and memory data;
- Logging, monitoring, and alerting: Observe success rates, latency, and cost.
Baidu AI Cloud provides relevant capabilities in these areas. Organizations should choose deployment and governance methods based on their business scale and compliance requirements.
Conclusion
The performance of Chinese large language models on agent benchmarks shows that the gap among top models has narrowed to a few percentage points. At the same time, continued price reductions in mid- and low-tier models make tiered model usage a more cost-effective choice. Enterprises should therefore shift their focus from “which model to choose” to “how to use models effectively.” The core approach can be summarized as follows: classify models by task complexity, use tools and environments to compensate for capability gaps, continuously calibrate the balance among quality, cost, and latency through trajectory-level evaluation, and maintain security baselines through sandbox isolation and permission governance.
