Chinese LLMs Enter a New Race: Dual Breakthroughs in Reasoning and Agent Capabilities
The competition among Chinese large language models is shifting from parameter scale to two capability tracks: strong reasoning and agent execution. This article examines how evaluation is moving from knowledge-oriented benchmarks like MMLU and GSM8K to execution-focused benchmarks such as ALE, where lightweight models are achieving notable Pass@1 results; the deepening engineering demands of code generation; the three technical pillars behind agent performance—planning, memory, and function calling; and practical guidance for evaluating and selecting lightweight models for real-world deployment.
A recent wave of lightweight models focused on vertical capabilities has emerged from Chinese LLM vendors. The competitive focus has shifted from sheer parameter scale to deep refinement of code generation (Coding) and agent (Agent) execution capabilities. The industry trend behind this: a model’s “intelligence” and its “ability to get things done” are now being evaluated separately, and enterprises have especially high expectations for the latter.
Evaluation Benchmarks Shift from Knowledge Q&A to Task Execution
For a long time, LLM capability assessment relied on knowledge- and reasoning-oriented benchmarks such as MMLU and GSM8K. These tests effectively measure a model’s “knowledge reserves,” but they cannot answer a key question: can the model work like a junior engineer—understand a vague goal, break it down into steps, and ultimately deliver a result?
This is the backdrop for a new generation of evaluation systems such as ALE (Agents’ Last Exam). Rather than simply asking questions, this benchmark builds simulated task environments with explicit goals, lists of available tools, and complex constraints. Models must autonomously plan sequences of actions and progressively complete tasks by calling tools such as code interpreters, search engines, and file operation interfaces.
On ALE’s Pass@1 metric (the proportion of tasks completed within a single attempt), a new generation of lightweight models has achieved notable breakthroughs. For example, M3.1 Flash from one Chinese vendor achieved a 29.6% pass rate, ahead of Kimi K3 from another major LLM provider at 28.3%. The gap looks small, but in execution tasks, every additional percentage point represents a quantifiable qualitative improvement in intent understanding, step planning, or tool-calling accuracy.
From a developer’s perspective, this shift in evaluation dimensions is critical. If a model can reach a roughly 30% one-shot success rate in scenarios such as automated e-commerce price comparison, cross-app schedule coordination, or generating runnable scripts directly from natural-language requirements, it has crossed the basic threshold for entering production environments and assisting humans with real work.
Code Generation: The Hidden Contest of Engineering Thinking
Although agentic capability attracts more attention on leaderboards, code generation remains the foundation that supports complex agents. A model that cannot write high-quality code will struggle to generate correct query syntax during tool calls, let alone handle advanced tasks such as “write a script to process data.”
Today’s top models are competing in the “deep water zone” of code semantics. This is no longer just about matching syntax templates; it requires the model to demonstrate:
- Repository-level context awareness: understanding cross-file structural dependencies rather than generating a function in isolation.
- Defect diagnosis and repair: given an error message, locating the root cause and proposing a fix patch, instead of simply regenerating a similar chunk of code.
- Performance sensitivity: considering time and space complexity when choosing algorithms, and avoiding brute-force, inefficient solutions.
Notably, lightweight Flash-class models are catching up fast in coding ability. Through distillation and alignment training on high-quality code corpora, these models can cover most everyday development scenarios while maintaining low inference latency. For generating non-core internal business scripts or writing data-cleaning rules, such models already offer strong practical value.
The Technical Levers of Agent Deployment: Planning, Memory, and Tool Protocols
Leading on the ALE leaderboard comes down to how well a model coordinates the “three pillars of agents”:
1. Task Planning (Planning)
The model must decompose a complex top-level instruction (such as “research mobile phone market trends and generate a report”) into a chain of subtasks (search industry news -> extract key metrics -> generate charting code -> summarize findings in text). This depends on the reasoning-enhancement mechanisms behind the model: through an internal process similar to chain-of-thought (CoT), each step’s output serves the final goal.
2. Memory Management (Memory)
Real agent tasks usually involve multi-turn interaction. The model needs to maintain memory of previous results within a workspace—whether retrieved knowledge fragments or variable names defined in code it has already generated. Expanded context lengths and key-information retrieval mechanisms play a crucial role here.
3. Tool Calling Protocols (Function Calling)
This is the most decisive variable separating models in recent evaluations. The model must understand the JSON Schemas that external systems expose via APIs and generate call requests that precisely satisfy parameter constraints. The advantage of new-generation models lies not only in avoiding distractor tools when selecting which to call, but also in dynamically correcting errors through natural-language feedback (for example, when a tool returns an error).
A Practical Guide to Evaluating and Choosing New Lightweight Models
With new models appearing constantly, developers evaluating them should focus on three dimensions:
- Don’t rely on a single leaderboard: Even a dynamic benchmark like ALE carries the risk of test-set saturation. Consider a combined view across code-competition benchmarks (e.g., HumanEval), real programming task benchmarks (e.g., SWE-bench), and comprehensive agent benchmarks.
- Weigh the cost-effectiveness of local deployment: The core selling point of Flash-class models is low cost. When building your own private cloud environment, benchmark throughput (tokens per second) and VRAM usage across different GPU specifications to find the optimal balance of performance and business cost.
- Build a business-level test set: Generic benchmarks cannot cover the terminology and operational logic of a specific industry. Extract 20 to 50 high-frequency, real operational scenarios from your enterprise knowledge base and build an internal evaluation set for regression testing—with particular attention to how many retries the model needs after filling in wrong tool parameters.
Outlook: The Last Mile from Usable to Genuinely Useful
Although the numbers on the ALE leaderboard are encouraging, models remain far from fully replacing human work. Current agent models are still weak in handling the stability of ultra-long-horizon tasks (spanning multiple days) and in exploration strategies for unfamiliar environments.
That said, the direction of technical evolution is clear: model vendors are embedding more engineering frameworks—such as retry mechanisms and human-in-the-loop approval flows—into their model service layers. For enterprise users, now is the ideal time to shift attention from “which model has the largest parameters” to “which model I can actually put to work in my business.” The model families that can excel in both coding and agent execution will be well positioned to lead the next wave of AI application growth.
