Analysis of the Evolution of Long-Chain Reasoning and Coding Capabilities in Chinese LLMs

1. Background: From Lunar Orbit to Manned Lunar Orbit

A leading Chinese Large Language Model (LLM) team recently released its latest flagship model. If we use a space exploration metaphor: the previous generation was akin to an “unmanned lunar orbit,” validating basic routes and engineering feasibility. The new generation is closer to a “manned lunar orbit,” demonstrating stability comparable to human workflows when handling real-world problems and long-horizon tasks.

The core changes in this generation focus on three dimensions:

• Significant improvement in reasoning capabilities: It ranked first among domestic models in multiple logical evaluations, reclaiming the position previously held by competing products.
• Pricing strategy adjustment: The unit price increased slightly from 21 to 27. This modest hike reflects the team’s confidence in the capability improvements while maintaining a relatively restrained commercialization pace.
• Stable Chain-of-Thought (CoT) length: The overall CoT scale in reasoning mode has not expanded drastically, keeping total costs controllable.

However, the behavioral characteristics of the non-reasoning mode have fundamentally changed. It is no longer a simple “quick responder” but functions more like an “adaptive reasoning model with a lower thinking budget.” Simple problems can be resolved within a few thousand tokens, but complex tasks can cause output consumption to surge to 20K or even 30K tokens. Because the token limit for non-reasoning mode is set at 15K, nearly half of the test samples exceeded this limit and could not be scored according to the rules, resulting in incomplete evaluation data for this mode.

The reasoning mode is similarly affected by excessively long CoTs. A small number of problems consistently exceeded the 80K token limit across multiple test runs, automatically receiving a score of zero. This is the primary reason for the lower median score in reasoning mode.

2. Long-Chain Reasoning: Progress and Costs in Solution Space Exploration

2.1 Core Improvements Over the Previous Generation

In various long-chain logic and multi-step reasoning tasks, the new generation model significantly outperforms its predecessor. The previous model scored well on identical problems but had a typical flaw: insufficient exploration of the solution space in scenarios with multiple valid answers. It tended to stop after finding only a subset of feasible solutions.

The new generation model resolves this issue by exploring possibilities more thoroughly and striving to identify all potential solutions. This is highly beneficial for tasks requiring exhaustive search, enumeration, and multi-path validation, such as constraint satisfaction, combinatorial optimization, and logic puzzles.

2.2 The Cost: Maxing Out the Thinking Budget

This improvement does not come without trade-offs. The new model often runs at the maximum thinking budget limit on complex logic problems:

• While overall token consumption remains on par with the previous generation, consumption for complex logic problems can be more than three times higher than the previous model.
• Compared to industry-leading models, when solving the same problem with identical full scores, leading models consume only 1/2 to 1/6 of the tokens used by this new generation model.

This indicates that improving thinking efficiency is a core issue the next generation of models must address. Simply increasing the length of reasoning does not linearly improve accuracy; the law of diminishing marginal returns is particularly evident in such tasks.

2.3 Engineering Implications

For developers, this brings several practical impacts:

1. Tiered cost estimation: Average token consumption cannot be used as a simple proxy for cost estimation. Tail-end consumption for complex tasks may far exceed expectations.
2. Setting reasonable timeouts and limits: The 15K limit in non-reasoning mode is frequently triggered during complex tasks. It is recommended to implement dynamic budget allocation at the application layer.
3. Focusing on “effective thinking” rather than “CoT length”: In Agent orchestration, developers can use task decomposition, tool invocation, and external validation to reduce the model’s ineffective exploration within a single reasoning chain.

3. Coding Capabilities: From One-Shot to Real-World Agent Scenarios

3.1 Shortcomings of the Previous Generation

The previous model saw significant improvements in one-shot coding capabilities—meaning, given a clear function signature or problem description, it could generate usable code in a single pass. However, its performance in real-world Agent and real-project scenarios was suboptimal. Main issues included:

• Lack of effective debugging methods; difficulty in autonomously locating and fixing compile-time or runtime errors.
• Insufficient context management in multi-file and multi-module collaboration scenarios.
• Limited adherence to existing project code styles and dependency constraints.

3.2 Improvements in the New Generation

In an earlier public coding preview version, the Agent programming capabilities were significantly enhanced. Looking solely at standard front-end and back-end programming tasks, its capabilities have surpassed some mainstream commercial coding assistants, entering the “usable” threshold.

Specific improvements include:

• Enhanced multi-round debugging: It can iteratively fix issues based on error messages instead of giving up after a single generation.
• Project-level context understanding: Improved ability to locate relevant files and understand call relationships within larger codebases.
• More stable tool invocation: More reliable interactions with file systems, terminals, and testing frameworks.

3.3 Remaining Challenges

Despite evident progress, there is still a gap before achieving a “fully autonomous software engineering Agent”:

• Success rates drop noticeably in ultra-large projects, legacy codebases, and complex build systems.
• Understanding requirements still relies on clear descriptions; performance is unstable under ambiguous requirements.
• Self-verification mechanisms for long-horizon tasks require strengthening, as the model may produce outputs that appear correct but are functionally erroneous.

4. Selection and Implementation Recommendations

4.1 Scenario Matching

Scenario Type	Recommended Mode	Precautions
Simple Q&A, information retrieval	Non-reasoning mode	Monitor token limits to avoid triggering overflows in complex tasks
Multi-step logic, mathematical reasoning	Reasoning mode	Ensure sufficient budget; account for tail-end consumption
Code generation, debugging	Agent mode	Requires integration with tests and human review
Long document comprehension, summarization	Non-reasoning mode	Long-text input capability is an advantage

4.2 Cost Control Strategies

1. Dynamic budget allocation: Estimate token demand based on task complexity rather than applying a one-size-fits-all limit.
2. Result caching and reuse: Build a caching layer for repetitive tasks to reduce duplicate reasoning.
3. Hybrid orchestration: Route simple sub-tasks to the non-reasoning mode and complex sub-tasks to the reasoning mode, centrally managed via an Agent framework.
4. Monitoring and alerting: Establish monitoring for excessively long outputs and limit-exceeding failures to detect anomalous consumption promptly.

4.2 Understanding Capability Boundaries

• Strong reasoning capability ≠ Use reasoning mode for all tasks. Applying reasoning mode to simple tasks is a waste of resources.
• “Usable” coding capability ≠ Completely replacing human review. Critical code still requires manual oversight.
• High evaluation scores ≠ Good production performance. Robustness in real-world scenarios requires independent validation.

5. Conclusion

This generation of Chinese LLMs has made substantial progress in two key directions: long-chain reasoning and Agent-based coding. Solution space exploration is more thorough, debugging capabilities are stronger, and project-level understanding is deeper. However, the thinking efficiency issue is further amplified—token consumption on complex tasks can be multiples of that of industry-leading models.

For developers and enterprise users, the key is not chasing the latest model version, but rather:

1. Understanding the model’s capability boundaries and cost structure.
2. Selecting appropriate modes and budget strategies based on business scenarios.
3. Compensating for the model’s inherent shortcomings through engineering practices (task decomposition, tool invocation, caching, monitoring).

The core competitive focal point for the next generation of models will likely no longer be “can it get the right answer,” but rather “at what cost can it get the right answer.” This is simultaneously a technical issue, an engineering issue, and a cost issue.