Chinese Domestic LLMs: A Practical Assessment of Capability Boundaries After Deep Practice
This article presents a rational assessment of Chinese domestic large language models based on deep hands-on practice. It examines three representative scenarios—programming, mathematical reasoning, and creative writing—to show where current models perform well and where they fall short. It then analyzes the root causes of the gap with leading models, focusing on scale, emergent capabilities, and the limits of post-training and engineering wrappers such as agent frameworks. Finally, it proposes a reusable evaluation checklist covering scenario-specific testing, non-retrievable tasks, guidance cost, boundary-aware usage, and the distinction between executors and collaborators. The goal is to help developers place models on the right tasks rather than force complex workflows onto models whose capabilities do not yet meet the requirement.
1. Background: From Heavy Use to Rational Cooling
Over the past period, many developers have gone through a typical enthusiasm curve with large models: early on, they try various models frequently and invest substantial effort in building complex workflows; as they clarify the boundaries of model capabilities, usage frequency drops noticeably. This does not mean the models are worthless. Rather, users gradually learn to distinguish two types of problems:
- Problems models are good at: tasks with clear goals, clear paths, and objective validation criteria.
- Problems models are not good at: open-ended problems that require independently discovering latent requirements and evolving or iterating toward a “good result.”
Understanding this boundary is a prerequisite for evaluating any large model. From the perspective of a deep practitioner, this article systematically reviews the current capability status of Chinese domestic large models, the root causes of the gap with top models, and a reusable evaluation methodology.
2. Observations in Three Typical Scenarios
2.1 Programming: The Closest to “Usable”
When task requirements are clear, current Chinese domestic models perform quite solidly in programming. Take a data-processing script as an example: as long as the requirement description is clear enough, the model often produces a runnable result in one pass:
# Clear requirement example: count endpoint call frequency in logs and output Top 10from collections import Counterdef top_endpoints(log_path, top_n=10):counter = Counter()with open(log_path) as f:for line in f:parts = line.split()if len(parts) >= 7:counter[parts[6]] += 1return counter.most_common(top_n)
The common features of such tasks are: clear input/output definitions, objective validation, and easy failure detection. Models perform well on these “closed problems,” clearly better than the previous generation of Chinese domestic models, and generally do not require repeated rework.
But once the requirement is vague—for example, “help me optimize this system”—the model exposes its weakness: it does not proactively ask follow-up questions, nor does it independently discover the metrics you actually care about (latency? cost? maintainability?). Instead, it mechanically follows a literal interpretation. This is essentially strong instruction-following ability but weak intent understanding and proactive planning ability.
2.2 Mathematics and Rigorous Reasoning: The Passing Line Has Not Yet Been Crossed
Mathematical reasoning is a touchstone for testing a model’s “real intelligence.” A representative test method is to find a proposition in public mathematical materials for which no proof is provided, require the model to independently complete a rigorous proof, and output it as a complete set of lecture notes in proper formatting. Such tasks cannot be completed through retrieval or imitation; they must rely on the model’s own reasoning ability.
Practice feedback is that domestic models still have a low success rate on such tasks and often require manual step-by-step guidance and paragraph-by-paragraph correction before converging to a usable result. Based on users’ cross-model observations, only a very small number of top international models can currently handle open-ended mathematical problems, and domestic models show a clear generational gap with them.
The “generational gap” here is not an engineering-level gap. Neither post-training, agent frameworks, nor prompt engineering can fundamentally close it—the root cause lies in the relationship between model scale and intelligence emergence.
2.3 Creative Writing: A Typical Weakness in Mechanical Execution
For authors who have written web fiction for years and are sensitive to plot structure, current models’ performance on story scripts is “clearly below standard”: plot progression is formulaic, there is no real dramatic tension, and the model cannot spontaneously distill a good narrative structure. This again confirms a rule: the more a task depends on tacit knowledge, taste, and creativity, the more obvious the gap between models and human experts.
3. The Nature of the Gap: Scale, Emergence, and the Limits of Post-Training
Why can’t post-training, architecture optimization, and agent wrappers solve this problem?
- Emergent capabilities depend on scale. Many complex capabilities—multi-step reasoning, implicit intent understanding, and autonomous planning—are not “taught” through fine-tuning; they emerge after model scale crosses a certain threshold. Current Chinese domestic models may not yet have reached this emergence passing line.
- Engineering wrappers can only amplify existing capabilities. Agent frameworks, tool calling, and chain-of-thought optimization are essentially orchestration on top of the model’s existing capabilities. If the model foundation is weak, the orchestrated system remains a “mechanical executor” and cannot evolve autonomously.
- The marginal cost of manual guidance is extremely high. The real experience of practitioners is that producing usable results requires building a large pipeline, fine-tuning each stage, and repeatedly retraining across multiple versions, consuming substantial time and tokens. This “hand-holding” mode precisely shows that the model itself is not yet sufficient to independently solve real problems.
A rough judgment is that only when the scale of parameters and compute increases by another order of magnitude is it likely to truly cross the threshold of intelligence emergence.
4. An Actionable Methodology for Model Evaluation
Based on the above practice, a checklist for rationally evaluating models can be distilled:
- Test by scenario: design independent test cases for programming, mathematical proof, long-form writing, and multi-turn intent understanding. Do not draw conclusions from a single scenario.
- Use “non-retrievable” tasks: prioritize problems for which no ready-made answer exists in the materials (such as propositions without proofs or open-ended solution design) to rule out the possibility of the model “copying homework.”
- Measure guidance cost: record the number of prompt rounds, correction counts, and token consumption required to reach a usable result. This is a hard metric for measuring “autonomy.”
- Use models rationally after identifying boundaries: once you confirm that a model’s boundary does not exceed your requirement threshold, there is no need to force a complex workflow—keep it simple and wait for the next capability leap.
- Distinguish “executors” from “collaborators”: most current models are qualified executors, but they are still some distance from being collaborators that can proactively discover requirements or even guide humans in reverse.
5. Conclusion
Chinese domestic large models already do well enough on closed tasks, but in dimensions of openness, creativity, and autonomous emergence, they still show a generational gap with top models, and this gap cannot be closed by engineering means alone. For developers, rather than piling complex workflows onto models with insufficient capabilities, it is better to establish a clear evaluation system: define each model’s boundaries, use models on closed problems they are truly good at, and keep judgment, taste, and direction control for themselves. When scale crosses the emergence threshold, it will not be too late to reignite enthusiasm.
