The New Landscape of Chinese LLMs: Open-Source Tier Rise and Capability Differentiation

1. Evaluation Background: From “Availability” to “Quality”

Over the past year, the discourse surrounding Chinese Large Language Models (LLMs) has rapidly shifted from “can it be used?” to “which is better, and for what purpose?” As foundational capabilities mature, evaluation frameworks have become more granular. Instead of relying on single-dimension benchmark scores, assessments now deconstruct capabilities such as long-context understanding, instruction following, hallucination control, and mathematical reasoning. This approach aims to reflect how models actually perform in real-world business scenarios.

The latest round of comprehensive evaluations reveals several noteworthy trends. First, the gap between open-source and top-tier proprietary models continues to narrow. Domestic open-source models have formed a clear “first tier” for the first time in aggregate scores. Second, different models exhibit distinct capability differentiation—there is no all-around champion, only scenario-specific fitness. This means developers can no longer rely solely on overall rankings; they must match models to their specific business requirements.

2. Aggregate Scores: Open-Source Tier Solidifies

Looking at the aggregate scores, the most prominent change in this evaluation round is the overall leap of open-source models. The top-ranking open-source model scored 61.46, achieving the highest score not only within the open-source community but also among all domestic models. It was followed by a lightweight open-source model scoring 58.95, with another domestic open-source model taking third place at 50.66. These three models constitute the first tier of domestic open-source LLMs, with aggregate scores concentrated between 50 and 62.

Notably, there is a significant score gap between this first tier and the subsequent groups. This indicates a “top-tier convergence” within the open-source community: teams with heavy resource investments, rapid iteration cycles, and robust evaluation feedback loops are pulling ahead. For enterprise users, this narrows the candidate pool for open-source selection, effectively lowering decision costs. Evaluating just the top three or four models is generally sufficient to cover most general-purpose scenarios.

However, aggregate scores are only the entry point. The actual deployment effectiveness is determined by specific sub-capabilities.

3. Capability Breakdown: No All-Rounder, Only Fitness for Purpose

3.1 Long Context: The Ceiling of High-Scoring Models

In ultra-long context tasks, the best-performing model achieved a score of 95.63, approaching perfection. What does this mean? Simply put, even when the input text reaches hundreds of thousands of tokens, the model maintains stable retrieval and reasoning capabilities for key information, avoiding “middle-of-context forgetting” or imbalanced processing.

Long context capabilities directly impact the deployment effectiveness of the following scenarios:

• Repository-level code understanding: Reading multiple files at once to perform cross-file dependency analysis and refactoring suggestions.
• Legal and compliance document review: Locating clause conflicts and risk points in lengthy contracts.
• Multi-turn dialogue memory: Maintaining consistency with user preferences and historical constraints throughout long conversations.

It is important to note that a high long-context score does not equate to excellent performance in all long-text tasks. In practical use, developers must also monitor retrieval accuracy, positional bias, and inference latency. High-scoring models are better suited for scenarios requiring comprehensive reading rather than sheer throughput.

3.2 Instruction Following: The Key to “Obedience”

The model scoring 88.97 in instruction following demonstrated balanced performance in this evaluation. This metric measures how well a model adheres to format constraints, role settings, and multi-step instructions. For developers, instruction following directly impacts API stability:

# Example: How instruction following affects structured output reliability
prompt = """
Please return the following information in JSON format:
- Field name: summary, type: string, max length 50 characters
- Field name: tags, type: array, max 3 elements
Return JSON only, no extra explanations.
"""
# Models with strong instruction following: Output can be parsed directly by downstream programs
# Models with weak instruction following: May include explanatory text, causing parsing failures

The impact of instruction following discrepancies is magnified in automation workflows, Agent orchestration, and data extraction scenarios. A model scoring close to 90 can generally support production-grade structured output requirements reliably. Lower scores often necessitate additional post-processing and retry mechanisms.

3.3 Hallucination Control: A Domestic Watershed

Hallucination control is the dimension that best reflects the current characteristics of domestic models. The top-ranking open-source model scored 67.78 in this category, holding a distinct advantage domestically. However, the best overall model scored 83.19, maintaining a visible gap.

Hallucination control scores directly affect the credibility of the following scenarios:

• Knowledge QA: Whether the model fabricates non-existent facts or citations.
• Code generation: Whether it calls non-existent library functions or parameters.
• Data analysis: Whether it produces seemingly plausible but incorrect conclusions in numerical calculations.

For enterprise applications, hallucination control is often more critical than “sounding good.” A model scoring in the 67–70 range can meet most requirements in controlled scenarios (e.g., when combined with Retrieval-Augmented Generation [RAG] and explicit context). However, for open-ended knowledge QA, it still requires citation verification and human review.

3.4 Mathematical Reasoning: The Value of Specialized Capabilities

In the mathematical reasoning dimension, the best-performing model scored 76.32, demonstrating advantages in mathematical deduction, logical operations, and complex problem decomposition. These capabilities are indispensable in:

• Engineering calculations and simulations: Requiring multi-step deductions where intermediate results must be error-free.
• Financial modeling: Involving calculations for interest rates, compound interest, and risk exposure.
• Algorithmic problem solving: Requiring comprehension of the problem and generation of executable code.

Note that a high mathematical reasoning score does not guarantee equally出色的 general conversational abilities. Such models are better utilized as “reasoning engines” embedded into specific workflows rather than serving as general-purpose assistants.

4. Selection Guide: From Scores to Deployment

Combining aggregate scores and sub-capability performance, we can outline a rough selection framework:

Scenario Requirement	Priority Dimension	Recommended Score Range
Long document processing	Long context	90+
Automated workflows	Instruction following	85+
Knowledge QA	Hallucination control	65+ (with RAG)
Math/Engineering calculations	Mathematical reasoning	70+
General conversation	Balanced aggregate score	55+

Furthermore, the first tier of open-source models now possesses the capability to compete directly with proprietary models. They offer inherent advantages in private deployment, data isolation, and cost control. For teams with compliance requirements or those needing deep customization, prioritizing top-tier open-source models is a more pragmatic choice.

5. Conclusion

The most significant takeaway from this evaluation round is not a specific score, but the trend of “capability differentiation.” Chinese LLMs have moved past the stage of homogeneous competition, entering an era of fine-grained matching based on scenarios and capability dimensions. The core task for developers has shifted from “choosing the strongest” to “choosing the most appropriate.” Understanding the technical implications behind sub-metrics and conducting weighted evaluations based on specific business needs is the most effective selection strategy at this stage.