Benchmarking Chinese LLM Capabilities at Scale: Where Open Models Are Closing the Gap and Closed Models Still Lead

Evaluation Background and Methodology

Quantitative evaluation of large language model capabilities remains a core concern for both academia and industry. As model parameter scales continue to expand and training paradigms evolve, relying solely on manual evaluation or a single metric can no longer fully reflect a model’s actual performance. A general-purpose LLM benchmark released in November 2026 brought together 19 mainstream models from China and abroad, building a systematic evaluation framework around five core task dimensions: agentic coding, mathematical reasoning, ultra-long context processing, precise instruction following, and hallucination control.

The design logic behind this evaluation system is straightforward: these five dimensions cover the capability scenarios most frequently encountered in enterprise applications and day-to-day developer workflows. From code generation to complex logical reasoning, from long-document understanding to output reliability, each dimension directly affects how usable a model is in production environments.

Notably, the benchmark introduced a rule that models with a score difference of less than 1 point are considered tied. This is not a statistical compromise but a professional judgment about performance fluctuation ranges—on the same evaluation set, a score drift of ±1 point across repeated runs of the same model is considered normal. This rule makes the final ranking more reflective of actual capability tiers and avoids misleading ordering caused by marginal score differences.

Closed-Source Models: Sustained Leadership in Overall Capability

At the top of the leaderboard, two closed-source models demonstrated clear advantages in comprehensive capability. The top-ranked model achieved a total score of 73.62, with particularly strong sub-task performance: 95.63 in ultra-long context, 88.97 in instruction following, and 83.19 in hallucination control. The balanced distribution across these three core metrics indicates strong stability when handling complex tasks requiring high-precision long-text understanding and multi-turn instruction following. The near-perfect ultra-long context score suggests the model is approaching human-expert level in large-scale document analysis, cross-chapter information integration, and long-conversation memory retention.

The second-place closed-source model scored 71.76 overall, with standout mathematical reasoning performance at 76.32. This indicates specialized strength in mathematical problem solving, logical deduction, and symbolic reasoning—capabilities critical for financial risk modeling, scientific research assistance, and complex algorithm design. The variation in strengths also reflects how different closed-source models adopt differentiated strategies in training data composition and architecture design.

The overall lead of closed-source models is not accidental. From a technical implementation perspective, closed-source models typically benefit from greater compute investment, more refined data governance pipelines, and more systematic alignment training. During reinforcement learning, closed-source models can undergo more extensive large-scale iterative optimization through human feedback signals. This closed loop of data, compute, and algorithms often puts closed-source models one to two generations ahead of open-source competitors in comprehensive capability. However, the lead is narrowing—a trend visible in the dense distribution of open-source models within the top ten of the overall leaderboard.

Open-Source Models Surge: China’s First-Tier Models in Focus

The most notable finding of this evaluation was the collective rise of open-source models. A Chinese open-source model topped the open-source category with a total score of 61.46, making it the highest-ranked Chinese model overall. Its hallucination control score of 67.78 stands out not only among Chinese models but also places it within a narrow gap of leading closed-source models. Hallucination control directly determines the trustworthiness of model output. For industries with high compliance requirements such as finance, healthcare, and law, this metric can matter more than raw reasoning ability.

Another Chinese open-source model followed closely with a total score of 58.95, with the two models forming the first tier of Chinese open-source offerings. The gap to the third Chinese model, which scored 50.66, is more pronounced, indicating that performance stratification has emerged within the domestic open-source landscape. From a technical standpoint, these open-source models generally adopt Mixture-of-Experts (MoE) architectures, using sparse activation mechanisms to maintain high parameter capacity while controlling inference costs. In terms of training strategy, open-source teams often place greater emphasis on data diversity and targeted reinforcement of long-text capabilities, which helps explain why open-source models can achieve scores approaching closed-source models in ultra-long context and hallucination control dimensions.

The progress of open-source models has profound implications for the industry. Developers can choose sufficiently powerful open-source base models for local deployment and data-private scenarios, avoiding compliance risks associated with sensitive data leaving controlled environments. At the same time, a thriving open-source ecosystem accelerates the evolution of inference optimization toolchains, fine-tuning frameworks, and evaluation benchmarks, creating a positive feedback loop.

Capability Analysis by Dimension: Distinct Technical Strengths

Shifting the perspective from total scores to individual dimensions reveals a clearer picture of each model’s applicable scenarios and technical characteristics.

In the agentic coding dimension, leading models performed relatively closely, all capable of completing complex multi-step code generation, cross-file refactoring, and API call sequence generation tasks. This dimension evaluates not only syntactic correctness but also the depth of programming intent understanding and the reasonableness of tool-call planning. As AI coding assistants evolve from code completion toward autonomous agents, this capability has become a key benchmark for assessing a model’s potential to boost developer productivity.

Mathematical reasoning showed clear stratification. Leading closed-source models scored significantly higher, while smaller models struggled in this dimension. Mathematical reasoning demands strong symbolic manipulation and intermediate-step memory, requiring substantial chain-of-thought training and solid mathematical corpus support. For organizations applying LLMs in scientific computing, quantitative analysis, and similar domains, this dimension carries direct reference value.

Ultra-long context is one of the metrics that best reflects architectural sophistication. The top-scoring model achieved 95.63, meaning it can accurately retrieve and correlate scattered key information across contexts spanning tens of thousands of words or more. Current mainstream approaches include long-context extrapolation techniques, sparse attention mechanisms, and hierarchical retrieval-augmented generation architectures. The practical value of this capability is evident in legal contract review, academic literature synthesis, and enterprise knowledge base Q&A—effective use of context length directly determines whether a model can handle book-length tasks.

Hallucination control tests a model’s self-restraint within its knowledge boundaries. Higher-scoring models tend to honestly admit “I don’t know” when facing questions beyond their training data coverage rather than fabricating plausible-sounding answers. Achieving this requires factual data filtering during training, targeted penalties for hallucinated samples during RLHF, and confidence calibration mechanisms during inference. For customer service systems, intelligent medical consultation, and government-facing inquiry platforms, hallucination control directly affects user trust.

Intense Competition in the Mid-Tier and What Tied Rankings Reveal

In the middle of the leaderboard, multiple models tied due to score differences of less than 1 point. On the surface, this is a product of the statistical rule. At a deeper level, it reflects that LLM development has entered a plateau phase—as architectures converge, training data sources become similar, and alignment techniques mature, capability gaps between models are being rapidly flattened. This homogenization trend is a double-edged sword for the industry: it reduces decision-making costs for enterprises selecting models, but it also means that simply stacking model parameters can no longer create a durable technical moat.

Mid-tier models also showed distinctive strengths in specific tasks. Some performed excellently in instruction following, precisely understanding user instructions with complex constraints—a capability closely tied to the diversity of instruction data used during supervised fine-tuning. Others demonstrated notable strength in agentic coding despite modest total scores, indicating that specialized capabilities can already hold practical value. This serves as a reminder that developers should not blindly chase leaderboard rankings when selecting models. Instead, they should align their choices with core business scenarios and pay attention to dimension-specific scores.

Practical Perspectives: Scenario-Based Model Selection

Based on the evaluation results, model selection strategies for different application scenarios can be summarized as follows:

For teams building long-text analysis and knowledge-intensive applications, prioritize models with high ultra-long context scores and strong hallucination control. In these scenarios, models must extract key conclusions from massive document collections while ensuring verifiable output—both low hallucination rates and high long-text recall are essential.

For AI-assisted programming, automated code review, and low-code development platforms, focus on agentic coding and instruction-following capabilities. A model that accurately understands team coding standards and follows specific style guidelines delivers far greater productivity value than one with strong reasoning alone.

For scenarios centered on mathematical computation and logical reasoning—such as financial risk assessment model development and scientific experiment design—mathematical reasoning scores are the most critical reference. It is also worth checking whether the model supports an explicit “thinking” mode toggle; many models show significantly improved reasoning accuracy when deep thinking mode is enabled.

For organizations with data compliance requirements, open-source models are the safer choice. Chinese open-source models have narrowed the gap with leading international closed-source models in hallucination control and long-text capabilities, and through private deployment combined with domain-specific fine-tuning, they can surpass general-purpose closed-source models on specific tasks.

Future Trends and Areas to Watch

This evaluation reveals several important signals about the distribution of LLM capabilities. First, the technology gap between closed-source and open-source models is shifting from an across-the-board divide to localized gaps, with open-source models now capable of competing head-to-head in specific capability dimensions. Second, ultra-long context and hallucination control have become the watershed metrics distinguishing model tiers, and future evaluation systems may further refine the sub-tasks within these dimensions. Third, the weight of agentic coding will continue to rise—as AI agents move from proof-of-concept to large-scale production deployment, model performance in multi-step task planning, tool invocation, and error recovery will directly determine commercial viability.

For technical decision-makers, LLM capability assessment should not be a one-time exercise. Model versions iterate rapidly. Establishing a capability tracking mechanism that combines quarterly benchmark retesting with internal evaluation sets and external public leaderboards is necessary to ensure technology choices remain aligned with evolving business needs. As model capabilities trend toward homogenization, engineering capabilities at the application layer—including prompt optimization, retrieval-augmented generation pipeline construction, and fine-tuning data pipeline development—will gradually replace model selection itself as the core variable determining the final effectiveness of AI applications.