How Large Is the Math Capability Gap Between Chinese LLMs and Top Closed-Source Models?

One Aggregate Score Doesn’t Tell the Whole Story

When comparing the math capability of Chinese large language models with top closed-source models, it is easy to fall into a simplistic “who ranks first” framing. A more productive approach is to decompose math capability into observable dimensions and examine how models differ across each one.

Math capability can be broken down into at least the following:

• Basic arithmetic and symbolic computation
• Algebraic manipulation and equation solving
• Geometric reasoning and spatial visualization
• Probability, statistics, and combinatorial counting
• Proof writing and logical deduction
• Mathematical modeling and problem reformulation
• Code-assisted solving and numerical computation
• Consistency maintenance across multi-step reasoning

The same model may perform well on the first few items while degrading noticeably on long proofs, complex modeling, or problems that require tool-based solving. Relying on a single composite score can easily mask these real capability boundaries.

Recent public leaderboards show Chinese frontier models and top closed-source models trading wins across many tasks. Several Chinese models stand out in code generation, engineering problem solving, and agentic tasks, while still trailing in some areas of deep reasoning, long-horizon planning, and general foundation capability.

Where First-Tier Chinese Models Stand

Consider a 34-benchmark evaluation suite. When a new-generation Chinese model was compared with an open-source first-tier model, it won 27 benchmarks and lost 7—evidence that it is competitive on most evaluation items. But a closer look at the 7 losses shows the gaps are uneven.

The narrowest losses:

• Software engineering composite tasks: 74.9 vs. 77.0 (−2.1)
• Enterprise-level complex tasks: 78.8 vs. 79.5 (−0.7)
• Automation tasks: 56.2 vs. 60.5 (−4.3)
• Base model capability evaluation: 28.1 vs. 36.3 (−8.2)

Among the wins, several stand out:

• Backend engineering tasks: 45.1 vs. 37.3 (+7.8)
• 3D modeling tasks: 37.1 vs. 32.3 (+4.8)
• Terminal operation tasks: 44.4 vs. 42.3 (+2.1)

This pattern of trading wins indicates that the Chinese first tier is no longer playing catch-up; these models are genuinely usable across many engineering and agent scenarios.

The picture is more sobering against top closed-source models. On the same 34 benchmarks, one Chinese model won 6 and lost 28. Its wins were concentrated in:

• Software engineering composite tasks: 74.9 vs. 74.2
• Enterprise-level complex tasks: 78.8 vs. 72.3
• Web building tasks: 78.9 vs. 78.8
• Daily-life search tasks: 60.2 vs. 46.2
• Agent tasks: 75.2 vs. 73.5
• Agent first-round exam: 79.9 vs. 77.9

Some of these wins are razor-thin—web building differs by just 0.1 points, effectively a tie—while others are wide, such as the 14-point lead in daily-life search. Overall, Chinese models can overtake top closed-source models in a narrow set of engineering-heavy and agent-heavy scenarios, but they remain behind in broader general capabilities.

Why Math Gaps Get Amplified on Leaderboards

Math and code benchmarks share a common trait: problems usually have definitive answers, scoring is objective, and results become “hard numbers.” But a model’s position on math leaderboards is influenced by several additional factors.

Benchmark difficulty distribution

Benchmarks differ in difficulty and question-type mix. Some lean toward secondary-school competition problems, some toward advanced mathematics, and some toward proofs. A model may perform well at medium difficulty yet decline quickly on hard proof problems. A score gap between two models may therefore reflect not only capability, but also how well the question distribution suits each model.

Stability of multi-step reasoning

Mathematical reasoning typically requires many steps. A model may achieve high accuracy on individual steps, but errors accumulate as the reasoning chain grows.

For example, take a proof that requires 10 reasoning steps. At 95% per-step accuracy, overall accuracy is roughly 0.95 raised to the 10th power, or about 60%. At 98% per-step accuracy, overall accuracy reaches about 82%. The per-step difference looks small, but the gap in final scores is significantly amplified.

Code-assisted solving

Modern LLMs increasingly solve math problems by generating code—using numerical computation, symbolic computation, or simulation. This binds math capability to code generation and tool-calling ability.

A model with slightly weaker mathematical intuition but strong code generation and stable tool calling can still score well on math benchmarks. Conversely, a model with solid math fundamentals but weak coding may fail on problems that require numerical solutions.

Benchmark leakage and data contamination

Questions from public benchmarks may appear in training corpora. If a model has seen similar problems, its score will be noticeably inflated. This phenomenon exists among closed-source models as well, though open-source training data is easier for the community to analyze.

Math score gaps on public leaderboards therefore do not equate to gaps in true reasoning capability.

How to Evaluate Math Capability More Accurately

For developers, a more reliable approach is to build a small evaluation set that closely mirrors your own business scenarios. Here is a workable framework.

Design questions in layers

Decompose math capability into tiers:

• Tier 1: basic computation and formula rewriting
• Tier 2: word-problem modeling
• Tier 3: multi-step reasoning and proofs
• Tier 4: comprehensive modeling with code-assisted solving

Assign several problems to each tier, covering a range of difficulty levels.

Score the process, not just the answer

Beyond checking final answers, verify whether the reasoning process is sound. Manually sample reasoning chains to identify cases where the model reaches the right answer through a flawed process.

Run multiple times to measure stability

LLM outputs are stochastic. Run the same problem multiple times and observe the distribution of correct answers. A model may occasionally solve a hard problem, but if overall stability is poor, its usability in production remains limited.

Use a real code execution environment

If a problem can be solved with code, let the model run code in a real execution environment rather than merely generating code snippets. This tests mathematical modeling, code generation, and runtime debugging simultaneously.

Engineering Considerations Beyond the Gap

In real-world workloads, math capability rarely appears in isolation; it is usually embedded in larger tasks:

• In data analysis, a model must first understand the data structure, then choose a statistical method, and finally interpret the results.
• In engineering computation, it must first abstract the problem, then build a model, and then generate runnable code.
• In agent scenarios, it must first plan steps, then call tools, and then validate intermediate results.

The math capability gap should therefore be assessed together with engineering readiness. A model that trails on pure math leaderboards may still be the better fit in practice if it integrates more smoothly, calls tools more reliably, and costs less.

Conversely, a model that leads on math leaderboards but is complicated to deploy, suffers high inference latency, or carries unpredictable costs is not necessarily suitable for every scenario.

The Gap Is Narrowing, but the Paths Differ

In this round of evaluation, the gap between Chinese frontier models and top closed-source models is clearly smaller than in the past. In engineering-heavy and agent-heavy scenarios in particular, Chinese models have gained the ability to pull ahead. This suggests that Chinese model iteration is shifting from chasing general leaderboards toward solving real problems.

However, in broader base capabilities, long-chain reasoning, and general foundation performance, top closed-source models retain an overall advantage. That advantage is not necessarily stronger mathematical reasoning itself; it may stem from:

• More balanced general foundation capabilities
• More stable long-context understanding
• More reliable tool calling and planning
• Higher training data quality and diversity

The math capability gap is simply a projection of these composite differences.

Conclusion

The math capability gap between Chinese LLMs and top closed-source models cannot be summarized by a single total score. The 34-benchmark view shows Chinese models trading wins with the open-source first tier, while against top closed-source models they remain at a stage of local leads and overall deficits.

For developers, the more meaningful step is not fixating on a leaderboard rank, but building a layered evaluation set of your own—focusing on multi-step reasoning stability, code-assisted solving capability, and usability in real business scenarios. Leaderboard gaps can serve as a reference, but they should not be the sole basis for model selection.