Benchmarking Math Capabilities of Chinese LLMs: From Single Scores to Sub-Skill Diagnostics

The Problem with Relying on a Single Total Score

In discussions about LLM capabilities, “mathematical ability” is often seen as a relatively objective dimension. This is because math problems typically have definitive answers, making evaluation less subjective than open-ended writing assessments. However, in practical technical evaluation, looking at a single composite score rarely tells you whether a model is genuinely better at mathematical reasoning.

A more reasonable approach is to decompose mathematical ability into sub-skills: algebraic operations, geometric reasoning, probability and statistics, logical proofs, numerical computation, word problem modeling, multi-step reasoning stability, and the ability to convert mathematical expressions into code. By observing these dimensions separately, you can better understand both the magnitude of the gap and where it actually lies.

In practice, models may show similar total scores on a leaderboard, but their performance variance across math sub-skills is often much larger. One model might excel at competition-style algebra problems but frequently drop points in long-chain mathematical derivations; another might perform steadily on word problems but fall behind on math tasks that require code-based solutions. Therefore, evaluating math capability should emphasize sub-score performance, error types, and consistency within the same difficulty range.

Three Layers of Bias Between Leaderboards and Real Math Ability

Public leaderboards typically list win/loss comparisons across multiple benchmarks. You might see something like “won 27 out of 34 tasks, lost 7.” While intuitive, such comparisons do not directly translate into differences in mathematical ability.

1. Bias from Benchmark Composition

Different evaluation sets cover very different task types. Some focus on code generation, others on software engineering, agent planning, or multi-turn tool invocation. Math is only one component. If most of the 34 benchmarks are not purely mathematical, then “winning more” reflects overall technical capability rather than mathematical reasoning per se.

2. Differences in Difficulty and Discrimination

Even within math-specific benchmarks, the difficulty distribution affects conclusions. A model might lead on medium-difficulty problems but fall behind on hard proof questions, and the total score might still look similar. Only by examining difficulty-stratified results can you determine the real gap in “upper-bound mathematical reasoning.”

3. Weight Differences Between Public and Self-Built Benchmarks

Many leaderboards include both public benchmarks and self-built evaluations. Public benchmarks are usually community-validated, with relatively transparent question sources, scoring methods, and difficulty distributions. Self-built evaluations may be more aligned with specific business scenarios, but they are less comparable across models. If a model overtakes on a few self-built benchmarks but lags on more public math benchmarks, cautious interpretation is required before declaring a “comeback” or “number one.”

Dissecting Win/Loss Results: Substantive Gaps vs. Boundary Fluctuations

To better understand mathematical ability gaps, classify results into three categories.

Category 1: Gaps Within the Error Margin

For example, a score of 44.4 versus 42.3 on a benchmark — a difference of 2.1 points. This could stem from sampling variance, prompt sensitivity, or output parsing differences, and does not necessarily indicate a stable difference in reasoning capability. For such items, it is better to run multiple evaluations and average the results, change random seeds, or control decoding parameters before drawing conclusions.

Category 2: Clear Single-Task Dominance

If a math-related task shows 45.1 versus 37.3, a gap of 7.8 points, the difference usually indicates a structural divergence on that task type. Possible causes include different training data coverage, reasoning strategies, code integration capabilities, or long-chain stability. Error sample analysis is recommended for these items.

Category 3: Significant Reverse Gaps

If another task shows 28.1 versus 36.3, an 8.2-point gap, the model has a clear weakness in that sub-domain or task format. Common math weaknesses include complex symbolic reasoning, multi-step proofs, conversion of math expressions into code, and constraint satisfaction in word problems. Similarly, if a model merely edges ahead on terminal tasks but leads significantly on 3D modeling or backend tasks, that does not directly imply stronger math ability—only that its capability distribution is more skewed toward certain tasks.

Thus, “winning more tasks” should not be equated with “stronger at math.” The key is whether the model consistently wins on tasks highly correlated with mathematical reasoning, and whether its losses are concentrated in core math capabilities.

Calm Comparison with Top Closed-Source Models: Where Does the Math Advantage Actually Appear?

When comparing Chinese first-tier models with top closed-source models, a common outcome is: overtaking on a few tasks, still lagging on many others. When you see “won 6 out of 34, lost 28,” check which tasks were actually won.

If the wins concentrate on deep software engineering, enterprise-grade agents, web page construction, lifestyle search, or agent exams—tasks that are not purely mathematical benchmarks—then you cannot conclude that “math ability has caught up.” These tasks mostly test tool invocation, environment interaction, long-horizon planning, and integrated reasoning; math is only one component.

Conversely, if the model still clearly lags on standard math competition problems, theorem proving, numerical reasoning, and math word problems, the math gap remains real. Top closed-source models tend to be stronger in these areas:

• Consistency in multi-step reasoning: fewer skipped steps, symbol errors, or logic breaks in longer chains;
• Mixed understanding of mathematical and natural language: more accurate extraction of constraints from complex problem statements;
• Collaborative use of math and code: more stable generated code for numerical computation, enumeration verification, symbolic differentiation, and graph search;
• Error recovery: better at backtracking, switching methods, and retrying after a failed reasoning path;
• Difficulty generalization: maintaining reasonable accuracy on math problems outside the training distribution.

These capabilities rarely show up in a single total score, yet they significantly affect real-world usage.

A Practical Method for Evaluating Math Ability: From Leaderboard Scores to Sub-Skill Diagnostics

If a development team or enterprise user genuinely cares about “math capability gaps,” I recommend building a sub-skill diagnostic pipeline rather than just staring at a composite leaderboard.

1. Build a Math Sub-Skill Matrix

Split by problem type and capability, for example:

• Elementary algebra and equations;
• Functions, sequences, and inequalities;
• Geometry and trigonometry;
• Combinatorics, probability, and statistics;
• Calculus and linear algebra;
• Mathematical modeling and word problems;
• Proofs and logical reasoning;
• Converting math problems into code.

Evaluate each sub-skill independently and observe accuracy and stability per item.

2. Distinguish “Can Do” from “Consistently Can Do”

The same question may produce different answers across samples. Use multiple sampling evaluation—for example, sample each question five times and compute the consensus accuracy, the pass@5 accuracy, and the average accuracy. If pass@5 is high but consensus accuracy is low, the model has some capability, but reasoning stability is insufficient. This difference is especially critical in math.

3. Document Error Types

Math errors should not only be counted as right or wrong. Break them down further:

• Misunderstanding of the problem statement;
• Incorrect formula reference;
• Symbolic derivation errors;
• Calculation mistakes;
• Missing units or constraints;
• Incomplete proof logic;
• Insufficient numerical precision;
• Errors in code implementation.

The distribution of error types is more instructive for subsequent optimization—such as prompt design, retrieval augmentation, code tool integration, or model routing—than the overall accuracy rate.

4. Validate Math Results Through Code Execution

For math tasks involving numerical computation, equation solving, function plotting, or probability simulation, let the model generate both reasoning steps and executable code, then validate results by actually running the code. For instance, the model first derives a formula, then generates solving code, and finally uses the execution output to verify the answer. This “reasoning–execution–verification” loop significantly improves the reliability of math results.

The flow is as follows:

Input math problem
   ↓
Model generates reasoning steps
   ↓
Generate executable solving code
   ↓
Run code to obtain numerical result
   ↓
Compare with the reasoning-based answer
   ↓
If inconsistent, trigger retry or revision

This mechanism is not platform-dependent. It is a general engineering practice that can run in containerized environments and be combined with common cloud infrastructure components—message queues, logging services, monitoring and alerting—to form a reliable evaluation pipeline.

What Conditions Would a Genuine Math “Comeback” Require?

From a technical standpoint, for a model to truly overtake in math capability, it would need verifiable progress in at least the following dimensions:

1. Longer and More Stable Reasoning Chains

Many high-difficulty math problems require dozens of reasoning steps. Chinese models already perform well in short- and medium-chain reasoning, but may still lose intermediate steps, forget conditions, or confuse symbols in long-chain tasks. Maintaining stability in long chains is the key differentiator.

2. Stronger Symbolic and Formal Reasoning

Mathematical language relies heavily on symbols, definitions, theorems, and inference rules. Formal math tasks—converting propositions into logical expressions, checking the validity of proof steps, deriving identities—place extremely high demands on models. If a model consistently improves in formal transformation and symbolic reasoning, its math ranking can move forward.

3. Deeper Integration of Math and External Tools

Real-world problems often require the model to invoke calculators, symbolic computation, numerical solvers, plotting tools, or data processing utilities. Top closed-source models are generally more mature at “when to call a tool, how to organize multi-step tool calls, and how to extract key information from tool outputs.” This is an easily overlooked but highly impactful part of math capability evaluation.

4. More Proactive Error Checking and Self-Correction

Math problems demand high correctness; even a small symbol error can lead to a wrong final answer. If a model can proactively verify intermediate results—checking whether both sides of an equation are equal, confirming whether the result satisfies the original constraints, and re-solving after detecting unreasonable numerical values—accuracy improves significantly. This “self-verification” capability is the key to moving from usable to reliable.

5. Broader and Deeper Training Data Coverage

Another foundation of math ability is high-quality mathematical corpora—textbooks, competition problems, proof libraries, math papers, solution sets, error cases, and so on. The breadth of data coverage determines how many question types the model can handle; its depth determines how difficult problems it can solve. Without sufficient high-quality math data, simply increasing model size or training duration has limited effect.

Trend Insights: From “Impossible to Close” to “Catchable”

Previously, the gap between Chinese first-tier models and top closed-source models in math tasks often followed a pattern: small gaps on easy problems, large gaps on hard ones; small gaps in single-step reasoning, large gaps in multi-step proofs; small gaps in pure-text math, large gaps in math combined with code. That pattern is now shifting.

Shift one: the gap on low- and medium-difficulty math tasks has narrowed significantly. Many Chinese models have reached usable levels on common exam questions, word problems, and basic competition tasks.

Shift two: on some math tasks that require tool invocation, Chinese models are beginning to show overtaking capability—especially in scenarios closely tied to code, data, and environmental interaction.

Shift three: on ultra-high-difficulty math problems, top closed-source models still hold an advantage, but it is no longer as absolute as in the early stages.

The essence of this situation is not that some model suddenly became “number one across the board.” Rather, the capability gap has evolved from single-point lag to “mutual wins and losses.” In the math dimension specifically, it is premature to declare an overtake—but it is fair to say that Chinese first-tier models have entered a catch-up zone and hold local advantages on certain math-related tasks.

A More Scientific Conclusion: Use “Capability Profiling” Instead of a Single Ranking

In math capability evaluation, the more engineering-relevant output is not a rank but a capability profile for each model. For example:

• Model A: Strong in algebra and word problems; moderate in proofs; general numerical stability; suitable for everyday math Q&A.
• Model B: Strong in competition-style algebra; weaker in geometry; stable in code-assisted math solving; suitable for technical computing scenarios.
• Model C: Balanced overall math capability; good in long-chain proofs; but math solving under tool invocation still fluctuates.
• Model D: Strong in high-difficulty math reasoning and modeling; but slower and more expensive to call.

Such profiling helps enterprise users select or route models based on actual tasks, rather than chasing a “Math #1” label.

In engineering systems, math tasks can also be split into difficulty tiers: use more cost-effective models for low-difficulty tasks, route high-difficulty tasks to stronger models, integrate a code execution environment for code-dependent math problems, and add multi-model cross-validation for critical results. These practices are more reliable than staring at a single leaderboard.

Summary: The Math Gap Persists, but Has Shifted from “Overall Lag” to “Local Competitiveness”

Overall, the mathematical capability gap between Chinese LLMs and top closed-source models cannot be captured by a single rank or total score. A more accurate assessment is this:

• On common math Q&A, basic competition tasks, and math word problems, Chinese first-tier models have significantly narrowed the gap, reaching usable or even good levels in some scenarios.
• On high-difficulty proofs, symbolic reasoning, long-chain multi-step reasoning, and deep math-code integration, top closed-source models still hold an advantage.
• On certain math-related tasks requiring tool invocation, environment interaction, or code-execution verification, Chinese models have shown the possibility of partial overtaking.

Thus, a “comeback to number one” is better understood as: within a complex evaluation system, Chinese first-tier models have entered a stage of mutual wins and losses with top closed-source models. Math capability is no longer a dimension that is unilaterally suppressed; it is a direction that can be continuously chased and locally broken through. For developers and enterprise users, the more pragmatic strategy is to build your own math evaluation set, focus on sub-skill accuracy, reasoning stability, and error types—rather than counting how many more tasks one model won on a leaderboard.