The Math Capability Gap in Chinese LLMs: What Benchmark Differences Reveal About Model Selection

1. What the Win-Loss Record Reveals About Capability Structure

In a recently published LLM capability evaluation, seven models — including several Chinese models and top closed-source models — were compared under the same evaluation system across 34 tasks. One Chinese model came out ahead on 27 of them but lost on seven. Those seven losses were not concentrated in a single dimension; they spanned software engineering, complex task automation, end-to-end enterprise scenario handling, frontend page generation, and overall base-model capability.

This result shows that labels such as “No. 1” or “the best Chinese model” do not fully describe what a model can do. What deserves more attention is that each of the seven lost tasks points to a specific capability bottleneck:

• DeepSWE-style tasks: test a model’s ability to plan, modify code, and run regression checks within complex software engineering work.
• Long-horizon tasks: benchmarks such as SWE-Marathon, which emphasize sustained decision-making over long contexts.
• Enterprise-level comprehensive tasks: require completing objectives under multiple constraints, including permissions, workflows, and tool calls.
• Automation tasks and frontend generation: involve multimodal understanding, structured output, and visual layout judgment.
• Base-model comprehensive capability: measures the combined performance of knowledge coverage, logical stability, and transfer ability.

What these tasks have in common is that none of them are simple single-turn Q&A or code completion. They all require the model to maintain state across long contexts, call external tools, respect implicit constraints, and perform multi-step reasoning. In the current LLM technology stack, these remain the hardest abilities to improve reliably by simply scaling up parameters or adding more training data.

2. Where Math Capability Fits in the Evaluation

Most tasks in the original benchmark were not direct math tests, but math capability — as a form of underlying reasoning — significantly affects all of the above. Complexity analysis, algorithm design, and boundary-condition reasoning in software engineering; numerical decision-making and rule-based reasoning in enterprise scenarios; path planning and state judgment in automation tasks — all depend on mathematical reasoning.

LLM math capability is usually evaluated at three levels:

2.1 Basic Arithmetic and Symbolic Reasoning

This level tests how well a model handles arithmetic, algebra, calculus, and other foundational content. Many general-purpose models already score high on public math benchmarks, yet still make mistakes in symbolic derivation and structural simplification of complex expressions. The reason is that large language models are fundamentally statistical models, while mathematical reasoning demands strict logical consistency and rule adherence.

2.2 Mathematical Modeling and Problem Translation

Real-world problems rarely arrive as standard math problems. The model must first translate a natural-language description into a mathematical formulation, then select an appropriate method to solve it. This process involves semantic understanding, constraint identification, and abstract modeling. Many models perform well on standard problems but degrade noticeably on noisy, real-world problems — often because errors occur during the problem-translation stage.

2.3 Consistency and Verifiability of the Reasoning Process

Unlike text generation, every step of a mathematical derivation can be formally verified. Recent evaluations increasingly include process verification: they check not only the final answer, but also whether the derivation steps are consistent, with no skipped steps or logical breaks. Some models produce correct results while containing faulty inferences in their intermediate steps. This “correct but untrustworthy” pattern carries significant risk in industrial settings.

3. Why Aggregate Scores Don’t Tell the Whole Story

The headline number — 27 wins against 7 losses — obscures an important fact: different tasks carry very different weights in real business. If a model excels at general Q&A but frequently fails on long-horizon automation tasks, it is not necessarily the best choice for business scenarios that require stable execution.

Consider a software engineering example. If a model has a high success rate at generating code but frequently introduces new bugs when modifying existing code, its overall development efficiency may be lower than that of a model that generates code somewhat more slowly but modifies it more reliably. Win/loss results on a benchmark are statistical outcomes under a specific threshold, and they do not map directly to real business value.

A more useful approach is to decompose evaluation results into a capability profile across dimensions. The diagram below shows an abstract capability radar:

        Basic Reasoning
           /\
          /  \
  Tool Calling    Long Context
        |        |
  Multi-step Planning --+-- Constraint Following
          \    /
           \  /
      Stable Execution

In practice, you can weight candidate models across these dimensions. Different businesses depend on very different capabilities: customer service emphasizes semantic understanding and constraint following; engineering assistance emphasizes tool calling and long context; data analysis emphasizes mathematical reasoning and multi-step planning.

4. Technical Causes Behind the Gap

The gap between Chinese models and top closed-source models on the seven lost tasks can be understood through three factors: training data distribution, inference-time compute budgets, and the tool-calling and execution-feedback loop.

Training Data Distribution

Math and code capability depend heavily on high-quality structured training data. Closed-source models typically have access to larger-scale code repositories, mathematical derivations, and logical reasoning chains. The diversity and quality of this data directly affect how well a model generalizes to unseen tasks. If the training corpus lacks examples of complex, long-horizon tasks, the model will exhibit a “every step is right, but the overall result goes wrong” pattern when facing such tasks.

Inference-Time Compute Budgets

Complex mathematical reasoning and multi-step planning usually benefit from additional thinking steps during inference. Whether the model is allowed to perform more intermediate computation internally directly affects the final result. Some models perform well on short-answer tasks but are limited on tasks that require deep thinking. Differences in inference-time compute budgets are one of the key reasons the base-model comprehensive capability item showed a clear gap in the benchmark.

Tool Calling and Execution Feedback

Software engineering and automation tasks rely heavily on external tool calls. The model must not only generate correct call instructions, but also parse tool outputs and adjust its strategy in time. This closed-loop ability — generate, execute, observe feedback, regenerate — requires targeted multi-turn interaction training. No matter how strong a model’s text generation is, without experience from execution feedback it will still struggle to deliver good results on these tasks.

5. A Math Capability Evaluation Strategy for Model Selection

For enterprises and engineering teams, public leaderboards alone are not a sufficient basis for model selection. A more reliable approach is to build a minimal evaluation set that reflects your own business characteristics, and to focus on how models perform on real tasks.

Here is a reference evaluation workflow:

1. Extract 30–50 typical task samples from your business systems;
2. Divide the samples into:
   - Single-turn reasoning problems: assess basic math and logic;
   - Multi-step task problems: assess planning and tool-calling ability;
   - Constraint-following problems: assess rule understanding and stability;
3. Generate responses with the same inference parameters for every sample;
4. Apply fixed scoring criteria, evaluated by a model or by humans;
5. Record error types: comprehension errors, reasoning errors, or execution errors.

The distribution of error types matters more than the number of errors. If a model reasons correctly but frequently fails at the tool-calling stage, it has strong decision-making but weak execution; another model might be stable at tool calls but frequently misinterpret the task. Different error profiles point to different optimization directions and different suitable use cases.

6. Practical Ways to Improve a Model’s Math Capability

If your team has already chosen a base model, the following methods can improve its performance on math and complex tasks.

6.1 Structured Reasoning in Prompts

Explicitly requiring the model to reason step by step and output its intermediate steps can significantly improve accuracy on math tasks. An example prompt structure:

Solve the problem with the following steps:
Step 1: Extract the known conditions and the unknowns from the problem;
Step 2: List the relevant formulas or theorems;
Step 3: Derive the solution step by step, stating the basis for each step;
Step 4: Check whether the derivation is internally consistent;
Step 5: State the final answer.

This forces the model into a longer reasoning chain and reduces errors caused by skipped steps.

6.2 Multi-Round Cross-Verification

For important computation tasks, adopt a strategy of generate, independently verify, and regenerate on inconsistency. Maintain two independent prompt templates or temperature settings, generate multiple answers to the same problem, and cross-check them. If multiple independent generations agree, confidence is higher; if they disagree, route the task to a stricter verification process.

6.3 External Tools as a Safety Net

For numerical computation or symbolic derivation, combine the model with external computation tools. The model understands the problem and generates the computation expression; the external tool performs the precise calculation; the result is then returned to the model for interpretation. This preserves reasoning flexibility while avoiding the instability of LLMs in exact computation.

6.4 Build a Domain Fine-Tuning Dataset

If your business is concentrated in a specific domain, you can build a small, high-quality fine-tuning dataset from historical tasks. The dataset should contain complete reasoning processes, tool-call sequences, and final results. A fine-tuned model is usually more stable on those specific tasks than a general-purpose model. Keep in mind that the quality of the fine-tuning dataset directly determines the outcome — low-quality samples can introduce new biases instead of helping.

7. Rethinking What “#1” Rankings Actually Mean

From a technical standpoint, what a leaderboard rank means depends on the task coverage, data distribution, and evaluation criteria behind it. The 27-to-7 record shows that Chinese models are already competitive with top closed-source models on the majority of tasks. But the seven lost tasks happen to cluster in long-horizon, multi-constraint, high-complexity scenarios — precisely the capability frontier that the industry is most focused on today.

Rather than debating which model is “No. 1,” it is more valuable to look at the balance of the capability profile. A model that leads in basic reasoning and general Q&A while continuously narrowing the gap on long-horizon tasks has more complete engineering usability. For technical teams, tracking how a model changes across capability dimensions over time is more valuable than watching a single ranking.

The real gap in LLM math capability is shifting from “can it solve problems” to “can it solve problems reliably under complex constraints.” This shift means that evaluation, selection, and optimization all need to become more fine-grained. For developers and enterprises, building your own evaluation system, understanding the technical causes of the gaps, and choosing the right technical route for your business are far more pragmatic than chasing leaderboard positions.