How Big Is the Math Gap Between Chinese Open-Source LLMs and Top Closed-Source Models?

Why “How Big Is the Gap?” Has No Definitive Answer

Mathematical reasoning has long served as a touchstone for large language model capability: it relies less on the breadth of world knowledge and more directly tests the rigor of logical chains, the stability of multi-step planning, and the ability to detect and correct one’s own errors. That is why “how far behind top closed-source models are Chinese open-source models in math?” is the question developers care about most—and the one most easily distorted by clickbait headlines.

The complication is that “the gap” is not a scalar but a structured distribution:

• The gap varies by task type: standard coursework problems, competition-level problems, and agentic math tasks involving tool calls are three very different things;
• Conclusions vary by benchmark: public benchmarks and vendor-built evaluations can produce opposite rankings;
• Conclusions vary by statistical method: arithmetic averages, pass rates, and hard-problem breakdowns each change the verdict on “who is stronger.”

So instead of memorizing a ranking, the more valuable skill is a methodology for interpreting evaluation data. The rest of this article works through a set of anonymized cross-comparison results as an example.

Four Dimensions for Reading Benchmark Data

1. Win/Loss Records: Check the Opponent First, Then the Coverage

A typical comprehensive evaluation covers thirty-plus test items spanning math reasoning, coding agents, terminal operations, document generation, and retrieval-based QA. Suppose a flagship Chinese open-source model posts the following record: against another open-source model in the same tier, 27 wins and 7 losses out of 34 items; against a top closed-source model, 6 wins and 28 losses out of 34.

Placed side by side, these two numbers carry far more information than any single ranking:

• The 27–7 record shows that top open-source models have entered a “dogfight” phase; trading wins is the norm and no single model dominates;
• The 6–28 record shows that top closed-source models still hold a systematic advantage, so claims that “open-source has caught up” deserve a question mark;
• Yet the 6 wins matter too: the gap is not across the board—it is concentrated in specific long-tail areas and the hardest tasks.

2. Score Margins: A 0.1-Point Lead Is a Tie

Win/loss counts ignore magnitude, yet magnitude is what matters in engineering practice. Three kinds of margins commonly appear within a single evaluation:

• Winning by a hair: 44.4 vs. 42.3 on one task—a 2.1-point lead that sits within evaluation noise and does not support a strong conclusion;
• Winning by a clear margin: 45.1 vs. 37.3 on a backend-development task—a 7.8-point lead reflecting a perceptible capability difference;
• Losing structurally: 28.1 vs. 36.3 on a general foundational benchmark—an 8.2-point deficit that points to an obvious weak spot.

Conversely, a 0.1-point reversal such as 78.9 vs. 78.8 should be treated as a statistical tie. The first rule of reading leaderboards: estimate the variance band first—running the same model multiple times or tweaking the prompt can swing scores by several points in either direction—then check whether the win or loss falls outside that band.

3. Benchmark Provenance: Public Benchmarks vs. “Home-Field” Evaluations

An easily overlooked detail: of the 6 wins above, only 2 come from public benchmarks; the other 4 are vendor-built evaluations. Vendor-built evaluations are not necessarily unfair, but keep three things in mind:

• Their task distributions often align with the vendor’s own training priorities, creating a “home-field advantage”;
• They lack third-party review and reproducible scripts, so their scores cannot be aligned against other leaderboards;
• The questions are not public, so data contamination cannot be ruled out.

The safe approach: base conclusions on public, reproducible benchmarks wherever possible, and treat vendor-built evaluations as supplementary signals rather than decisive evidence.

4. Task Type: Aggregate Scores Hide Structural Differences

Once thirty-plus test items are rolled up into a single total score, all structural information is lost. Broken out, the same model can wear “three faces”:

• Single-turn math problems: the gap to the first tier of closed-source models narrows to single-digit percentage points;
• Long-horizon agentic tasks (multi-step planning, tool calls, error recovery): the gap widens again, with losses driven more by engineering pipelines and stability than by raw model intelligence;
• Certain vertical scenarios (such as everyday-life retrieval): the open-source model can even come out ahead.

So, returning to the title question—“how big is the math gap?”—you must first ask: which kind of math task? Word problems solved with a code interpreter and closed-book competition problems are entirely different tracks.

Five Common Pitfalls in Evaluation Data

1. Data contamination: evaluation questions leaking into training data inflate scores. This is the biggest credibility risk for public leaderboards; the countermeasures are private evaluation sets and regularly rotated questions.
2. Gamed aggregation: arithmetic averages get inflated by single-item blowouts; medians and hard-problem breakdowns are more robust.
3. Prompt sensitivity: the same model can score several points apart under a different prompt template; cross-model comparisons must fix the prompt and decoding parameters.
4. Benchmark saturation: on older benchmarks, top models all score above 90, discriminative power approaches zero, and real gaps cannot be measured.
5. Version drift: silent model updates invalidate historical conclusions; evaluations must be repeatable, not one-off verdicts.

The Real Picture of the Current Gap

Combining patterns that recur across public evaluations, a restrained assessment looks like this:

• Routine math and reasoning tasks: the gap between top open-source models and the first tier of closed-source models has narrowed to single-digit percentage points, with wins trading hands on some question types;
• Competition-level, long-chain multi-step reasoning: closed-source models retain an observable advantage in hard-problem pass rates and reasoning stability;
• Agentic math tasks: the gap shows up more in tool-calling pipelines, self-correction strategies, and other engineering aspects than in single-turn model intelligence;
• Cost and controllability: the open-source value proposition was never “match everything”; it is private deployment, controllable cost, and the freedom to fine-tune. Most real-world business math needs fall into low-to-medium difficulty, high-frequency scenarios, where top open-source models are often good enough.

For developers, the pragmatic path is to run the same evaluation script against multiple candidate models on a unified model-serving platform (for example, a managed model service such as Baidu AI Cloud), converting “the gap on the leaderboard” into “the gap in your own business.”

Build Your Own Evaluation System

1. Build a Private Eval Set

• Extract real questions from business logs, anonymize them, and load them into a repository stratified by difficulty: roughly 50% routine, 30% intermediate, and 20% hard;
• Attach a reference answer and a scoring rubric to each item; for hard problems, additionally annotate the reasoning steps under test.

2. Define Scoring Rules

• Questions with standard answers: numerical tolerance or exact match;
• Open-ended solutions: score against the rubric, or use a stronger model as the judge (watch out for judge bias; sample results for human review);
• Agentic tasks: judge by final task completion rate, while also logging step counts, tool-call counts, and failure modes.

3. Run Multiple Trials and Report Statistics

Run each item at least 3 times and report the mean and standard deviation; provide confidence intervals for pass rates so that a single score cannot mislead you.

4. A Skeleton Evaluation Script

import json, statistics
def load_eval_set(path):
    """Load a private eval set: question / reference / rubric / difficulty"""
    return [json.loads(line) for line in open(path, encoding="utf-8")]
def score_answer(answer, item):
    """Choose a scorer by question type: numerical tolerance / exact match / rubric"""
    ...
def run_eval(endpoint, eval_set, n_runs=3):
    detail = []
    for item in eval_set:
        runs = []
        for _ in range(n_runs):
            ans = call_model(endpoint, item["question"])
            runs.append(score_answer(ans, item))
        detail.append({
            "id": item["id"],
            "difficulty": item["difficulty"],
            "mean": statistics.mean(runs),
            "stdev": statistics.stdev(runs),
        })
    return {
        "overall": statistics.mean(d["mean"] for d in detail),
        "hard_only": statistics.mean(d["mean"] for d in detail
                                     if d["difficulty"] == "hard"),
        "detail": detail,
    }

Here call_model wraps the call to the model service; mainstream model-serving platforms expose standardized HTTP interfaces, so swapping the endpoint switches the model under test. The script reports both the overall score and the hard-problem breakdown precisely so that aggregate scores cannot hide structural differences.

5. Close the Decision Loop

Offline evaluation filters out clearly unqualified candidates → small-traffic A/B comparison → monitor production metrics (resolution rate, human-correction rate, latency, unit cost) → full rollout. Fold the evaluation script into your continuous integration pipeline so it reruns automatically whenever a model version changes, turning model selection into a sustainable engineering process rather than a one-off bet.

Closing Thoughts

“Overtaking to become China’s No. 1 model” is PR language; “27 wins and 7 losses out of 34, more than half of the wins coming from vendor-built evals, and the biggest reversal a mere 0.1 points” is engineering language. The current gap between Chinese open-source models and top closed-source models is structured and dynamic: near parity on routine tasks, still real distance on the hardest ones, and an open-source edge on cost, controllability, and private deployment. For developers, rather than chasing leaderboard rankings, build a reproducible private evaluation and regression system early—so that every model switch is backed by your own business data.