How to Scientifically Evaluate the Mathematical Reasoning Capabilities of Large Language Models

I. Introduction: The Real Problem Behind “Who is Better at Math”

Whenever a new model is released, debates inevitably emerge in tech communities: some claim the new version has surpassed the first-tier models, while others argue it only matches an older iteration. Both sides present “hands-on test cases,” but the discussions usually end nowhere because there is no consensus on “what problems to test, how to determine correctness, or how many runs were executed.”

Treating these debates as methodological case studies rather than final conclusions reveals four universally valuable questions:

1. Why is mathematical capability always a focal point of attention?
2. Why do rankings based on intuition or public benchmarks often contradict each other?
3. When two models have similar answer accuracy, what other dimensions can differentiate them?
4. How should we correctly interpret relative positioning statements like “Model A is somewhere between Model B and Model C”?

The following sections will answer these four questions and provide an actionable evaluation framework.

II. Why Mathematical Capability is the “Touchstone”

1. Verifiable Answers. Unlike open-ended text tasks, math problems have objectively correct or incorrect answers, meaning evaluation results do not depend on reviewer preferences. This makes it one of the few capability axes that supports large-scale automated evaluation with controllable error margins.

2. Composable Reasoning. A high-difficulty math problem often requires dozens of derivation steps. Any single-step error will propagate and amplify backward. Therefore, it is highly sensitive to a model’s long-chain reasoning, self-correction, and error suppression capabilities—precisely the core bottlenecks for models in real-world development scenarios like agentic task planning and multi-step tool calling.

3. Transferable Capabilities. Industry experience widely suggests a strong correlation between mathematical reasoning proficiency and code generation quality, as well as complex instruction-following capabilities. Maintaining a math evaluation set for regression testing is typically much cheaper than maintaining a code evaluation set of equivalent scale and discriminative power, allowing it to serve as a “proxy metric” for the latter.

III. Why Standard Benchmarks are Failing

The premise of trustworthy evaluation is acknowledging the systematic biases of the evaluation tools themselves:

• Score Saturation. On mainstream public benchmarks, the accuracy of top-tier models is approaching the upper limit, with differences falling within fractions of a percentage point. At this point, the variance falls within the noise margin, and leaderboards no longer possess discriminative power.
• Data Contamination. Public problems are highly likely to appear in the training corpus. Models might have “memorized the answers” rather than “calculated them,” meaning high scores do not equate to high capability.
• Question Type Loopholes. Multiple-choice benchmarks give models an opportunity to take “shortcuts”—such as exploiting option distribution patterns or substituting values to verify—allowing them to guess a significant proportion of questions correctly without full derivation.
• Single Sampling. Many public evaluations run each problem only once. However, a model’s output inherently fluctuates under non-zero temperature sampling, making single-run results statistically unstable.

The Discriminative Value of Open-Ended Math Problems. Because of these limitations, industry focus is shifting toward “open-ended math problems”—problems without provided standard answers, multiple solution paths, and difficulty levels near the research frontier. With no ready answers in the corpus to memorize, models must rely on genuine generalization capabilities to push forward step by step. Practical experience shows that models capable of making “effective progress” (i.e., providing credible and verifiable intermediate conclusions) on such problems remain very rare. This constitutes a critical watershed for capability stratification: The gap between model tiers is not drawn by public benchmark scores, but by “what class of problems they can meaningfully tackle.”

It is important to note that “solving open-ended problems” is not black and white. A reasonable recording method is to track the distribution of “effective progress steps” a model makes on a set of open-ended problems within a fixed attempt budget (e.g., allowing N independent tries per problem), rather than just recording the presence or absence of a final answer.

IV. Chain of Thought: The Second Dimension Beyond Right and Wrong

An easily overlooked fact is: Two models with identical answer accuracy might have vastly different practical utility. The Chain of Thought (CoT) is the core entry point for examining this difference and should cover at least four dimensions:

1. Conciseness. Is there excessive redundant enumeration, repetitive trial-and-error, or self-contradiction? A strong reasoning model often provides shorter, more direct chains because its ability to “see through in one step” is stronger. Overly long, highly repetitive CoTs usually indicate the model is brute-forcing rather than performing mathematical reasoning.
2. Structural Integrity. Are there “skipped steps,” “circular arguments,” or “unsubstantiated substitutions” in the derivation? Even if these flaws don’t appear in the final answer, they determine whether the reasoning process is trustworthy, which is critical in scenarios requiring manual review of the derivation.
3. Naturalness of Style. “Writing like a human” is essentially a test of style generalization. Natural reasoning includes clear explanations of motivation (“Let’s first consider…, because…”), whereas unnatural reasoning often mechanically lists conditions. In assisting scientific research or educational scenarios, this dimension is as important as accuracy.
4. Error Locatability. When an answer is wrong, can you quickly pinpoint which step went astray? A clearly structured CoT allows for manual correction of a single step before rerunning, while a chaotic CoT offers no debugging leverage.

A practical piece of advice: Do not treat CoT length simply as a quality score. Proof questions require complete chains, while calculation questions favor concise chains. The key is to examine the information density per unit length, not the length itself.

V. Building a Reproducible Evaluation Framework

Subjective evaluation isn’t without value, but it has two fatal flaws: it is irreproducible and non-cumulative. Translating personal impressions into an automated evaluation pipeline is the only viable path for enterprise-level model selection.

1. Test Set Construction. Use stratified sampling: a Basic Layer (textbook difficulty, verifying no regression), an Advanced Layer (competition difficulty, testing derivation), and a Frontier Layer (open-ended problems, probing generalization limits). The ratio can be adjusted based on business focus. Adhere to two golden rules: apply equivalent transformations to problems and rotate them regularly to combat contamination; tag each problem with knowledge points and reasoning step counts for subsequent capability profiling, rather than just staring at the total score.

2. Execution Specifications. Fix the prompt template; avoid “chatty” queries. Set temperature to zero, run each problem at least 3 times, and record the variance. Unify answer extraction rules and use symbolic computation for equivalence checking (√2 and 1.414 should be judged as the same answer) rather than naive string matching.

3. Reference Implementation. Below is a universal evaluation skeleton. Readers can connect it to their platform’s model service API:

import statistics
from dataclasses import dataclass, field
@dataclass
class MathSample:
    pid: str
    problem: str
    reference: str      # Empty string for open-ended problems, indicating no automatic correctness check
    level: str          # basic / advanced / open
    tags: list = field(default_factory=list)
class LLMClient:
    """Universal model invocation interface. Implement the chat method according to your platform's documentation."""
    def chat(self, prompt: str, temperature: float = 0.0) -> str:
        raise NotImplementedError
def extract_answer(response: str) -> str:
    """Extract the final answer from the response, e.g., agreeing to extract the line starting with 'Final Answer:'."""
    ...
def verify(answer: str, reference: str) -> bool:
    """Recommended to use symbolic computation tools for equivalence checking to avoid misjudgments caused by string equality."""
    ...
def run_eval(client: LLMClient, samples: list, runs: int = 3) -> dict:
    report = {}
    for s in samples:
        traces, results = [], []
        for _ in range(runs):
            resp = client.chat(
                f"Solve the following problem and provide the complete reasoning process:\n\n{s.problem}",
                temperature=0.0,
            )
            traces.append(resp)
            if s.reference:
                results.append(verify(extract_answer(resp), s.reference))
        entry = {"traces": traces}
        if results:
            entry["pass_rate"] = statistics.mean(results)
            entry["stability"] = 1 - statistics.pvariance([int(r) for r in results])
        report[s.pid] = entry
    return report

Two design points are worth emphasizing: First, preserving the raw output traces is more important than preserving scores. Manual CoT analysis, rule-based scoring, and error attribution all rely on the raw text. Second, pass rate and stability must be reported as a pair. An “average 80% but intermittently correct” model and an “average 80% with stable output” model are completely different products.

4. Deployment Architecture. The evaluation pipeline is recommended to be scheduled as a cron job or scheduled task on a container platform. The test sets and results should be archived to object storage, and inference logs and evaluation metrics should be integrated into a unified logging and monitoring alerting service. Taking general container, object storage, and monitoring capabilities available on cloud platforms (such as Baidu AI Cloud) as an example, you can achieve “daily regression + threshold alerting” without building a custom ops system: once the pass rate of a certain layer drops below a preset threshold, an alert is triggered, treating model capability degradation as an engineering incident.

VI. How to Correctly Interpret “Ranking-Style” Conclusions

With the framework in place, we can revisit common community statements and distinguish which contain information and which are noise:

1. “It’s hard to say which of the two models is better”—The difference is within the noise margin. This is a perfectly valid conclusion. When the score difference is smaller than the inter-run variance, any assertion of “who is better” is an over-interpretation. The correct action is to increase the targeted size of the test set, rather than accumulating more “user impressions.”
2. “The same model performs differently on different endpoints”—A fact that must be recorded. The same underlying model can exhibit behavioral differences significant enough to affect evaluation conclusions under different deployment forms and parameter configurations. Therefore, the primary subject of evaluation records should be the triplet “Model Version + Access Endpoint + Parameter Set,” rather than a vague model name.
3. “A is somewhere between B and C”—Qualitative anchors are useful but have limited resolution. Using a familiar reference model to position a new model aligns with human cognitive habits and holds reference value when an evaluation set is lacking. However, the human brain cannot reliably distinguish differences smaller than “a single version gap.” Such positioning can only serve as a hypothesis to be tested, to be confirmed or refuted by the pipeline.
4. “Haven’t tested it, no comment”—The baseline of evaluation credibility. An evaluation report that allows outputs of “no data available” is far more valuable than one that draws conclusions everywhere. The evaluation system should explicitly write “untested” into the report template, resisting the pressure to “provide a definitive answer.”

VII. Conclusion

The debate over LLM mathematical capabilities appears to be a war of words on the surface, but at its core, it is a systems engineering problem. Public benchmarks are saturated and distorted, open-ended problems are difficult to evaluate automatically at scale, and assessing CoT quality relies on a combination of manual review and rules. All of this must be anchored by a reproducible evaluation pipeline to accumulate into a team asset. For developers and enterprise technical teams, rather than arguing endlessly in comment sections, it is better to spend two weeks building a small-scale, hundred-problem evaluation set: clarify layers, fix standards, preserve raw outputs, and report metrics in pairs. The ultimate value of evaluation is never to produce a leaderboard, but to ensure that every “who is better” question receives a verifiable, reproducible, and cumulative answer.