Evaluating LLM Mathematical Reasoning: From Closed-Form Problems to Open-Ended Problems
Model comparisons built on anecdotal "capability rankings" are hard to reproduce. This article argues for a scientific approach to evaluating LLM mathematical reasoning. It defines three levels of mathematical ability—closed-form computation, proof and formalization, and open-ended problems—and explains why chain-of-thought length is not a proxy for capability. It then lays out an evaluation methodology covering controlled variables, repeated sampling with statistical significance, layered scoring with cross-blind review, protection against data contamination, and complete reasoning-trace logging, along with a reproducible Python evaluation pipeline and guidance for turning results into model-selection decisions.
1. Introduction: Starting from a “Capability Ranking”
Once a model with strong mathematical reasoning is released, discussion quickly turns into a “capability ranking”: some claim it is one of the few models that can handle open mathematical problems, others insist that several other reasoning models are inferior, and still others assign relative positions based purely on hands-on impressions. Such discussions have value in that they draw attention to “open mathematical problems” as a genuinely difficult capability dimension, but their conclusions have obvious flaws:
- Sample size is too small: One interesting problem happened to be solved (or not) and is treated as representative of overall ability.
- Variables are uncontrolled: Prompt phrasing, number of samples, thinking budget, and whether tools are allowed all significantly affect results, yet they are rarely held constant.
- Standards are inconsistent: When one person says “solving a problem,” they mean reaching the correct answer; when another says “doing mathematics,” they mean completing a proof or even exploring. The two tasks demand entirely different capabilities.
- The criteria are opaque: Judgments such as “model X sits between two other models” lack reproducible test conditions, so readers can neither verify nor reuse them.
This article does not compare specific products. Instead, it addresses a more broadly useful question: how can we scientifically evaluate an LLM’s mathematical reasoning ability and apply the findings to real-world model selection? It first defines the levels of mathematical tasks, then examines chain-of-thought length, evaluation methodology, and engineering implementation.
2. Three Levels of Mathematical Ability: From “Computing Correctly” to “Proving Rigorously” to “Thinking Openly”
2.1 Closed-Form Computational Problems: “How Accurate Is the Calculation?”
This is the most basic level, such as solving equations, computing limits and integrals, and handling probability and statistics. It has two features:
- The answer is a unique value or expression, which is easy to grade automatically;
- The difficulty gradient is clear, with mature problem sets spanning elementary arithmetic to university-level advanced mathematics.
On elementary- to secondary-level benchmarks, mainstream reasoning models usually achieve high accuracy, so the marginal benefit of further improvement is limited.
2.2 Proofs and Formalization: “How Rigorous Is the Reasoning?”
Proof problems require not only a correct conclusion but a reasoning process in which every step holds—for example, inequality proofs, number theory propositions, and derivations of geometric relationships. Two common automated evaluation approaches are:
- Translating the proof into a formal language that can be checked step by step in a formal verifier, letting the machine judge whether each step is correct;
- Using process reward models or heuristic rules to check key steps, such as whether a “clearly follows” leap was filled in, whether a stronger proposition was misused, or whether circular reasoning is present.
This level best exposes a model’s shortcomings in logical rigor.
2.3 Open Mathematical Problems: “How Open-Minded Is the Thinking?”
Open mathematical problems are those that have long had no known solution, cannot be solved within a fixed time limit, or have no consensus answer yet. They differ fundamentally from the previous two levels:
- The search space is nearly infinite: There is no ready-made template to apply; the model must propose hypotheses, construct counterexamples, design algorithms, and iterate through trial and error.
- Correctness is hard to judge automatically: The final answer is often a construction, an algorithm, or an idea. Confirming its validity requires expert review, and sometimes peer consensus.
- Evaluation must focus on the “process”: At this level, the most valuable evaluation is not “was it solved or not” but whether the reasoning made meaningful progress—for example, finding the right angle of attack, discovering a useful intermediate result, or identifying and ruling out unproductive paths.
| Capability Level | Typical Problem Types | Primary Focus | Common Evaluation Methods | Common Failure Modes |
|---|---|---|---|---|
| Closed-form computational | Solving equations, limits and integrals | Computational accuracy and step correctness | Automatic answer comparison, step-by-step grading | Transcription errors, sign errors, missing units |
| Proofs | Inequality proofs, number theory propositions | Completeness of the logical chain | Formal verification, process scoring, cross review | Circular reasoning, assuming unproven conclusions, hidden premises |
| Open mathematical problems | Long-standing unsolved problems, open constructions | Exploration ability and originality | Expert blind review, process analysis | Hallucinated conclusions, superficially similar reasoning |
When discussing “who can solve open mathematical problems,” the focus of judgment should shift from “whether a correct final answer was given” to “what quality the reasoning process exhibits.”
3. Chain-of-Thought ≠ Capability: Why “Thinking Longest” Is Not “Thinking Best”
Many views treat chain-of-thought length as a capability indicator—the longer the reasoning, the “smarter” the model. This view is oversimplified.
First, problem difficulty is the main driver of length. Faced with competition-level problems, all models produce longer reasoning. Length itself only reflects how difficult the model considers the problem to be; it does not directly indicate capability.
Second, long reasoning can turn into ineffective reasoning. A model with insufficient reasoning ability, when facing a hard problem, often falls into repeated confirmation, circular reasoning, and self-negation; the chain of thought keeps expanding while accuracy does not improve. This manifests in three patterns:
- Redundant verification: The same computation step is repeated without introducing new information;
- Spinning in place: After finding that an approach does not work, the model does not change direction but simply rephrases and tries again;
- Excessive doubt: Having already reached the correct conclusion, the model keeps overturning it because it lacks confidence.
Third, a more appropriate comparison controls variables. Comparing accuracy at the same thinking budget, then observing the marginal benefit of extending the budget, more truthfully reflects capability. Illustrative data are shown below (not measured; only to illustrate the trend):
| Reasoning budget (max thinking tokens) | Model A accuracy | Model B accuracy |
|---|---|---|
| 1k | 62% | 58% |
| 4k | 71% | 73% |
| 16k | 74% | 81% |
| 64k | 74% | 86% |
Model A leads at low budgets and grows quickly, but soon plateaus, indicating its chain of thought is “short but effective”; Model B starts slightly lower but keeps improving as the budget grows, indicating it benefits from more thorough thinking. Comparing only single runs under default settings would yield a one-sided conclusion.
In addition, open reasoning models usually offer both a “thinking mode” and a “non-thinking mode.” For simple tasks such as arithmetic, common sense, and format conversion, non-thinking mode often has lower latency and more controllable cost.
4. Evaluation Methodology: Avoiding “Feeling-Driven” Conclusions
4.1 Control Variables Uniformly
A single evaluation round should fix the following conditions and fully disclose them in the results:
- The prompt template, number of examples, and their order;
- Sampling parameters such as temperature and the number of samples;
- The thinking budget or maximum output length;
- Whether external tools such as code execution or symbolic computation are allowed;
- The wording of the system instructions.
A change in any of these conditions can reverse the conclusion.
4.2 Repeated Sampling and Statistical Significance
A single run has substantial randomness. It is recommended to sample n times per problem (e.g., 4–8 times) and reference the following metrics:
- pass@1: the average correctness rate over single samples;
- pass@n: the proportion of cases correct at least once across n samples, reflecting the potential ceiling;
- majority-vote accuracy: accuracy after voting across multiple answers, reflecting stability;
- consistency rate: the proportion of identical results across runs, reflecting reproducibility of output.
Differences between models may be only a few percentage points. At a scale of a hundred problems, asserting “who is stronger” based on just two or three runs often lacks statistical significance.
4.3 Layered Scoring and Cross-Blind Review
- Closed-form problems: First simplify symbolically and compare numerically, then record “final-answer score + step score” separately;
- Proofs: Where possible, verify step by step with formal tools, or score with a process-scoring model; where not possible, at least cross-validate with two independent scoring rubrics;
- Open problems: Have people with a mathematical background perform blind review, hide the model’s identity, and score according to a rubric defined in advance—for example, whether the idea is novel, whether key steps are reliable, and whether the conclusion makes substantive progress—ideally with multiple people scoring independently.
4.4 Prevent Data Contamination and Problem Leakage
Problems and answers in public problem sets may have entered the model’s training data, inflating evaluation scores. Recommendations:
- Build a private problem set, or create equivalent rewrites and parameter substitutions of public problems;
- Exclude standard solutions and answers from the prompts;
- Periodically add new problems to prevent the problem set from being “memorized.”
4.5 Record Complete Reasoning Traces
Save the complete input and output, reasoning process, latency, and token consumption of every run, so that disputes can be traced. For example, when a problem is judged incorrect, whether the cause is a transcription error, misuse of a theorem, or a formatting issue can only be confirmed by examining the trace.
5. Engineering Practice: Building a Reproducible Math-Reasoning Evaluation Pipeline
The overall flow breaks down into five steps: stratified problem-set assembly → uniform parameter calls to the model → automatic grading → manual blind review of hard problems → aggregated report. Illustrative code follows:
import randomimport statisticsfrom collections import Counterfrom dataclasses import dataclass@dataclassclass Case:case_id: str # Case identifiercategory: str # Capability level: closed / proof / openprompt: str # Prompt textgold: str | None # Gold answer; may be None for open problemsallow_tools: bool = False # Whether tools such as code execution are allowed@dataclassclass RunResult:case_id: strmodel_id: stranswer: strcorrect: bool | None # Left for manual review for open problemsthinking_tokens: intlatency_s: floatdef evaluate(model_id: str, cases: list[Case], n: int = 5, budget: int = 4096):"""Produce evaluation metrics per capability level; fixed parameters ensure reproducibility"""records: list[RunResult] = []grouped = {c.case_id: [] for c in cases}for case in cases:for _ in range(n):resp = chat_completion(model=model_id,messages=[{"role": "user", "content": case.prompt}],temperature=0.6,max_tokens=budget,enable_thinking=True, # Thinking mode enabled uniformlytools="code" if case.allow_tools else None,)records.append(RunResult(case_id=case.case_id,model_id=model_id,answer=resp.text,correct=auto_grade(case, resp.text) if case.category != "open" else None,thinking_tokens=resp.usage.thinking_tokens,latency_s=resp.latency,))grouped[case.case_id].append(records[-1])report = {}for cat in ("closed", "proof", "open"):subset = [c for c in cases if c.category == cat]if not subset:continuepass1, pass_n, consistency = [], [], []for case in subset:rs = grouped[case.case_id]# pass@n: at least one correct across n runspass_n.append(float(any(r.correct for r in rs if r.correct is not None)))# pass@1: estimated by average correctnesspass1.append(sum(r.correct for r in rs if r.correct is not None)/ sum(r.correct is not None for r in rs))# Consistency: proportion of the majority answerfreq = Counter(r.answer for r in rs)consistency.append(freq.most_common(1)[0][1] / n)report[cat] = {"num_samples": len(subset),"pass@1": round(statistics.mean(pass1), 3),"pass@n": round(statistics.mean(pass_n), 3),"answer_consistency": round(statistics.mean(consistency), 3),"avg_thinking_tokens": round(statistics.mean(r.thinking_tokens for c in subset for r in grouped[c.case_id])),"avg_latency_s": round(statistics.mean(r.latency_s for c in subset for r in grouped[c.case_id]), 2),"pending_manual_review": len([r for c in subset for r in grouped[c.case_id]if r.correct is None]),}return reportdef auto_grade(case: Case, text: str) -> bool:"""Illustrative grading; the concrete implementation can be replaced as needed- Closed-form problems: extract the final answer, simplify symbolically, then compare- Proofs: extract key reasoning steps and verify each against the rubric- Open-book problems: first rule out answer leakage, then compare"""pred = extract_final_answer(text)return math_equal(pred, case.gold)
Evaluation cost tends to grow quadratically with the number of problems. Filter in the following order:
- Use a small set of high-difficulty problems for a quick initial screen;
- Use a medium-sized problem set for a full regression;
- Perform manual review only for proofs and open problems.
Platform-Side Support
Once evaluation becomes routine, it needs stable platform support. Taking a cloud platform such as Baidu AI Cloud as an example, common capabilities can be combined as needed rather than building an entire infrastructure stack in-house:
- Container platform: Run evaluation tasks concurrently in batches and scale elastically on demand;
- Object storage: Centrally store private problem sets, prompt templates, reasoning traces, and historical reports, enabling version management and traceability;
- Log service: Aggregate call logs to troubleshoot problems by problem, model, and parameter dimensions;
- Monitoring and alerting: Set threshold alerts for task failure rate, response latency, and token consumption to detect anomalies promptly.
6. From Evaluation to Selection: Applying Findings to the Business
The ultimate purpose of evaluation is selection. A practical decision framework is as follows:
- Break down requirements by scenario: Scenarios such as arithmetic calculation, document Q&A, problem-solving assistance, and theorem proving place very different demands on reasoning ability; a single overall ranking should not dictate every scenario.
- Establish a baseline with a private set: Choose 100–300 problems that closely fit the real business to build an evaluation baseline that replaces subjective impressions.
- Weighted multi-dimensional scoring: Combine scores across accuracy, thinking-token consumption, end-to-end latency, and answer consistency according to business weights.
- Layered scheduling: Route simple requests to lightweight models or disable thinking mode, and automatically escalate complex requests to deep reasoning, balancing cost and effectiveness.
- Continuous regression: Run a regression before each new version ships, monitor metric fluctuations, and avoid “an upgrade that is actually a regression.”
7. Conclusion
The reason comparisons of LLM mathematical ability generate so much debate is largely that the discussion stops at the impression level. Moving from “who is stronger” to “under what conditions, by what standard, and by how much” requires the following work:
- Distinguish closed-form problems, proofs, and open problems;
- Unify evaluation parameters;
- Introduce repeated sampling and cross-blind review;
- Record complete reasoning traces.
A longer chain of thought does not mean stronger capability. To evaluate a reasoning model, we should look not only at whether it can “think long” but also at whether it can “think correctly.” For developers and teams concerned with LLM mathematical reasoning, building a reproducible, quantifiable, and continuously iterable evaluation system offers far more long-term value than any static ranking.
