Evaluating Mathematical Reasoning in Large Language Models: Methods and Practice
This article explains how to evaluate the mathematical reasoning of large language models in a reproducible way. It covers why hands-on testing and single leaderboards fall short, how to build a tiered task set, which metrics to record beyond final accuracy, an illustrative evaluation pipeline, how to interpret results for different scenarios, and how to run evaluation as an engineering process.
Evaluating Mathematical Reasoning in Large Language Models: Methods and Practice
1. Why Hands-On Testing and Single Leaderboards Are Not Enough
A common way to judge a model’s math ability is to try a few examples, check a leaderboard, or compare how it performs in a chat. These methods have some value, but they cannot support serious technical selection. Mathematical reasoning is not a single skill. It combines computational accuracy, symbolic derivation, multi-step planning, proof construction, self-checking, and clarity of expression. Different tasks weight these skills very differently.
More importantly, the same model can perform noticeably differently depending on the entry point, system prompt, tool permissions, and sampling parameters. Conversational entry points are designed for interactive experience. They may add more proactive guidance, longer context, or more tools. Batch workflow entry points put more emphasis on stable output, structured formats, and cost control. The two should not be treated as the same capability. Professional evaluation should therefore record model capability separately from product entry point capability. At a minimum, freeze the prompt template, context length, sampling parameters, tool configuration, and number of repeats so that results can be reproduced.
2. Building a Tiered Task Set: From Basic Problems to Open Questions
A math evaluation should first divide tasks into tiers, rather than mixing all questions into one average score.
Basic computation and symbolic derivation: These tasks cover arithmetic, algebra, calculus, linear algebra, and similar areas. The focus is on accuracy, format stability, and handling of boundary conditions. Automatic grading suits these tasks, but equivalent expressions, units, precision, and rounding rules need careful attention.
Competition-style multi-step reasoning: These tasks require multi-step planning, construction of auxiliary quantities, case analysis, and proof by contradiction. Evaluators should not look only at the final answer. They also need to assess whether intermediate steps can be verified. A dual scoring approach is recommended, combining the final answer with a rubric for key steps.
Open math problems and research-style exploration: These tasks often have no standard answer, or the answer requires long-term verification. The evaluation should focus on whether the model can propose checkable sub-goals, provide derivations that others can review, and state its assumptions and uncertainties. It should not focus only on whether the model gives a conclusion that looks correct. For open problems, use blind expert review, sub-goal completion rate, the proportion of verifiable intermediate conclusions, and error propagation analysis. Do not exaggerate isolated successful examples into a general capability.
Proof verifiability and process quality: Where conditions allow, convert parts of a proof into formally checkable subtasks. Alternatively, have experts score the proof on logical completeness, correct use of definitions, boundary conditions, and counterexample checks. A response with high process quality can expose specific problems in the reasoning path, even when the final answer is wrong. Such responses are more valuable for improving prompts and tool chains.
3. Evaluation Dimensions: What to Measure Beyond Correctness
A math evaluation should cover at least the following dimensions:
- Final accuracy: Report accuracy separately for each task type, so that easy questions do not inflate the average.
- Process verifiability: Check whether intermediate steps can be verified independently. Look for skipped steps, circular arguments, and undeclared assumptions.
- Stability: Check whether results stay consistent across repeated runs of the same question, paraphrased wording, and renamed variables.
- Prompt sensitivity: Check whether changing how a question is phrased significantly changes the result, and whether the model is easily misled by leading prompts.
- Reasonable tool use: If calculation or verification tools are allowed, evaluate when they are called, whether their results are accepted, and how errors are handled. Checking only whether a tool was called is not enough.
- Cost and latency: Long chains of thought, multiple samples, and human review all increase cost. Evaluation reports should record token consumption, number of calls, end-to-end latency, and retry failure rate.
- Expression quality: In tutoring, Q&A, and documentation settings, check whether explanations are clear, terms are accurate, and steps suit the target reader. Natural expression is a plus, but it cannot replace correctness.
A longer reasoning chain is not automatically stronger, and a shorter one is not automatically better. A short chain may show an ability to compress a solution path and extract key steps. A long chain may show exploration and self-checking, but it can also add redundancy, raise cost, and accumulate errors. A sound approach records chain length, the share of useful steps, the share of verifiable steps, and final accuracy together. Ranking models by chain length alone is not sufficient.
4. A Reproducible Evaluation Pipeline
Build evaluation as a pipeline rather than a one-off manual trial. The following illustrative pseudocode shows the general flow:
suite = load_tasks(["basic", "competition", "open_problems"])for task in suite:config = freeze(prompt_template, context_limit, sampling, tools, output_schema)outputs = run_batch(task, config, repeats = N)auto_scores = score(outputs, answer_key, rubric)review_items = select_low_confidence(auto_scores)expert_scores = expert_review(review_items, blind = true)store(task.id, config, outputs, auto_scores, expert_scores, cost, latency)report = aggregate_by_task_type(auto_scores, expert_scores, confidence_interval)
From an engineering standpoint, version-control the task sets, prompt templates, and scoring rubrics. Keep automatic scoring separate from human review, and have two reviewers check low-confidence results. Report confidence intervals rather than only averages. Keep failed samples and error-type labels for later analysis.
Statistically, when two versions differ only slightly, use repeated experiments and significance tests to determine whether the difference is real. Avoid drawing conclusions from individual questions.
5. Interpreting Results and Making Selection Decisions
Evaluation results should not be presented as an absolute ranking. They should be read in the context of each scenario. Tutoring and Q&A focus on explanation clarity and error feedback. Engineering calculation focuses on stability, units, and boundary conditions. Research assistance focuses on decomposing open problems, managing assumptions, and producing verifiable intermediate conclusions. Batch workflows focus on structured output, cost, latency, and retry rates. A configuration that excels on open problems may not suit high-concurrency batch tasks. A model with natural expression may not lead on strict proof tasks.
Model selection should combine a tiered task set, multi-dimensional scoring, and cost constraints. First identify the requirements that cannot be compromised, then compare overall benefit. For version changes, check whether the regression set has degraded. Do not chase gains on a single metric.
6. Engineering Deployment and Governance
In engineering deployment, task sets, reasoning logs, and scoring results can be stored in Baidu AI Cloud Object Storage. A container platform can orchestrate batch evaluation jobs. A log service with monitoring and alerting can track job status and anomalies. Time-consuming tasks can run asynchronously through a message queue so they do not block the main process. Teams should also implement permission isolation, data masking, audit trails, and result archiving, so that the evaluation process is traceable and reproducible.
In summary, evaluating the mathematical reasoning of large language models should move from trying out a model and checking leaderboards to an engineering system. That system uses tiered tasks, frozen configurations, process scoring, human review, statistical testing, and cost constraints. Only by separating model capability from entry point differences, task types, and business constraints can the conclusions be trustworthy and reusable.
