Mathematical Reasoning Evaluation for Large Language Models: A Reproducible Practical Framework
This article presents a reproducible framework for evaluating mathematical reasoning in large language models. It covers five evaluation dimensions, a layered benchmark design, answer-judging methods, statistical interpretation, common biases, and deployment practices. Readers can use the framework to compare models and configurations with more reliable process-level and outcome-level evidence.
Why Mathematical Reasoning Needs a Dedicated Evaluation Design
Mathematics is one of the more direct settings for testing reasoning ability. Problems usually have clear correct answers, and conclusions can be verified. This also introduces a pitfall: looking only at the final answer can easily overlook errors in the reasoning process. A solution with a flawed step that happens to produce the correct numerical answer does not indicate reliable reasoning. On the other hand, research-level open problems have no standard answers and can only rely on judgment by domain experts. Therefore, an evaluation framework needs to cover both whether the answer is correct and whether the process is rigorous.
Breaking Down the Evaluation Dimensions
We recommend measuring mathematical reasoning ability across the following five dimensions:
- Correctness: Whether the final conclusion matches the reference answer.
- Process rigor: Whether each step of the derivation is justified, and whether there are skipped steps, circular arguments, or missing conditions.
- Stability: Whether the conclusion remains consistent when the same problem is solved multiple times.
- Expression quality: Whether the steps are clear, notation is defined consistently, and a reader can verify the solution step by step.
- Resource cost: Whether reasoning length, response time, and invocation cost are within an acceptable range for the business.
The weight of each dimension depends on the application scenario. Educational tutoring places more emphasis on process and expression, while engineering computation places more emphasis on correctness and stability.
Building a Layered Evaluation Set
The evaluation set is the foundation of the entire process. It can be roughly divided into three difficulty tiers: basic calculation and algebraic transformation, competition-style comprehensive problems, and open research questions. Layering prevents averages from hiding differences. A model may be near perfect on basic questions but drop noticeably on comprehensive problems; if only overall accuracy is reported, this information is lost.
Questions can come from public question banks or a self-built question bank. Regardless of the source, data contamination must be checked, that is, whether evaluation questions have already appeared in the training corpus. Common practices include approximate string matching and testing equivalent rewrites of questions to see whether scores drop significantly.
Each question should include a reference answer and key solution points, and should be independently reviewed by two annotators. When the two annotators disagree, a third person should arbitrate, and the basis for arbitration should be recorded.
Answer-Judging Methods
Judging methods vary by question type.
- Multiple-choice and integer answers: Exact matching is sufficient.
- Algebraic expressions: First perform symbolic normalization, then check equivalence, avoiding false judgments caused by different notation.
- Numerical results: Set a reasonable tolerance range and state where the tolerance comes from.
- Proof and open-ended problems: Use a rubric to score items such as completeness, correctness, and rigor, have multiple experts conduct blind reviews, and test inter-rater consistency.
The following pseudocode expresses only the logical structure and does not depend on a specific language or platform:
function evaluate(model, problem_set, sample_count):records = []for each problem in problem_set:for i from 1 to sample_count:answer, reasoning = model.solve(problem.statement)correct = judge(answer, problem.reference_answer)append to records: {problem_id: problem.id, difficulty: problem.difficulty, correct: correct, process_score: human_or_rule_score(reasoning)}return summarize(records)
This process has four key points: sample each question multiple times, record difficulty labels, score the process and the conclusion separately, and retain raw records during the statistics stage for review.
Result Statistics and Interpretation
Let the number of questions be n and the number of correct answers be c. The accuracy is p_hat = c / n. With small sample sizes, accuracy can fluctuate substantially. The standard error sqrt(p_hat(1 - p_hat) / n) can be used to estimate uncertainty, or a Wilson confidence interval can be reported. If the confidence intervals for two versions overlap substantially, one should not directly conclude that one is better than the other.
When multiple samples are used, report both single-sample accuracy and majority-vote accuracy. The gap between them reflects output instability. Stratified results should be reported separately, with representative error cases included to help identify problem types.
Common Biases and How to Avoid Them
- Impression bias: Seeing the answer before scoring can lead to influence from writing style or length. The solution is blind review, hiding output source and length information.
- Prompt sensitivity: Different prompt templates may produce noticeable differences. Templates should be fixed and recorded, and several template variants should be compared when necessary.
- Length preference: Longer reasoning is not necessarily more correct. The relationship between reasoning length and accuracy should be analyzed separately during evaluation.
- Insufficient sample size: Conclusions from a few dozen questions are not enough to support model selection decisions. Ensure a sufficient number of questions at each difficulty level.
- Evaluation leakage: If questions or answers enter training data, scores can become artificially inflated. The question bank needs to be updated regularly.
Deployment Recommendations
- Establish a baseline: After every model or configuration change, run a regression evaluation on the same evaluation set.
- Keep versioned records: Save the evaluation set version, judging rule version, sampling parameters, and run date to ensure reproducibility.
- Align with business scenarios: Educational tutoring, research assistance, and engineering computation have different key metrics, so the proportion of each question type in the evaluation set should be adjusted accordingly.
- Review continuously: Periodically sample and check automatic judging results, and fix gaps in the judging rules.
