Evaluating LLM Mathematical Reasoning: Metrics, Chain of Thought, and Model Selection
Learn how to evaluate LLM mathematical reasoning with practical scoring rubrics, reasoning-quality metrics, and a private test set. This guide compares detailed and concise reasoning, provides a reproducible evaluation example, and explains how to balance accuracy, latency, cost, and security when selecting a model.
1. Define What You Want to Measure
Before evaluating a model, break “mathematical ability” into dimensions that can be scored. This guide uses three: open-ended problem solving, chain-of-thought quality, and clarity of explanation. Each needs an operational definition, a scoring method, and a passing threshold. Without these, scores from different reviewers will not be comparable.
Open-Ended Problem Solving
Definition: The problem has no ready-made solution template. The model must choose an approach, construct a mathematical model, and reach a verifiable conclusion.
Example problem: Given any sequence of n integers, prove that there is a nonempty block of consecutive terms whose sum is divisible by n.
Scoring rubric, out of 4:
- 4: The conclusion is correct, the proof is complete, and every step can be verified.
- 3: The conclusion is correct. A few steps are omitted but can be filled in.
- 2: The approach is sound, but a key lemma is missing.
- 1: There is a clear approach, but the derivation or conclusion contains errors.
- 0: There is no meaningful progress, or the conclusion is incorrect.
Passing threshold: An average score of at least 3.0 across the same problem set.
Chain-of-Thought Quality
Definition: Whether the reasoning steps presented between the problem statement and the conclusion are necessary, orderly, and free of logical gaps.
Use three metrics:
- Effective step count: The number of steps directly relevant to the conclusion.
- Redundancy rate: The proportion of all sentences that repeat earlier content or are irrelevant to the conclusion.
- Key-step coverage: The proportion of human-annotated essential steps covered by the output.
Example thresholds are a redundancy rate below 20% and key-step coverage of at least 90%. These are starting points, not universal standards; adjust them to reflect the risks of your application.
Clarity of Explanation
Definition: Whether the intended reader can understand the explanation and whether concepts are used accurately.
Scoring method: Two reviewers independently assign scores from 1 to 5. If their scores differ by more than one point, they discuss the output and agree on a final score.
2. Detailed vs. Concise Reasoning
| Dimension | Detailed reasoning | Concise reasoning |
|---|---|---|
| Accuracy | Can cover more derivation paths and reduce omissions, but longer outputs may drift from the main argument | Depends on the correctness of key transitions; incorrect jumps can be harder to detect |
| Response latency | Longer generation time | Faster responses |
| Token cost | Higher | Lower |
| Suitable tasks | Derivations requiring exhaustive case analysis or step-by-step verification | Problems with established solution methods and tight response-time requirements |
| Main risk | Loss of focus and divergent conclusions | Hidden logical gaps and more difficult auditing |
The better choice depends on the task, model, and data. A shorter answer does not necessarily indicate a stronger model. Correctness must still be verified on a test set.
3. Example Prompt and Validation Method
The following example provides a complete prompt setup and evaluation procedure.
Problem: Evaluate the definite integral ∫₀¹ x·eˣ dx.
System instruction:
You are a mathematics teaching assistant. Present the derivation in no more than 6 steps, with one transformation per step, and give the result on the final line. Do not repeat the problem or explain elementary concepts.
User input:
Evaluate ∫₀¹ x·eˣ dx.
Expected output:
- Let u = x and dv = eˣ dx, so du = dx and v = eˣ.
- By integration by parts, ∫₀¹ x·eˣ dx = [x·eˣ]₀¹ − ∫₀¹ eˣ dx.
- Evaluate the terms: [x·eˣ]₀¹ = e and ∫₀¹ eˣ dx = e − 1.
- Substitute to obtain e − (e − 1) = 1.
Result: 1
Run settings: Set temperature to 0 and max_tokens to 512. Run the same problem three times. A temperature of 0 reduces variation but does not guarantee identical outputs, so repeated runs are still necessary.
Evaluation:
- Mark the result as correct if the final answer exactly matches the reference answer, 1.
- Count effective steps and compare the count with the four expected steps.
- Record latency and token usage for each run.
4. Build a Small Private Test Set
Difficulty Levels
Use three levels:
- Basic: Single-step calculations, such as evaluating a limit.
- Intermediate: Algebraic proofs requiring two or three steps.
- Difficult: Open-ended problems requiring the construction of an auxiliary quantity.
Problem Distribution
A suggested distribution is 30% basic, 40% intermediate, and 30% difficult. Include coverage of calculus, linear algebra, number theory, and combinatorics.
Sample Size
With too few problems, chance variation can strongly influence the results. Use at least 10 problems per mathematical branch and at least 40 problems overall.
Prevent Test-Set Contamination
Do not publish the test questions. Once public, they may enter training data and distort subsequent evaluation scores.
Annotation and Review
Annotate every problem with a reference answer and its essential solution steps. Manually review at least 20% of model outputs and record the findings to help assess reviewer agreement.
Automated grading works well for problems with reference answers, using exact matching or numerical comparison. Open-ended problems still require human scoring against the rubric.
5. Evaluation Workflow
- Define the task set and document the reference answer and key steps for every problem.
- Select metrics: accuracy, average open-ended problem score, redundancy rate, P95 latency, and token cost per problem.
- Fix inference parameters and run every problem at least three times.
- Grade automatically where possible, and have reviewers score open-ended problems using the rubric.
- Review a sample of outputs and measure agreement between reviewers.
- Record the model version, test date, inference parameters, and test-set version to support reproducibility.
6. Cost, Latency, and Security in Enterprise Deployment
Enterprises often deploy inference services using container platforms and GPU resources from major cloud providers. The choice should reflect compliance requirements, service regions, and the existing technology stack. Regardless of provider, consider four areas.
Cost
For token-priced services, longer reasoning increases costs, potentially sharply as problems become more difficult. Estimate monthly spending by multiplying the average cost per problem by average daily request volume and the number of days in the month.
Latency
Track both P50 and P95 latency. P95 provides a better view of slow responses experienced by users and is useful for setting an acceptable user-experience threshold.
Data Security
Problems and user inputs may contain sensitive information. Before using a service, check the provider’s data-retention period and policy on using customer data for training. Remove or mask sensitive information when necessary.
External Tool Calls
If the model uses a calculator or code sandbox, restrict execution permissions, timeouts, and network access. Validate tool results rather than treating them as automatically correct.
7. Model Selection Recommendations
If your workload consists mainly of open-ended problems and errors carry a high cost, prioritize average open-ended scores and manual review findings. Be prepared to accept higher latency.
For high-volume processing of standard problems with strict response-time requirements, prioritize accuracy and P95 latency. Concise reasoning may help control costs in this setting.
In either case, evaluate models on your own test set before making a decision. Public leaderboards may use different problems, inference settings, and scoring criteria from those relevant to your application. Treat them as a starting point for shortlisting, not as the final basis for selection.
