Evaluating LLM Math Capabilities: A Systematic Framework and Engineering Practices
As large language models are increasingly deployed in math Q&A, formula derivation, data analysis, and logical reasoning, engineering teams need a reliable way to answer one question: does a given model actually possess usable math capability? Judging by a handful of problems run once is misleading, because math ability is strongly scenario-dependent. This article presents a systematic, reproducible evaluation framework covering problem-set design, difficulty tiers, outcome and process scoring, stability testing, open-ended rubrics, engineering cost, and online regression. It shows how to build a sustainable evaluation pipeline, select models by fit rather than by an abstract "best" ranking, and avoid common pitfalls—so teams can tell whether a model can reliably complete math tasks in a specific business context, whether its reasoning is trustworthy, whether errors are controllable, and whether cost is acceptable.
1. Introduction: Math Capability Can’t Be Judged by Single Cases
Once large language models entered business scenarios such as math Q&A, formula derivation, data analysis, and logical reasoning, engineering teams often face the same question: how do you determine whether a model has usable math capability?
Many evaluators rely on a simple approach—pick a few math problems, have the model answer them once, and conclude that the model is strong when the results are correct and weak when they are wrong. The problem with this approach is that math capability is strongly scenario-dependent. Basic arithmetic, algebraic equations, probability and statistics, calculus, geometric proofs, and open-ended modeling tasks each place different demands on a model. Some models perform stably on short chains of computation but stumble during multi-step derivation or abstract proofs, skipping steps, dropping conditions, or incorrectly substituting variables. Others produce reasonably complete approaches on open-ended problems yet fail on standardized questions because of formatting, precision, or unit issues.
Evaluating a model’s math capability therefore needs to evolve from a “single Q&A experience” into “systematic engineering evaluation.” This covers not only problem design but also scoring methods, reasoning-process assessment, stability analysis, cost and efficiency, online regression, and risk governance. The following sections present a practical, deployable evaluation framework from a technical-engineering perspective.
2. Evaluation Goals: From “Can It Compute” to “Can It Reliably Complete Math Tasks”
A complete math-capability evaluation system should answer at least the following questions:
Is the result correct? Does the final answer meet expectations and can it be judged automatically?
Is the process trustworthy? Is the reasoning chain complete, or does it skip steps, omit assumptions, jump logically, or contradict itself?
Is it stable across multiple runs? Under repeated sampling or different prompting, does the same problem still maintain a high pass rate?
Is it robust against perturbations? When variable names change, problem statements are rephrased, data formats shift, or redundant conditions are added, can the model still solve correctly?
Can it handle open-ended problems? For questions with no single standard answer, can it model the problem reasonably, provide verifiable steps, and state the limiting conditions?
Are cost and latency acceptable? Math reasoning often requires longer outputs and more computation steps. Businesses need to judge whether this capability is usable within real-world cost constraints.
Math-capability evaluation therefore cannot rely on a single metric such as accuracy; it should build a multi-dimensional metric system.
3. Problem-Set Design: First Make the Problems Themselves “Evaluable, Judgeable, Reproducible”
The first principle of high-quality evaluation is that the evaluation set itself must be sound. If problems are vague, answers ambiguous, and standards inconsistent, then all downstream metrics are untrustworthy.
3.1 Problem Sources
Evaluation problems can come from the following types:
- Standard examples from basic math textbooks;
- Adapted versions of competition problems, practice problems, and word problems;
- Real statistical calculations, metric analyses, and formula derivations that arise in business scenarios;
- Math proofs that require multi-step reasoning;
- Open-ended questions that require building a model, making assumptions, and drawing conclusions;
- Realistic problems that contain distractor information and require filtering conditions.
Note that problems should avoid depending entirely on a fixed text, a specific website, or a particular problem bank, so that the model cannot answer from memory—otherwise the evaluation fails to reflect true reasoning ability.
3.2 Difficulty Tiers
It is recommended to divide evaluation problems into multiple levels:
| Level | Type | Evaluation Focus |
|---|---|---|
| L1 | Basic computation | Arithmetic accuracy, units, precision |
| L2 | Algebraic equations | Equation transformation, root finding, condition checking |
| L3 | Probability and statistics | Understanding distributions, expectation and variance, conditional probability |
| L4 | Calculus | Limits, derivatives, integrals, series convergence |
| L5 | Geometry and linear algebra | Spatial relationships, matrix operations, vector derivation |
| L6 | Proofs and counterexamples | Logical completeness, use of assumptions, counterexample construction |
| L7 | Open-ended modeling | Modeling soundness, clarity of assumptions, explainability of conclusions |
Different business scenarios demand different difficulty levels. For example, financial statement analysis focuses more on L1–L3, while research assistance or complex consulting focuses more on L4–L7.
3.3 Problem Design Principles
A reusable evaluation problem should ideally satisfy the following conditions:
- Clear objective: The problem has a single or decomposable goal and does not mix multiple capability dimensions;
- Judgeable answer: The final answer has a clear standard, or can be scored by a rubric;
- Traceable steps: Key steps can be broken into checkable nodes;
- Controlled distractors: Irrelevant conditions may be added, but they must not create semantic ambiguity;
- Version-manageable: Problems, answers, scoring rules, and prompt templates should all be versioned;
- Resistant to memorization: Variants can be generated via variable substitution, numerical changes, and narrative rewriting.
3.4 Structured Sample
Each evaluation problem should be stored as structured data rather than plain text. For example:
{"problem_id": "math-001","category": "calculus","difficulty": "L4","question": "Given the rate of change of a function on a certain interval, determine the number of its extrema and explain the basis.","expected_answer": "Must list the key judgment steps and provide a final conclusion.","rubric": ["Correctly identifies the properties of the function","Completes the key derivation","Explains boundary or limiting conditions","States a clear final conclusion"],"tolerance": "Conclusion-type problems are scored by key points"}
This structured approach supports automatic scoring, manual review, error attribution, and later version regression.
4. Metric System: Balancing Results, Process, Stability, and Efficiency
Evaluation metrics are recommended to fall into four categories: outcome metrics, process metrics, stability metrics, and engineering metrics.
4.1 Outcome Metrics
Outcome metrics measure whether the final answer is correct. Common metrics include:
- Accuracy: number of correctly answered problems / total number of problems;
- Partial-credit rate: partial score awarded by step or key point;
- Pass@k: the proportion of cases where at least one of multiple attempts is correct;
- Tolerance-correct rate: numerical problems counted as correct within an allowed error margin.
For a numerical problem, tolerance can be defined as:
function is_close(actual, expected, tolerance):return absolute(actual - expected) <= tolerance
The size of the tolerance should be set according to the business. Scenarios such as finance, engineering, and healthcare usually require stricter tolerances, while open-ended estimation scenarios can relax them appropriately.
4.2 Process Metrics
Process metrics determine whether a model is “apparently correct but actually unreliable in reasoning.” You can check:
- Whether a key condition is omitted;
- Whether the wrong formula is used;
- Whether logical jumps appear in intermediate steps;
- Whether there are contradictions;
- Whether it fabricates nonexistent data or theorems;
- Whether it provides an auditable derivation path.
For proofs, process metrics are especially important. Even if the final conclusion of an answer is correct, it should not be considered high-quality output if the reasoning chain contains serious errors.
4.3 Stability Metrics
Large-model outputs have a degree of randomness. Running the same problem multiple times may yield different answers. Stability evaluation recommends multiple sampling runs for each problem.
A common approach:
Run each problem n times;Measure pass rate, average score, and error-type distribution;Compute standard deviation;Identify high-variance problems.
If a model has a high average accuracy but its results fluctuate heavily across runs, it is not suitable for high-reliability business scenarios.
4.4 Engineering Metrics
Math capability ultimately has to enter business systems, so engineering metrics must be considered:
- Average response latency;
- Long-text generation cost;
- Reasoning-step length;
- Failure-retry rate;
- Manual-review ratio;
- System throughput;
- Log traceability;
- Version-rollback capability.
Complex math tasks often require longer reasoning chains, but if an overly long reasoning chain makes cost and latency uncontrollable, it is also unsuitable for large-scale use. Model selection should therefore strike a balance between the “capability ceiling” and “engineering affordability.”
5. How to Evaluate Open-Ended Math Problems
Comparisons in the original context often focus on “who is stronger at complex math problems.” But in real deployment, open-ended math problems cannot be judged simply as right or wrong.
Open-ended math problems may include:
- Modeling problems with no unique answer;
- Problems that require multiple assumptions and discussion of different cases;
- Problems that require explaining an approach without a fully formalized proof;
- Problems that require reasonable estimation grounded in a business context;
- Problems that require pointing out insufficient conditions or ambiguity.
For such problems, a multi-dimensional rubric is recommended.
Open-Ended Scoring Dimensions
| Dimension | Description |
|---|---|
| Condition identification | Whether it identifies known conditions, unknown variables, and implicit assumptions |
| Model selection | Whether it chooses a reasonable method, e.g., statistics, geometry, optimization, or calculus |
| Derivation process | Whether the steps are coherent and reviewable |
| Reasonableness of result | Whether magnitude, units, and range are reasonable |
| Stated limitations | Whether it explains applicable conditions, error, and risk |
| Quality of expression | Whether it is clear, well-structured, and easy for business staff to understand |
A 1-to-5 scale can be used:
1 point: Wrong direction or unusable;2 points: Partial approach, but key steps missing;3 points: Basically correct approach, but incomplete process;4 points: Complete approach with minor flaws;5 points: Reasonable modeling, clear derivation, usable conclusion, and sufficient explanation of limitations.
For open-ended problems, automatic scoring can serve only as a first-pass filter; manual sampling review is still required—especially for problems involving business decisions, risk assessment, or compliance interpretation.
6. Evaluation Execution Flow: From Sample Preparation to Report Output
A reusable evaluation process typically includes the following steps.
6.1 Establish a Baseline
First, select a set of baseline problems and baseline prompt templates as a reference for comparing later versions. The baseline should not change frequently, otherwise it becomes impossible to judge whether a model’s capability has improved or regressed.
The baseline set can include:
- High-frequency business problems;
- Boundary problems that are easy to get wrong;
- Long-chain reasoning problems;
- Problems with distractor conditions;
- Open-ended modeling problems;
- Safety and compliance problems.
6.2 Standardize Prompt Templates
Prompt templates affect model performance. During evaluation, the prompting approach should be fixed to avoid different versions using different prompts, which would make results incomparable.
A prompt template can include:
- Role description;
- Output-format requirements;
- Whether to show derivation steps;
- Whether assumptions are allowed;
- Numerical-precision requirements;
- Unit requirements;
- How to handle insufficient conditions.
For example, the model can be asked to:
Please analyze step by step and output the final result.If conditions are insufficient, point out the missing conditions; do not force a definitive conclusion.Round numerical results to the specified precision.
6.3 Batch Runs and Multiple Sampling
Run each problem multiple times to observe stability. Pseudocode:
function evaluate_dataset(dataset, model_caller, judge, repeats):report = []for item in dataset:outputs = []for i from 1 to repeats:response = model_caller(item.prompt)outputs.append(response)scores = []for response in outputs:score = judge(item, response)scores.append(score)report.append({"id": item.id,"pass_rate": calculate_pass_rate(scores),"average_score": calculate_average(scores),"stability": calculate_stability(scores),"errors": classify_errors(item, outputs)})return summarize(report)
6.4 Combining Automatic Scoring with Manual Review
Automatic scoring is suitable for:
- Numerical calculations;
- Expression results;
- Multiple-choice questions;
- Proofs with a clearly defined format;
- Structured output fields.
Manual review is suitable for:
- Open-ended modeling;
- Complex proofs;
- Semantically ambiguous problems;
- Problems where automatic scoring fails;
- High-scoring but suspicious cases;
- Low-scoring cases that may contain valuable innovative solutions.
A mechanism of “automatic first-pass filter + manual sampling review + focused re-check” is recommended.
6.5 Error Attribution
An evaluation report should not give only an overall score; it should also analyze error types. Common error types include:
| Error Type | Typical Manifestation |
|---|---|
| Conceptual error | Misunderstanding definitions, formulas, or theorems |
| Computational error | Arithmetic or algebraic-transformation mistakes |
| Logical error | Broken reasoning chain, circular argument |
| Condition omission | Failure to handle boundaries, domain, or applicable range |
| Format error | Output structure not meeting requirements |
| Hallucination error | Fabricating conditions, data, or theorems |
| Strategy error | Poor method choice, redundant process |
| Insufficient stability | Large variance in results across multiple runs of the same problem |
Through error attribution, you can determine whether a problem stems from the model’s capability, the prompt design, or the evaluation set itself.
7. Engineering Implementation: Building a Sustainably Operable Evaluation Pipeline
Once evaluation evolves from a one-off experiment into a long-term mechanism, it requires engineering development.
A typical evaluation pipeline can include the following modules:
Sample management — Store problems, answers, rubrics, versions, tags, difficulty, and business scenarios.
Task scheduling — Trigger evaluation tasks by version, model strategy, prompt template, and sampling count.
Batch execution — Run evaluation tasks in a containerized environment to ensure consistency and reproducible results.
Result storage — Save raw outputs, scoring results, error classifications, elapsed time, cost, and other information.
Metrics dashboard — Display accuracy, stability, latency, cost, error distribution, and version trends.
Alerting mechanism — Trigger alerts when key metrics drop, failure rates rise, or cost becomes abnormal.
Regression mechanism — Automatically run regression tests after each update to model strategy, prompt template, or evaluation set.
When building such capabilities on Baidu AI Cloud, evaluation samples, run logs, result metrics, and task configurations can be placed respectively in object storage, a container platform, log services, and monitoring/alerting components. This forms a traceable, replayable, and comparable evaluation system—ensuring a standardized evaluation process while supporting ongoing optimization.
8. Model Selection Recommendations: Aim for “Fit,” Not “Strongest”
In real business contexts, there is no absolute math capability divorced from scenario. Selection should revolve around business goals.
8.1 High-Reliability Computation Scenarios
If the business demands strict correctness—such as financial reconciliation, engineering-parameter calculation, or compliance verification—prioritize:
- High accuracy;
- Low variance;
- Auditable derivation;
- Strict formatting;
- Automatic failure mechanisms;
- Manual-review workflows.
Here, you should not focus only on “whether it can answer correctly the first time”; you should also focus on whether it remains stable across multiple runs.
8.2 Long-Chain Reasoning Scenarios
For complex proofs, research derivations, and complex modeling tasks, the model needs strong multi-step reasoning ability. But such tasks tend to produce longer outputs and higher latency. You therefore need to set:
- A maximum reasoning length;
- Timeout strategies;
- Segmented verification;
- Intermediate-step checks;
- Manual confirmation at key nodes.
8.3 High-Frequency Lightweight Scenarios
If the business scenario is only simple statistical interpretation, formula hints, or draft generation, there is no need to over-pursue extremely long reasoning chains. A more suitable strategy is:
- Control output length;
- Improve response speed;
- Reduce invocation cost;
- Set clear formatting;
- Perform secondary verification on high-risk results.
8.4 Open-Ended Consulting Scenarios
For open-ended math problems, the model’s value is not only to give an answer but also to:
- Break down the problem;
- Propose assumptions;
- Explain limitations;
- Provide multiple solutions;
- Help users understand the process.
Such scenarios are better served by a “model assistance + human decision” pattern.
9. Common Pitfalls and Improvement Suggestions
Pitfall 1: Looking Only at the Final Answer
A correct final answer does not mean a correct process. For complex math problems especially, a model may obtain the right answer through guessing, formatting coincidence, or memorization. Step-level checks must be combined.
Pitfall 2: Evaluating With Only a Few Problems
A small number of problems is easily influenced by chance. It is recommended to cover multiple difficulty levels, multiple problem types, and multiple business scenarios.
Pitfall 3: Ignoring Prompt Differences
Changes in prompts significantly affect output. During evaluation, fix the template, or record different prompt templates as separate control experiments.
Pitfall 4: Not Distinguishing Capability From Cost
Longer reasoning chains may improve accuracy but also raise cost and latency. Engineering must evaluate both effectiveness and resource consumption simultaneously.
Pitfall 5: Lacking a Regression Mechanism
After changes to model strategy, prompt templates, knowledge bases, or business rules, the absence of regression evaluation makes it easy for performance to degrade without timely detection.
Pitfall 6: Treating Automatic Scoring as the Final Verdict
Automatic scoring suits standardized problems, but open-ended problems, proofs, and business-explanation problems still require manual sampling review. Low-confidence outputs and high-risk outputs, in particular, should retain a human backstop.
10. A Recommended Long-Term Evaluation Mechanism
To keep the evaluation system continuously delivering value, it is recommended to form the following closed loop:
Collect real failure cases — Feed online errors, user feedback, and manually corrected results back into the evaluation set.
Build a golden sample set — Select stable, representative problems as a long-term regression baseline.
Regularly update perturbation samples — Test model robustness by changing values, variable names, and narrative style.
Establish a version-comparison mechanism — After each update to model strategy or system rules, compare against the previous version.
Set threshold alerts — Trigger investigation when accuracy, stability, latency, or cost exceeds thresholds.
Formulate manual-review standards — Clarify which problems must be reviewed manually and which results can pass automatically.
Accumulate business knowledge — Organize domain terminology, formula conventions, output formats, and common errors into reusable assets.
11. Conclusion
Evaluating a large model’s math capability should not stop at simple comparisons or a single experience. Businesses need a systematic, reproducible, and extensible evaluation framework that covers problem design, difficulty tiers, process scoring, stability testing, open-ended rubrics, engineering cost, online regression, and risk governance.
Truly effective math-capability evaluation is not about pursuing an abstract “ranking”; it is about answering business questions: Can the model reliably complete math tasks in a specified scenario? Is its reasoning process trustworthy? Are errors controllable? Is cost acceptable? Can it be continuously monitored and improved?
With a sound evaluation system, technical teams can more accurately identify a model’s strengths and boundaries, avoid being misled by incidental cases, and choose the right technical path in real business contexts—turning math capability from a “demo effect” into something “engineering-ready.”
