LLM Mathematical Reasoning Evaluation and Model Selection: A Reproducible Task-Mapped Approach

    For enterprises and engineering teams, the key question is not which model tops a leaderboard, but how to evaluate mathematical reasoning in a reproducible, statistically interpretable, and explainable way and map it to real business tasks. This article presents a layered evaluation framework covering basic arithmetic and unit conversion, standard algebraic and calculus derivations, competition-level multi-step reasoning, and research-level open problems. It discusses correctness and process reliability, contamination-resistant benchmark design, expert review for open problems, scoring and reporting workflows, cost and latency trade-offs, and a selection approach based on tiered gates plus weighted scoring.

    In technical selection discussions, claims that a model is stronger at math often come from anecdotal comparisons on a few problems. For enterprises and R&D teams, the valuable question is not who ranks first, but how to evaluate LLM mathematical reasoning with reproducible, statistically analyzable, and explainable methods and map it to specific business tasks. Mathematical ability is not a single dimension. It includes basic arithmetic, symbolic derivation, competition problem solving, proof generation, open-problem exploration, tool use, and result verification. Without a layered evaluation system, it is easy to mistake natural expression or a longer chain of thought for stronger reasoning.

    1. From Leaderboard Thinking to Task Mapping

    The first step in evaluation is to define task boundaries. Different business scenarios have very different requirements for mathematical ability: educational scenarios emphasize readable steps and error localization; financial risk control emphasizes numerical stability and auditability; research assistance emphasizes hypothesis management and proof checking; engineering computation emphasizes tool use and unit consistency. When selecting a model, the question should not be simply whether the model can solve hard problems. It should be whether, on the target task distribution, accuracy, process reliability, and cost meet the constraints. A practical approach is to divide tasks into four layers: L1 basic arithmetic and unit conversion, L2 standard derivations such as algebra and calculus, L3 competition-level multi-step reasoning, and L4 research-level open problems. The first three layers can use automated scoring with sampled human review, while L4 must involve expert review and formal or semi-formal verification.

    2. Correctness and Process Reliability

    For mathematical tasks, the final answer matters, but the process matters as well. In multi-step derivations, a model may arrive at a correct answer by accident because of an incorrect intermediate assumption, or it may be misjudged because its expression is nonstandard. Evaluation should record final-answer accuracy, step-level correctness, whether key assumptions are explicit, whether errors can be localized, whether units and boundary conditions are checked, and whether the model proactively states uncertainty. Process scoring can use a rubric such as: 0 points: the conclusion is wrong and the process is unusable; 1 point: the conclusion is correct but the process is missing; 2 points: the process is basically correct but contains non-critical errors; 3 points: the process is complete and reviewable. For proof tasks, evaluation should check definition references, variable ranges, induction steps, and counterexample handling. Chain-of-thought length is not a capability metric. Overly long reasoning chains can increase error propagation and cost, so concise and verifiable derivations should be preferred.

    3. Benchmark Design and Contamination Control

    Public benchmarks are susceptible to training-data leakage, so evaluation sets should include private problems, paraphrased problems, and dynamically generated problems. Private problems are written by domain experts with held-out answers. Paraphrased problems change values, conditions, or wording to test generalization. Dynamically generated problems use parameterized templates to reduce memorization risk. During evaluation, the prompt template, sampling temperature, maximum output length, and retry strategy should be fixed, and versions should be recorded. For the same model, sample at least multiple times and report the mean, variance, and confidence interval rather than drawing conclusions from a single run. For open-ended problems, include options such as “undecidable” and “needs additional conditions” to encourage the model to recognize incompleteness in the problem itself.

    4. Evaluating Open Mathematical Problems

    Research-level open problems often have no standard answer and may even require long-term verification. The evaluation focus should shift from whether an answer is given to whether the model proposes a testable path forward: whether it formalizes the problem clearly, lists known results and assumptions, provides verifiable intermediate propositions, identifies potential counterexamples, and distinguishes conjectures from proofs. Double-blind expert review can be used, with at least two reviewers scoring independently and a third reviewer introduced when disagreement is significant. For parts that can be formalized, a general-purpose formal proof checker can be connected for mechanical verification. For parts that cannot be formalized, human review records should be retained. Special care should be taken to guard against “plausible-looking long proofs”: reviewers should check whether each step follows from the preceding text, whether definitions are quietly changed, and whether boundary conditions are ignored.

    5. Scoring Statistics and Experimental Workflow

    The evaluation workflow should be engineered: define the task set, run sampling, perform automated scoring, conduct manual spot checks, and produce statistical reports. The following pseudocode is an example:

    1. function evaluate(model, tasks, k, rubric):
    2. records = []
    3. for task in tasks:
    4. samples = []
    5. for i in 1..k:
    6. output = model.generate(task.prompt, temperature=task.temp, max_tokens=task.max_tokens)
    7. samples.append(output)
    8. auto_scores = [auto_judge(task, s) for s in samples]
    9. records.append({
    10. task_id: task.id,
    11. layer: task.layer,
    12. samples: samples,
    13. auto_scores: auto_scores,
    14. pass_at_k: any(auto_scores),
    15. mean_score: average(auto_scores)
    16. })
    17. manual_sample = stratified_sample(records, ratio=0.1)
    18. manual_scores = [human_review(r, rubric) for r in manual_sample]
    19. return aggregate(records, manual_scores)

    The statistical report should include at least: pass rate by layer, pass@k, average score, variance, agreement between manual and automated scoring, average output length, number of tool calls, cost per task, and latency. For critical business tasks, maintain a regression set and rerun it after every model or prompt change to detect degradation.

    6. Cost, Latency, and Maintainability

    Mathematical reasoning often requires longer outputs and multiple samples. During model selection, calculate “effective accuracy / cost” and “effective accuracy / latency”. If multiple sampling plus voting is used to improve accuracy, the sampling cost should be included. For steps that can be tool-assisted, prioritize letting the model call calculation, symbolic derivation, or proof-checking capabilities, and evaluate the reliability of the orchestration. Output readability, format stability, and error-message quality are also maintainability metrics: structured output is easier to audit, and explicit uncertainty statements make human takeover easier.

    7. Selection Recommendations

    A practical approach is “tiered gates plus weighted scoring”: first set minimum accuracy gates for L1/L2, then score L3/L4 according to business weights; set expert-review gates for open problems; and set upper limits for cost and latency. Do not replace business validation with a single leaderboard, and do not equate natural expression directly with reliable reasoning. A more robust approach is to build a small, private, continuously updated evaluation set, combine automated scoring with human review, and produce a traceable selection report. An AI system is part of an engineering system, and mathematical capability evaluation should be layered, measurable, and reviewable in the same way as engineering acceptance, ultimately serving the stability and maintainability of real tasks.