Evaluating Mathematical Reasoning in Large Language Models: A Practical Selection Guide

    This guide explains how to evaluate the mathematical reasoning capabilities of large language models beyond simple single-question accuracy. It covers six evaluation dimensions, layered benchmark design, combined scoring for answers and reasoning processes, tier-based model comparison, the distinction between long chain-of-thought and tool-augmented reasoning, a reproducible evaluation pipeline, and scenario-based selection advice, helping teams build auditable, continuously updated evaluation systems for informed model selection.

    Evaluating Mathematical Reasoning in Large Language Models: A Practical Selection Guide

    1. Why Math Rankings Should Not Rely on Single-Question Accuracy

    When comparing the math capabilities of large language models, a common approach is to test them on a handful of competition problems, word problems, or open questions, then rank them based on subjective impressions. This has some reference value, but it cannot support enterprise selection decisions or long-term technology planning. The reason is that mathematical reasoning is not a single skill but a combination of abilities: basic arithmetic, symbolic manipulation, multi-step derivation, constraint management, counterexample awareness, result verification, tool coordination, and explanatory clarity. Models differ in their strengths across these dimensions, and a single score can easily hide real weaknesses.

    A better approach is to break “math ability” into observable dimensions, build a layered evaluation set, and report results with weighted scores and confidence intervals. Leaderboards can serve as a starting signal, but they should not replace a reproducible evaluation pipeline. For open-ended math problems in particular, the final answer is often not a unique string, so scoring must also examine problem decomposition, the reasonableness of assumptions, proof rigor, and verifiability.

    2. Defining the Dimensions of Mathematical Reasoning

    We recommend evaluating along the following six dimensions. Each dimension should have its own questions and scoring rules rather than being collapsed into a single overall accuracy number.

    Capability Dimension Focus Typical Evaluation Method Selection Relevance
    Basic computation and symbolic manipulation Arithmetic, algebraic transformation, unit conversion, symbolic derivation Short questions with automated grading Filters out models with weak fundamentals
    Multi-step reasoning and intermediate checks Long derivation chains, constraints, case analysis Multi-step problems with process scoring Indicates usability for complex business problems
    Open-problem decomposition Problem modeling, assumption management, sub-goal breakdown Open questions with scoring rubrics Distinguishes research assistance and advanced analysis ability
    Proof rigor and counterexample awareness Logical closure, boundary conditions, counterexample construction Proof problems with human review Assesses reliability in high-risk scenarios
    Long context and tool coordination Reading long materials, calling computation tools, substituting results back Tool-augmented evaluation Indicates engineering integration potential
    Expression and interpretability Clear steps, traceable conclusions, accurate terminology Human scoring and spot checks Affects teaching, reporting, and audit scenarios

    One point deserves emphasis: writing that “sounds more human” does not mean stronger math ability. Writing style affects readability, but the core of mathematical capability remains derivation correctness, sufficient verification, and the ability to close the loop on open problems. Nor is a longer chain of thought always better: if the chain is long but lacks back-substitution checks and counterexample testing, it can actually accumulate more errors.

    3. Benchmark Design: Layered, Leak-Resistant, and Reviewable

    The evaluation set should cover four layers: basic problems, integrated problems, competition problems, and open problems. Basic problems enable fast screening; integrated problems reveal the stability of multi-step reasoning; competition problems provide differentiation; and open problems assess higher-order decomposition ability. Question sources should be as diverse as possible, and some dynamically generated or internal questions should be retained to reduce inflated scores caused by training data leakage.

    Scoring should combine a final-answer score, a process score, and a verifiability score. For closed-form questions, the final answer can be graded directly. For open questions, use a rubric such as:

    1. Does the model correctly identify the problem type and constraints?
    2. Does it provide a reasonable modeling or decomposition path?
    3. Are key steps justified, without unjustified jumps?
    4. Does it check boundary conditions, special values, and counterexamples?
    5. Can the conclusion be formally or numerically verified?
    6. When it fails, does it produce a diagnosable error explanation?

    Latency, invocation cost, failure rate, and stability should also be recorded. A model that answers correctly in a single sample is not necessarily stable across multiple samples and different temperature settings. Selection should focus on “pass rate plus variance,” not on the best single example.

    4. Scoring and Tier-Based Reporting

    Assign weights to each dimension and adjust them for the business scenario. For example, education scenarios weight step-by-step interpretability more heavily; research assistance weights open-problem decomposition; math Q&A in enterprise knowledge bases weights stability and the boundaries of appropriate refusal. Aggregation can use weighted averages with confidence intervals. For pairwise comparisons, preference win rates can be computed, but per-layer results must be preserved so that aggregate scores do not hide localized weaknesses.

    Without referencing specific product versions, results can be described as capability tiers: tier-one models can typically complete the full loop of modeling, decomposition, verification, and correction on open problems; tier-two models handle complex multi-step problems stably but still need human guidance on open problems; tier-three models perform well on basic problems but lose points on complex derivations and boundary checks. Tier descriptions are more robust than simple rankings and better suited to technical decision-making.

    5. Long Chain-of-Thought vs. Tool Coordination

    The value of a long chain of thought lies in exposing intermediate steps for inspection and correction; it is not a capability in itself. Evaluation should focus on whether the model proactively substitutes results back for verification at key junctures, constructs counterexamples, switches methods, and performs consistency checks on tool outputs. For computable subproblems, calling a general-purpose computation tool and verifying its result is usually more reliable than simply extending the reasoning chain.

    From an engineering standpoint, “standalone model answering” and “tool-augmented answering” must also be distinguished. The two should not be mixed in the same leaderboard. Tool-augmented evaluations should record the number of tool calls, failure recovery behavior, and final verification, so that gains from tools are not mistakenly attributed to the model’s native capability.

    6. A Reproducible Evaluation Pipeline

    The following neutral pseudocode illustrates how to organize batch evaluation and result aggregation:

    1. Input: layered question bank Q, candidate model set M, scoring rules R, sampling configuration C
    2. result_table = empty
    3. for q in Q:
    4. for m in M:
    5. sample = m.generate(q.prompt, config=C)
    6. parsed = extract final answer and reasoning process from sample
    7. score = R(parsed, q.reference_answer, q.rubric, q.dimension)
    8. record(q.id, m.id, score, latency, invocation_cost, failure_reason, sample_count)
    9. Aggregate by dimension:
    10. compute weighted mean, pass rate, variance, and confidence intervals
    11. output human spot-check conclusions for open questions
    12. flag models strong on basic problems but weak on open problems
    13. flag models with high variance or high failure rates
    14. Output:
    15. capability-dimension radar data, tier recommendations, risk notes, and a re-evaluation plan

    The key to this pipeline is not code complexity but traceability: every question, every sample, and every scoring decision should be reviewable. Enterprises should retain question bank versions, prompt templates, scoring rules, and spot-check records so that regression comparisons can be made after model upgrades.

    7. Scenario-Based Selection Advice

    Start by clarifying which category your business problem falls into. If the workload is mainly basic computation and standard Q&A, prioritize stability, latency, and cost. If it involves complex engineering calculations, increase the weight of multi-step reasoning and tool coordination. If it supports research or open-problem exploration, focus on problem decomposition, assumption management, and the verification loop. For high-risk conclusions, human review and external verification are mandatory; model output should not be treated directly as a final proof.

    At the deployment level, a general-purpose evaluation and runtime environment can be built on common cloud services such as object storage, container platforms, log services, and monitoring/alerting—for example, the corresponding products on Baidu AI Cloud. The question bank and results go into object storage, evaluation jobs run on the container platform, process logs are collected centrally, and anomalies or latency spikes surface through monitoring alerts. This turns model evaluation from a one-off experience into a continuous regression process.

    8. Common Pitfalls and Conclusion

    Common pitfalls include: using a small set of questions to represent overall capability; equating long chains of thought with strong reasoning; mistaking writing style for mathematical rigor; ignoring sampling variance and cost; and converting subjective praise on open problems directly into leaderboard conclusions. More reliable conclusions come from layered evaluation, transparently weighted scoring, human spot checks, and continuous regression.

    In summary, evaluating mathematical reasoning in large language models should center on capability dimensions, layered question banks, process scoring, verification mechanisms, and scenario-specific weights. Leaderboards can point in a direction, but the truly usable basis for selection is a reproducible, auditable, continuously updatable evaluation system.