How to Evaluate Mathematical Reasoning in Large Language Models: Evaluation Design and Practice

    Evaluating mathematical reasoning in large language models requires more than checking whether the final answer is correct. This article presents a practical framework for designing open-ended evaluation tasks, distinguishing memorization from genuine reasoning, applying multidimensional scoring, conducting expert review, interpreting differences between models, and building a reproducible evaluation pipeline. It also discusses collaboration between models and human experts, hallucination risks, variation testing, and long-term regression tracking.

    Introduction

    The potential of large language models (LLMs) for mathematical reasoning has become a major focus in artificial intelligence research. Unlike simple question answering, mathematical reasoning requires several capabilities at once, including symbolic understanding, logical deduction, and rigorous communication. The most important—and most difficult—challenge in the evaluation process is determining whether a model can genuinely reason rather than reproduce answers encountered in its training data.

    In many user scenarios, a model can fluently solve textbook exercises or standard competition problems but performs much worse when faced with open-ended questions that require constructing an argument. This reveals a substantial gap between “getting the answer right” and “being able to reason.” A reliable evaluation system must therefore address task design, scoring methods, and result interpretation together.

    1. Four Principles for Designing Evaluation Tasks

    1. Avoid Standard Answers

    If evaluation problems are taken directly from public textbooks, problem banks, or research papers, a model may have encountered similar content during training. Its response may then reflect retrieval or pattern matching rather than reasoning performed at evaluation time.

    One effective approach is to select propositions from large mathematical knowledge bases that are listed without accompanying proofs. These propositions are often recorded as facts to be proven, while the source text does not provide the reasoning process. As a result, the model cannot simply reproduce a proof through text matching.

    For example, an evaluation task can select a proposition that appears in a knowledge base without a proof and ask the model to prove it while producing a complete English lecture note in LaTeX. This type of task is naturally open-ended. The model must organize the assumptions, definitions, lemmas, and reasoning steps on its own. Missing steps or unjustified transitions are likely to become visible in the final document.

    2. Use Sufficiently Open-Ended Tasks

    Multiple-choice and fill-in-the-blank questions primarily evaluate the final result. They do not adequately test whether the reasoning path is valid.

    Open-ended proof tasks require the model to provide intermediate steps and may allow it to select its own proof strategy, such as contradiction, induction, or a constructive proof. This provides a more realistic view of how deeply the model understands the underlying mathematical structure.

    3. Require Structured Output

    Asking a model to produce lecture notes in LaTeX is not only a readability requirement. It also tests the model’s ability to express mathematical ideas symbolically.

    The rigor of mathematical reasoning is often reflected in formula definitions, theorem statements, environment usage, and cross-references. Frequent errors in LaTeX syntax, nested mathematical environments, or internal references may indicate that the model’s symbolic representation is not sufficiently stable.

    4. Make the Reasoning Process Observable

    In mathematical tasks, models often generate an internal reasoning process. Evaluators may ask the model to provide an explicit reasoning summary or inspect the reasoning trace through an interactive interface, where such access is available.

    This makes it possible to determine whether the model took an inefficient path and to compare reasoning styles across models. Some models produce concise reasoning and quickly identify the key idea. Others explore many cases and perform repeated checks. Although both may reach the correct conclusion, their efficiency and readability can differ substantially.

    2. Core Evaluation Workflow

    Step 1: Define the Capability Boundary

    Before constructing a benchmark, clarify which capabilities the evaluation is intended to measure.

    Mathematical reasoning can be divided into areas such as algebra, number theory, geometry, topology, and analysis. The difficulty varies considerably across these fields. The test set should therefore be stratified by domain and include multiple difficulty levels, ranging from foundational to advanced problems.

    If the benchmark contains only difficult problems, too few models may be able to make meaningful progress, reducing its ability to distinguish performance. If it contains only basic problems, most models may receive high scores, which also reduces its evaluation value.

    Step 2: Construct the Test Set

    Select several propositions without proofs from a mathematical knowledge base and remove information that could reveal the answer. This may include deleting suggestive passages while retaining the definitions and notation required to understand the task.

    Each proposition should have an independent task card that records its source, mathematical domain, expected difficulty, and potentially relevant tools or concepts. The test set should not be too small. In general, it should contain at least 20 to 30 propositions to support more stable conclusions.

    Step 3: Establish Multidimensional Scoring Criteria

    Scoring should not focus only on whether the final conclusion is correct. It should cover at least the following five dimensions:

    • Correctness: Whether the conclusion is valid and the reasoning contains no fatal errors.
    • Completeness: Whether important definitions, assumptions, or auxiliary lemmas are missing.
    • Rigor: Whether the proof introduces unstated assumptions, makes unjustified jumps, or confuses related concepts.
    • Readability: Whether the LaTeX formatting is consistent, the structure is clear, and the document follows mathematical lecture-note conventions.
    • Efficiency: Whether the reasoning is of a reasonable length and complexity and avoids unnecessary repetition.

    Different dimensions may receive different weights. For a proof task, correctness should carry the greatest weight. For a lecture-note generation task, completeness and readability may be equally important. After the weights are defined, they should be calibrated through scoring by human experts.

    Step 4: Conduct Expert Review

    Automated scoring cannot fully determine whether a mathematical derivation is correct. Domain experts must therefore review the model outputs.

    A double-blind process can be used so that reviewers do not know which model produced a given answer. They score the proof based only on the evidence contained in the reasoning text. Reviewers should also record common error types, such as:

    • Incorrect case analysis
    • Confusion between necessary and sufficient conditions
    • Missing boundary cases
    • Unjustified assumptions
    • Invalid use of a theorem or lemma

    These records can provide useful guidance for future model improvement.

    Step 5: Present and Interpret the Results

    Evaluation results should not be reduced to a single overall score. A radar chart can show a model’s relative performance across multiple dimensions, while pass rates for individual propositions can reveal more specific differences.

    For example, a model may achieve a much higher pass rate on number-theory propositions than on topology propositions. This domain-level difference provides more useful information about the model’s capability boundary than an aggregate score alone.

    3. Interpreting Differences Between Models

    Reasoning Style and Depth

    On the same test set, models may differ not only in whether their conclusions are correct, but also in how they reason.

    Some models produce concise and controlled reasoning, quickly identify the key lemma, and generate proofs that resemble the working notes of a human expert. Others produce very long reasoning traces, repeatedly exploring alternative paths. They may eventually reach the same conclusion, but their process is more complicated and difficult to read.

    Evaluators should distinguish verbosity from depth. If every step in a long reasoning trace contributes meaningful progress, the length may reflect deeper reasoning. If the trace merely repeats the same verification or revisits an identical approach, it is primarily an efficiency problem.

    Distinguishing Memorization from Reasoning

    Even when a model successfully proves a proposition without a provided proof, evaluators should not immediately conclude that it possesses genuine reasoning ability.

    Variation testing is also necessary. Change one parameter, structural element, or constraint in the original proposition and observe whether the model can adapt. If the model fails quickly on the variation, its original success may have depended on matching the source text or a familiar pattern. If it can still construct a reasonable proof, the result is stronger evidence of general reasoning ability.

    Changes Across Model Versions

    Model updates can produce fluctuations in mathematical performance. A new version may score better on standard problems while performing worse on open-ended tasks. This kind of trade-off is not unusual during model development.

    An evaluation framework should therefore support long-term tracking. After each model update, the same test set should be used for regression testing whenever possible. This makes it easier to identify genuine changes in capability rather than changes caused by a different benchmark.

    4. Practical Collaboration Between Models and Human Experts

    Use a Staged Collaboration Process

    In mathematical reasoning, an LLM should not be treated simply as a replacement for an expert. It is more appropriately used as an interactive mathematical assistant.

    A practical approach is to use two stages:

    1. The model generates an initial proof and identifies the lemmas and assumptions on which it relies.
    2. A human expert reviews, completes, and revises the draft.

    Under this arrangement, the model explores possible approaches and produces formal text, while the human expert evaluates the mathematical meaning of the approach and its final rigor.

    Guard Against Hallucinations and Overconfidence

    The greatest risk in mathematical reasoning is not that a model cannot solve a problem, but that it may appear to have solved one when it has not.

    A model may generate a well-structured lecture note containing a hidden error in the middle of the derivation. Non-expert users may have difficulty detecting such errors. During both evaluation and real-world use, prompts should therefore instruct the model to state clearly when it cannot prove a claim or is uncertain.

    An evidence-reporting mechanism can also be used, requiring the model to identify the basis for each critical step. In high-risk scenarios, external verification tools or symbolic computation systems should be used to support the review process.

    Treat Explanations as an Evaluation Metric

    A model that produces the correct conclusion but cannot explain why it is correct has limited value in research and education.

    Evaluators should additionally ask the model to summarize the overall proof strategy in a short paragraph. This tests whether the model can clearly describe the structure of its argument and may reveal whether it understands the mathematical idea rather than merely generating locally plausible steps.

    5. Engineering the Evaluation Environment

    To make the evaluation process reproducible and scalable, it is useful to package it as an automated pipeline. The following pseudocode illustrates a general evaluation workflow:

    1. def evaluate_math_reasoning(model, propositions, output_dir):
    2. results = []
    3. for prop in propositions:
    4. prompt = (
    5. f"Here is a proposition from a mathematical knowledge base:\n{prop.statement}\n"
    6. "The original source does not provide a proof. Construct a complete and rigorous proof "
    7. "and produce an English lecture note in LaTeX. Also show your reasoning process."
    8. )
    9. response = model.generate(prompt)
    10. # Automatically extract the LaTeX code and attempt to compile it
    11. latex_status = compile_latex(response.latex_body)
    12. # Send the remaining text to an expert for scoring
    13. expert_score = expert_reviewer.score(response)
    14. results.append({
    15. "proposition_id": prop.id,
    16. "latex_status": latex_status,
    17. "expert_score": expert_score,
    18. "chain_length": response.chain_length,
    19. })
    20. return aggregate_and_report(results)

    During deployment, the model service, evaluation tasks, and expert-scoring interface can be connected through a workflow platform. For example, Baidu AI Cloud can be used with general-purpose data storage and messaging components to build a similar evaluation pipeline for task distribution, result collection, and report generation.

    The workflow does not need to depend on a specific third-party tool. It can be customized according to the team’s requirements.

    6. Conclusion

    Evaluating mathematical reasoning in large language models is an evolving technical challenge. A good evaluation framework should not only answer the question, “Which model is smarter?” It should also explain which model is more dependable for a particular type of mathematical task.

    Open-ended evaluation based on propositions without provided proofs can help distinguish memorization from reasoning. A multidimensional scoring system can capture subtle differences across correctness, completeness, rigor, readability, and efficiency. An engineering-oriented pipeline makes the evaluation process reproducible and traceable.

    Current mainstream models already show clear differences in their ability to handle open mathematical problems. Some can consistently address advanced proof tasks, while others remain strongest on standard problem formats. As models continue to evolve rapidly, evaluators must regularly update test sets and recalibrate scoring criteria.

    Ultimately, the purpose of evaluation is not merely to produce a leaderboard. It is to help people identify the most appropriate form of intelligent collaboration for mathematical research, education, and engineering practice.