LLM Mathematical Reasoning Evaluation: Verifiable Test Methods and a Model Selection Checklist

    This article explains how to evaluate the mathematical reasoning of large language models with verifiable methods rather than relying on whether LaTeX compiles. It covers test-task design, a structured prompt template, a scoring rubric with proof correctness as a hard gate, a review workflow, evidence requirements for capability tiers, cost measurement per correct proof, a selection checklist, and key risks and limitations.

    Why LaTeX Compilation Alone Is Not Enough

    A common way to evaluate a model’s mathematical ability is to check whether the generated write-up compiles. This only shows that the formatting is valid; it does not show that the proof is correct. A perfectly formatted document may contain incorrect reasoning steps or cite theorems that do not exist. Evaluation must therefore put mathematical correctness first and treat formatting as a secondary criterion.

    Another common problem is mistaking “producing complete text” for “completing a proof.” A model may skip justification at a key step, hide the gap with “obviously holds,” or make a plausible-sounding assertion. Reviewers need to check the proof step by step to find such issues.

    1. Designing Test Tasks

    Test tasks should meet three conditions: the proposition has a clear mathematical definition, reviewers can determine whether the answer is correct, and the problem is not directly covered by the model’s training data.

    There are three common sources of tasks. The first is lemmas in open-source mathematical projects that have not yet been proved; the definitions and context in the project can serve as background input. The second is rewritten versions of textbook exercises, where the conditions are changed so the answer cannot be looked up directly from the original text. The third is special cases or generalizations of known results, which require the model to organize the derivation itself.

    Each task category should contain no fewer than 30 samples, selected by stratified sampling according to difficulty. For each problem, record the following information: task source, full proposition, background material file and line numbers, and the reference answer or key steps used by reviewers.

    2. Prompt Template

    The more structured the prompt, the easier it is to compare model performance. The following is a basic template; replace the content in square brackets according to the task:

    1. You are a mathematical research assistant. Prove the following proposition.
    2. [Background]
    3. [Paste relevant definitions and known results, with source file and line numbers]
    4. [Proposition]
    5. [Write the full statement, including the domains of the variables]
    6. [Requirements]
    7. 1. First list all known results used in the proof, and mark which come from the background material and which are standard results you cite.
    8. 2. Give the proof and state the justification for each step.
    9. 3. If you cannot complete the proof, clearly identify the step where you are stuck. Do not skip steps or claim that the proof is complete.
    10. 4. Output an English LaTeX document using the amsthm environment, with theorems, lemmas, and proofs numbered separately.
    11. 5. Do not output any explanatory text outside the document.

    Requirement 3 is important. It allows the model to admit failure, making it possible to distinguish between “incorrect proof” and “honestly incomplete.” If the template does not allow failure to be acknowledged, the model is more likely to produce a plausible but incorrect proof that appears complete.

    3. Scoring Metrics and Review Process

    The table below lists six metrics. Proof correctness is a hard gate: if this item is judged incorrect, the entire task receives 0 points, and the remaining metrics are not counted.

    Metric Definition Scoring method Weight
    Proof correctness The conclusion and reasoning are mathematically valid Two reviewers judge independently; pass = 1, otherwise 0 40%
    Step completeness No skipped steps; each step has a justification 0 to 2 points 20%
    Reviewability Whether reviewers can check the proof step by step within a reasonable time Record review time per task 15%
    Citation accuracy Cited theorems exist and are stated correctly Count the proportion of incorrect citations 10%
    Compilation and formatting LaTeX compiles and environments are used correctly Automated compilation plus manual spot checks 5%
    Token efficiency Average token consumption across passing samples Calculated from call logs 10%

    The review process is as follows. First, two reviewers independently re-derive the key steps and give their judgments. Second, tasks with disagreements are adjudicated by a third reviewer, and the reasons for the disagreement are recorded. Third, for parts that can be formalized, proof assistants such as Lean or Coq can be used to assist verification. Fourth, aggregate the scores for each metric and save the raw outputs and review records for later inspection.

    When an open-ended problem has no standard answer, the definition of “complete” should be agreed upon in advance. For example, only a proof that covers all cases and justifies every step should count as complete; an output that gives only an idea without completing the proof should be recorded as “partially complete” and excluded from the accuracy rate.

    4. Evidence Requirements for Model Capability Tiers

    Common online claims about “model tiers” usually lack verifiable data: no model name or version, no evaluation problems, no sample size, and no scoring criteria. This article therefore does not provide rankings for any specific model.

    To build your own tiers, follow these steps:

    1. Fix the model version and invocation parameters, and record the version number, temperature setting, and maximum output token count.
    2. Test all candidate models with the same set of problems and the same prompt template.
    3. Run each model at least three times and report the range of variation in pass rates, so that accidental success is not mistaken for stable capability.
    4. When reporting results, include the sample size, scoring criteria, and number of reviewers.

    Only when these conditions are met can comparison conclusions be reproduced by others. Claims without this information should be treated as hypotheses awaiting verification.

    5. Cost Measurement Method

    The cost of mathematical reasoning mainly comes from output tokens. The longer the reasoning process, the more tokens are consumed. Therefore, when comparing costs, look at “cost per correct proof,” not the price of a single call.

    The formulas are as follows:

    • Cost per call = input tokens ÷ 1000 × input unit price + output tokens ÷ 1000 × output unit price
    • Cost per correct proof = average cost per call × total number of calls ÷ number of passing samples

    The following information must be recorded; none of it can be omitted: model name and version, actual input and output token counts, number of tasks, billing unit, and measurement period. If only a “quota share” can be provided without the above information, the number is not meaningful for comparison.

    Use the table below to record test results:

    Model Number of tasks Number passed Average input tokens Average output tokens Cost per call Cost per correct proof
    To be filled To be filled To be filled To be filled To be filled To be filled To be filled

    6. Selection Checklist

    When selecting a model, first determine the task type, then compare budget and verification conditions. The table below gives selection conditions, not specific products:

    Task type Budget ceiling Output format requirement Verification method Manual review cost
    Textbook-level standard proofs Low Standard LaTeX document Spot checks plus automated compilation Low
    Open or unknown proposition exploration Medium to high Full derivation and citation notes Step-by-step manual review High
    Batch document generation Limited by sample size Fixed template Automated checks plus sampled review Medium
    Publication or teaching final Quality first Complete proof and citation list Two independent reviewers High

    When filling in the selection table, use pass rates, token consumption, and review time from your own tests rather than rankings from others.

    7. Risks and Boundaries

    Model outputs must be reviewed by humans, especially when they are used in papers, courses, or external documents. Model-generated proofs may contain subtle errors, and the errors often appear in steps that look obvious.

    Pay attention to copyright when submitting problems and background materials to models. Before using code or documentation from open-source projects, confirm whether the license permits the intended use. When unpublished research content is involved, confirm internal confidentiality requirements in advance.

    When calling APIs, prompts and outputs may contain sensitive information. Remove personal data, keys, and internal addresses before submission, and redact logs before storing them.

    If a model is integrated into an enterprise system, any cloud platform that meets the requirements can be chosen. When choosing, pay attention to service-level agreements, available regions, concurrency quotas, and call-log retention policies. This article does not recommend any vendor and does not imply that any platform endorses the views expressed here.

    FAQ

    How can I verify the correctness of a mathematical proof?
    You can proceed in three steps. First, humans independently re-derive the key steps. Second, for parts that can be formalized, use tools such as Lean or Coq for verification. Finally, perform numerical checks on special values or small-scale cases. Numerical checks cannot prove a general proposition, but they can quickly reveal obvious errors.

    How should I troubleshoot an overly long chain of thought?
    First, calculate the distribution of output tokens per call and identify repeated passages. Then check whether the prompt contains contradictory constraints, such as requiring both “conciseness” and “detailed step-by-step reasoning.” If the problem is difficult, split the proposition into several sub-propositions and test them separately, set an output token limit, and observe how the pass rate changes.

    What should I pay attention to when evaluating open-ended problems?
    Reviewers should agree on the definition of “complete” in advance and avoid adjusting it after seeing the results. Distinguish between “proof completed” and “reasonable direction given but proof not completed.” Reviewers should ignore the writing style of the model output and judge only the mathematical content. In addition, record whether the model output is highly similar to conclusions in public literature, so that you can judge whether the reasoning comes from existing material.