Evaluating Mathematical Abilities of Frontier LLMs: Methods, Protocols, and Limits

    Mathematical benchmarks are increasingly used to gauge the reasoning level of large language models, but a reported pass rate is meaningful only when its evaluation protocol is clear. This article examines how sampling budgets, answer extraction and normalization, verifier design, tool permissions, and output types shape reported scores. It explains why pass@1 and pass@k are not directly comparable when attempt counts differ, distinguishes four kinds of model deliverables, reviews key limits such as benchmark contamination and verifier bias, and proposes a reproducible evaluation pipeline. The goal is to move from leaderboard comparisons toward capability attribution: what the model delivered, how it was verified, what it cost, and where the boundary lies.

    Mathematical ability evaluation is becoming an important window into the reasoning level of large language models. Yet when an evaluation report gives pass rates for several frontier models, the question readers should really ask is not simply “who ranks higher or lower,” but: under what protocol were these scores obtained? What did the model actually deliver for each problem? Was it a final answer, a complete reasoning process, a verifiable proof, or a computational result obtained through external tools? If these premises are not standardized, the comparability of scores is greatly reduced.

    One notable detail in evaluation reports is that different models may not use the same number of attempts per problem. Some models attempt each problem only once, while others attempt it multiple times. This single difference is enough to make surface-level pass rates directly incomparable. Multiple sampling significantly increases the probability of “at least one correct answer,” while single sampling is closer to the model’s stable performance in one-shot reasoning. If a report does not also specify the sampling budget, answer extraction rules, and verification method, then “what the model delivered behind the score” remains an open question.

    1. Scores Are Not Capability: Sampling Budget Determines Comparability

    In LLM mathematical evaluation, the most common metrics are single-attempt pass rate and multi-sample pass rate. The former is often denoted pass@1, meaning the probability that the model answers correctly in one generation under a fixed prompt; the latter, such as pass@k, means the success rate when the model is allowed to generate k candidate answers and a correct answer is taken from among them. These two metrics measure different capabilities: pass@1 focuses more on the stability and accuracy of one reasoning attempt, while pass@k focuses more on whether the model has the potential to “encounter a correct solution” through multiple explorations.

    If different models use different attempt counts in an evaluation but their results are placed in the same table for direct comparison, a clear budget bias arises. Multiple attempts not only increase computational cost but also change the nature of the task: the model can filter, vote on, or select from multiple candidate solutions, or a verifier can choose among them. For mathematical problems, this difference is especially pronounced, because a sign error, a missed boundary condition, or a calculation mistake can cause a single answer to fail, while multiple samples may cover such errors.

    Therefore, a rigorous evaluation should at least report the following information:

    • The number of samples per problem, k;
    • Whether sampling parameters such as temperature and top-p are used;
    • Whether multi-candidate voting or retries are allowed;
    • Answer extraction and normalization rules;
    • Verifier type and decision thresholds;
    • Whether external tool calls are allowed;
    • Confidence intervals or variance information.

    Without this information, a pass rate can only be regarded as an observation under a specific protocol, not as a direct equivalent of the model’s intrinsic mathematical ability.

    2. Why Differences in Attempt Counts Matter

    Differences in attempt counts first affect the statistical basis. Suppose a model has a single-attempt correct rate of p. Under an independent and identically distributed assumption, the probability of at least one correct answer in k attempts is approximately 1-(1-p)^k. Even if p is not high, pass@k rises quickly as k increases. Therefore, a model with a lower single-attempt pass rate but better multi-sample performance cannot be simply compared with a model that has a higher single-attempt pass rate but is tried only once.

    Second, differences in attempt counts change the distribution of error types. A single failure may come from a calculation error, a format error, a misinterpretation of the problem, or an interruption in the reasoning chain; failure after multiple attempts is more likely to indicate that the model lacks a key concept, cannot construct a proof, or cannot stably handle long-tail problems. If an evaluation reports only the overall pass rate without distinguishing failure causes, it obscures the real research boundary.

    A more reasonable approach is to report a “budget-performance curve” at the same time: the horizontal axis is the sampling budget or computational cost, the vertical axis is the pass rate, and variance under different budgets is marked. This makes it possible to observe whether a model is “highly stable at low budget” or “highly promising at high budget.” For mathematics research tasks, the latter is not necessarily more valuable, because research emphasizes reliable derivation and verifiable conclusions rather than a correct answer found by chance after blind repeated attempts.

    3. What the Model Actually Delivers: Four Types of Outputs

    When discussing mathematical ability, it is necessary to distinguish what the model actually delivers. At least four categories can be identified:

    1. Final answer: Only a numerical value, expression, or option is provided. This type of evaluation is easy to automate, but it cannot determine whether the reasoning process is reliable, and it is difficult to rule out guessing, format matching, or data contamination.
    2. Natural-language reasoning chain: A step-by-step explanation is provided. It can show the line of thought, but natural-language steps may contain arguments that look plausible but are actually wrong, and automatic verification is difficult.
    3. Verifiable proof or formalized process: The proof is converted into a form that can be checked by a verifier. This type of output is closer to the requirements of mathematical research, but construction costs are high, and current models still have clear limits in formalization and lemma selection.
    4. Tool-call trace: This includes the use of external tools such as symbolic computation, numerical computation, search, or proof checking. Tools can improve result reliability, but they also raise a “capability attribution” problem: does the score improvement come from model reasoning or from tool wrapping?

    Therefore, an evaluation report should not give only a pass rate; it should explain which type of result the task requires the model to deliver. If a problem requires only a final answer, a high score cannot directly imply that the model has proof-construction ability. If tool calls are allowed, a high score also cannot be directly equated with the model’s intrinsic reasoning ability.

    4. Key Dimensions of the Mathematical Research Boundary

    From current evaluation practice, the research boundaries of LLM mathematical ability are mainly concentrated in the following areas.

    First, benchmark contamination and data leakage. Once a mathematical problem is public, it may enter training corpora. High scores on common problem types may partly come from memorization rather than reasoning. Evaluations should introduce temporal splits, private test sets, variant problems, and adversarial rewrites to reduce the impact of contamination.

    Second, verifier bias. Automatic verifiers usually rely on exact matching, symbolic equivalence, or numerical approximation. Exact matching may misjudge equivalent expressions, numerical approximation may let incorrect proofs pass, and symbolic verification may be limited by computational resources. The capability boundary of the verifier itself directly affects evaluation conclusions.

    Third, unverifiable process. The readability of a natural-language reasoning chain does not equal correctness. A model may give a fluent proof that contains logical gaps. To evaluate process quality, step-level annotation, manual spot checks, or formal checks are needed, but these methods are costly.

    Fourth, tool dependence and capability attribution. When computation tools and proof checkers are allowed, models can complete more complex tasks, but evaluations must distinguish “the model proposes the approach” from “the tool performs the computation.” Otherwise, scores will mix model capability with engineering wrapping capability.

    Fifth, long-tail innovation and open problems. Current evaluations mostly focus on competition problems and textbook problems with standard answers, while mathematical research depends more on conjecture generation, counterexample construction, proof strategy selection, and literature understanding. These abilities are difficult to measure with single-attempt pass rates, and they are among the clearest current limits of evaluation.

    5. Engineering Recommendations for a Reproducible Evaluation Framework

    To make evaluation closer to real capability, a unified-protocol evaluation pipeline can be built. The pseudocode is as follows:

    1. for model in models:
    2. for problem in benchmark:
    3. candidates = sample(model, problem, budget=k, temperature=t)
    4. normalized = normalize_answers(candidates)
    5. judged = []
    6. for answer in normalized:
    7. result = verify(problem, answer, verifier_set)
    8. judged.append(result)
    9. record(
    10. model=model,
    11. problem=problem,
    12. budget=k,
    13. pass_at_1=judged[0].correct,
    14. pass_at_k=any(r.correct for r in judged),
    15. failure_types=classify_failures(judged),
    16. tool_calls=extract_tool_trace(candidates)
    17. )

    In this framework, the key is to unify the sampling budget, unify answer normalization, combine verifiers, and record failure types. The verifier can use a multi-level strategy: first perform format and numerical checks, then symbolic equivalence checks, and when necessary introduce formal checks or manual review. When reporting results, pass@1, pass@k, average attempt cost, failure-type distribution, and confidence intervals should be given at the same time.

    For developers and enterprise users, a more practical approach is to build a small private evaluation set for business scenarios, fix the sampling budget, and record model outputs, verification results, and manual review conclusions. When building such pipelines on cloud infrastructure, general container platforms, object storage, logging services, and monitoring and alerting can handle task scheduling, result retention, and quality tracking, but the core of evaluation remains protocol consistency and verification reliability.

    6. Conclusion: From Leaderboard Comparisons to Capability Attribution

    The value of evaluating the mathematical abilities of frontier LLMs is not to produce a simple ranking, but to help researchers understand what a model can do, what it cannot do, and under what budget and verification conditions it can do it. Different attempt counts, different verifiers, and different tool permissions all change the meaning of scores. A truly rigorous evaluation should restore a “score” to a combination of “protocol + deliverable + verification result,” and explain failure types and capability boundaries.

    Only when an evaluation can answer “what did the model deliver, how do we verify it, what is the cost, and where is the boundary” can mathematical ability evaluation move from leaderboard comparison toward capability attribution and provide a more reliable basis for LLM reasoning research, engineering applications, and safety assessment.