Boundaries and Comparability in Evaluating Large Language Models for Math

    Evaluating Large Language Models (LLMs) on math tasks is a standard benchmark for testing reasoning capabilities, but comparing scores directly can be misleading. Differences in sampling budgets, attempt counts, and evaluation criteria fundamentally impact comparability. This article examines the core parameters of LLM math evaluations, analyzes what different model outputs actually represent, defines capability boundaries from standard problems to research-level proofs, and outlines best practices for designing reproducible and meaningful evaluation pipelines.

    Mathematical ability is often considered the litmus test for the reasoning capabilities of large language models (LLMs). The reasons are straightforward: math problems require precise comprehension, consistent multi-step deduction, correct symbolic manipulation, and verifiable conclusions. Unlike open-domain QA, math tasks usually have clear correct answers, making them ideal for assessing whether a model possesses stable reasoning skills.

    However, trusting a leaderboard score requires more than just looking at difficulty and performance. A crucial detail often overlooked in raw evaluation materials is that different models attempt each problem a varying number of times. Some evaluate each question once, while others attempt multiple times. Furthermore, the reported metrics might correspond to different statistical definitions. Directly comparing these numbers horizontally leads to misinterpretation. We must ask: what exactly did the model produce behind these scores? A final answer, a complete derivation, a formal proof, executable code, or a tool-calling trace? The quality of these artifacts determines what the evaluation truly signifies.

    1. Evaluation Criteria Before Scores: Why Attempt Counts Matter

    In LLM math evaluations, the number of attempts per question is not a trivial experimental detail; it is a core budget parameter that directly affects the metric’s meaning. If a model samples independently k times per question and is marked correct if any attempt succeeds, this is effectively a “pass@k” or “best-of-k” metric. This is not equivalent to “pass@1”, where a model attempts the problem only once.

    From a probabilistic perspective, multiple attempts increase the chance of at least one hit but also significantly inflate sampling costs. A model’s score at k=3 cannot be directly compared to another model’s score at k=1. Even if both are reported as percentages, the underlying budgets differ, breaking comparability. A rigorous approach should report: attempts per question, sampling temperature, maximum inference length, tool-calling permissions, judge rules, total sampling cost, and whether the metric is pass@1, pass@k, or human-graded process scoring.

    A simplified statistical process can be represented as follows:

    1. Input: Question set Q, attempts k, sampling config S, judge J
    2. for q in Q:
    3. outputs = []
    4. for i in 1..k:
    5. y = model.generate(q, S)
    6. outputs.append(y)
    7. score = J(q, outputs) # Can return pass@1, pass@k, process score, or failure type
    8. record(q, outputs, score, budget info)
    9. Report:
    10. Aggregate by question type, difficulty, and failure type
    11. Clarify attempts, cost, and evaluation criteria

    If only a total score is presented without the budget and criteria, readers cannot tell if differences stem from model capability, sampling frequency, judge leniency, or data distribution. The first principle of math evaluation should be: define the budget before discussing the score.

    2. Behind the Scores: What Did the Model Actually Produce?

    Math task outputs are not simply “answer right/wrong”. We can categorize them into at least four layers:

    1. Final Answer: Numerical values, expressions, choices, or conclusions. This is the easiest part to auto-evaluate, but also the easiest way to mask process errors.
    2. Derivation Process: Natural language reasoning chains, algebraic transformations, geometry auxiliary line explanations, etc. Assessing the correctness of the process is much harder to automate than evaluating the final answer.
    3. Formal Proofs or Executable Verification Materials: Translating proofs into verifiable formal steps or providing runnable calculation verification code. These artifacts are highly verifiable, but the generation barrier is significantly higher.
    4. Tool Call Traces: The process of calling calculation, symbolic computation, search, or verification tools. Traces reflect whether a model can decompose problems, select appropriate tools, and verify results.

    Therefore, evaluation should ideally include result scoring, process scoring, verifiability scoring, and efficiency scoring. A model might get the final answer right but have gaps in its derivation; it might have a sound proof idea but non-standard symbolic notation; or it might accidentally hit the right answer after multiple attempts. Looking only at the final percentage flattens all these differences.

    Failure types should also be systematically recorded, such as: misunderstanding the problem, calculation errors, broken reasoning chains, proof gaps, format/judge mismatches, tool misuse, memorization of training data, and instability in long-context reasoning. Categorizing failures transforms “low scores” into actionable engineering problems.

    3. From Competition Problems to Research-Level Questions: Where Are the Boundaries of Math Capabilities?

    Math evaluation difficulty can generally be divided into: standard problems, variant problems, combinatorial generalization problems, proof generation, and open problem assistance. The capability boundaries differ across these levels.

    On standard and common variant problems, models may show strong pattern recognition and step imitation, but this is not synonymous with robust reasoning. If questions closely resemble training data, high scores might partly come from memorization and template matching. Combinatorial generalization problems require the model to reorganize reasoning in unfamiliar structures, which is more likely to expose issues with long-chain stability.

    Proof generation is a more critical boundary. Research-level math requires not just “calculating the answer” but proposing verifiable proofs, finding counterexamples, constructing auxiliary objects, and identifying implicit conditions. Current LLMs can assist in generating proof sketches, searching for counterexamples, and organizing known conclusions, but ensuring a proof has no gaps is difficult. Formal verification tools can act as a bridge: translating natural language proofs into checkable steps to distinguish between “plausible” and “verifiably correct.” However, formalization itself is costly and demands higher formal expression and tool-collaboration capabilities from the model.

    Open problem assistance is even further from automated scoring. Models might offer analogies, conjectures, or proof strategies, but these outputs require domain expert judgment. Here, the evaluation focus shouldn’t be a single accuracy rate, but rather suggestion effectiveness, verifiability, error discoverability, and manual review costs.

    4. Design Points for Reproducible Evaluation

    To make math evaluations genuinely meaningful, consider adopting the following design principles:

    • Fixed Budget: Unify attempts per question, sampling parameters, and tool permissions within the same leaderboard. If comparing different budgets is necessary, report pass@1, pass@k, and costs separately.
    • Layered Reporting: Break down by question type, difficulty, knowledge points, and failure types, rather than just looking at total scores.
    • Process Scoring: Conduct sampled blind human evaluation or rule-assisted scoring for derivation chains, proof structures, and verification materials.
    • Contamination Detection: Use dynamic variants, held-out sets, and similarity checks to reduce inflated scores caused by training data contamination.
    • Reproducible Records: Save question versions, model versions, sampling configurations, judge versions, and raw outputs.
    • Cost Dimensions: Record total sampling attempts, inference length, and tool calls to avoid ignoring “high budget for high scores.”
    • Error Analysis Loop: Convert failure cases into regression test cases to continuously monitor capability drift.

    For developers and enterprise users, math evaluation should not be just a one-off leaderboard reading, but an engineered regression test: build layered test sets around real business tasks, fix evaluation budgets, record failure types, and automatically compare results during model upgrades. Only by restoring “scores” into a combination of “budget, artifacts, criteria, and failure modes” can we accurately understand the current boundaries of LLM math capabilities and make reliable technical selections and system designs based on them.