How to Evaluate LLM Mathematical Reasoning: An Engineering Approach

    Accurately evaluating the mathematical reasoning capabilities of large language models requires a systematic, repeatable engineering approach rather than relying on subjective impressions. This article provides a comprehensive framework for defining evaluation objectives, constructing leak-proof task sets, implementing layered result verification, and tracking key metrics. It also covers best practices for horizontal comparison, quota management, and applying evaluation results to practical engineering decisions.

    I. Why Systematic Evaluation is Necessary

    Subjective perceptions of a large language model’s capabilities are easily skewed by single interactions. A brilliant derivation might lead you to overestimate a model, while a failed response could lead to unfair dismissal. Mathematical reasoning tasks are verifiable, making a more reliable approach possible: breaking evaluation down into a repeatable workflow—fixing tasks, standardizing conditions, recording results, and analyzing distributions. This framework is applicable for internal model selection, version regression testing, and performance tracking.

    II. Define Evaluation Objectives First

    Before starting an evaluation, three questions need to be answered:

    1. What capability is being evaluated? Theorem proving, symbolic computation, word problem solving, and open-ended problem exploration demand different things from a model, and task design should reflect this.
    2. What decision will the results drive? Model selection, regression testing, and cost-benefit analysis focus on different metrics.
    3. What auxiliary tools are allowed? Can the model invoke external calculation tools or use multi-turn prompting? If conditions are not standardized, results cannot be compared.

    It is recommended to document these details in an evaluation specification sheet and check against it before each run.

    III. Constructing the Task Set

    3.1 Task Sources

    Tasks can generally be divided into three categories:

    • Questions with standard answers: Derived from textbooks, exercise sets, or verified propositions, suitable for statistical accuracy rates.
    • Propositions with public proofs: Require the model to provide a complete derivation. Before assigning, verify the references to ensure a known proof exists, avoiding unsolved problems being treated as standard questions.
    • Open-ended problems: Suitable for exploratory evaluation. Results require line-by-line expert review and should not be directly counted toward the pass rate.

    3.2 Preventing Data Leakage

    If questions and answers frequently appear in public corpora, the model’s performance might stem from memorization rather than reasoning. Common mitigation strategies include:

    • Making equivalent rewrites to problems, changing variable names, values, or narrative order.
    • Using new problems outside existing evaluation sets.
    • Comparing performance on original vs. rewritten problems; significant discrepancies indicate memorization.

    3.3 Unified Task Record Structure

    Each question should use a standardized record format for subsequent statistics:

    1. {
    2. "task_id": "alg-geo-0231",
    3. "category": "proof",
    4. "source": "textbook_exercise",
    5. "difficulty": "medium",
    6. "statement": "Problem statement text",
    7. "reference_solution_available": true,
    8. "reference_citation": "Reference information",
    9. "allowed_tools": ["none"]
    10. }

    IV. Result Verification

    Verifying mathematical outputs is more complex than classification tasks and typically requires a phased approach:

    1. Final conclusion check: Compare against the reference answer. Only applicable for questions with definite answers.
    2. Derivation step review: Check the validity of each inference step-by-step. Expert review or formal verification tools can handle this.
    3. Citation and originality check: If the output cites theorems, verify accuracy. If it closely resembles existing literature, record it separately to distinguish between re-derivation and pure recitation.
    4. Completeness judgment: Check whether the derivation covers all cases and edge conditions.

    A common misconception is equating a correct final answer with correct reasoning. These should be scored separately. Cases where the conclusion is correct but the argument is flawed are very common in proof-based tasks.

    V. Evaluation Metrics

    It is recommended to record at least the following metrics:

    Metric Description
    First-pass rate Proportion of tasks yielding a correct conclusion in a single output
    Valid proof rate Proportion of tasks where the conclusion is correct and every step holds
    Fatal error rate Proportion of tasks with critical errors invalidating the conclusion
    Consistency Proportion of identical results across multiple runs of the same task
    Average elapsed time Time from task submission to obtaining a usable result
    Resource consumption Number of input/output tokens or quota usage

    A calculation example is shown below:

    1. from dataclasses import dataclass
    2. @dataclass
    3. class TaskResult:
    4. task_id: str
    5. final_correct: bool
    6. steps_valid: bool
    7. elapsed_s: float
    8. tokens: int
    9. def summarize(results: list[TaskResult]) -> dict:
    10. n = len(results)
    11. if n == 0:
    12. return {}
    13. return {
    14. "first_pass_rate": sum(r.final_correct for r in results) / n,
    15. "valid_proof_rate": sum(r.final_correct and r.steps_valid for r in results) / n,
    16. "avg_elapsed_s": sum(r.elapsed_s for r in results) / n,
    17. "avg_tokens": sum(r.tokens for r in results) / n,
    18. }

    VI. Common Interference Factors

    The same model can perform vastly differently under various configurations. Fix these variables during evaluation or record them comprehensively:

    • Prompts: Wording, examples, and whether an outline is provided.
    • Reasoning intensity or thinking budget: Many services offer different tiers; varying tiers alter both elapsed time and accuracy.
    • Sampling parameters: Parameters like temperature affect output stability.
    • Context: In multi-turn conversations, previous content might influence subsequent derivations.
    • Sample size: A few questions are insufficient for conclusions. Break down statistics by task category and observe fluctuation ranges through multiple runs.

    VII. Principles for Horizontal Comparison

    When comparing two models, the following conditions should be met:

    1. Use the same task set, prompts, and verification standards.
    2. Run each task multiple times, focusing on the distribution of results rather than a single run.
    3. Report failure cases and categorize them by reason (e.g., incorrect conclusion, step jumps, invalid citations).
    4. Report accuracy, elapsed time, and resource consumption together, avoiding a single-dimension view.

    Generalizing a model’s overall capability based on the results of one or two difficult problems is generally unreliable.

    VIII. Usage Management and Task Orchestration

    Long-running inference tasks can easily exhaust quotas. If a task fails midway, the incurred costs are hard to recover. In engineering, consider the following practices:

    • Split tasks: Slice large tasks into sub-problems, saving intermediate results after each segment.
    • Set checkpoints: Write to storage after each sub-task completes; resume from the breakpoint if interrupted.
    • Monitor quotas: The scheduler records used quota and pauses with notification when approaching the threshold.
    • Tiered routing: Select the model or reasoning tier based on task complexity. Use cost-effective configurations for code review, defect finding, and simple feature development, reserving high-end tiers for complex derivations.

    Example code:

    1. def run_batch(tasks, budget_checker, runner, store):
    2. for task in tasks:
    3. if task.id in store: # Skip completed tasks, supports resume from breakpoint
    4. continue
    5. if not budget_checker.has_quota(task.estimated_cost):
    6. print("Insufficient quota, pausing execution")
    7. break
    8. result = runner(task)
    9. store.save(task.id, result) # Persist immediately after each task completes

    IX. Applying Results to Engineering Decisions

    Evaluation results should serve specific decisions rather than simply forming a leaderboard. It is recommended to:

    • Establish an evaluation dashboard to display metric changes by version, task category, and time.
    • Annotate the scope of application for each conclusion. For example, “Performs stably on algebraic proofs, but sample size is insufficient for open-ended problems.”
    • Periodically re-run evaluations. Server-side models may update, causing performance on the same API to shift over time.
    • Compile typical errors found during manual review into a case library to improve prompts and workflows.

    X. Summary

    The core of systematic evaluation lies in fixing conditions, breaking down dimensions, and preserving evidence. Mathematical reasoning tasks particularly require scoring answer correctness and argument validity separately. As long as the process is repeatable, evaluation conclusions can accumulate over time, supporting more reliable technical selection and resource allocation.