Evaluating LLM Math Reasoning: Methodology and Engineering Practices

    Math reasoning evaluation for large language models cannot be reduced to a single ranking. This article explains why math ability should be evaluated in layers, covering computation and format compliance, standard problem solving, and open-ended reasoning. It also covers benchmark design, contamination prevention, shortcut resistance, answer and reasoning scoring, statistical confidence, regression-friendly evaluation pipelines, and scenario-based selection guidance.

    1. Start with a premise: math ability is not one-dimensional

    Community debates often reduce a model to “rank N in math.” In engineering, this framing is difficult to verify because “math ability” is a composite concept. To make evaluation reproducible, it must be broken down into measurable levels.

    L1: Computation and format compliance. This layer covers multi-step arithmetic, unit conversion, symbolic manipulation, and outputting the answer in the required format. It measures whether a model avoids low-level errors. Such tasks are easy to grade and can be evaluated at large scale, but they offer limited discrimination among modern models and are often saturated.

    L2: Conventional problem solving. This layer includes textbook problems, standard competition questions, and word problems that can be graded automatically. The solution paths are frequent in training corpora, so models can often complete them through pattern matching combined with local reasoning. This layer makes up the majority of public math benchmarks.

    L3: Open-ended problems. These problems require multi-round loops of “conjecture—construct—verify—refute.” There is no ready-made template and no unique intermediate path. The model often has to decide which approach is feasible before executing it. The key difference from L2 is that the answer cannot be guessed through pattern matching. As a result, differences between models are significantly amplified here, and performance at this level better reflects usefulness in real research settings.

    Conclusion: any overall ranking that does not distinguish these levels has limited information value. A model’s lead in L1 and L2 cannot be extrapolated to L3. Conversely, some models may perform strongly on L3 while not leading on conventional L2 tasks.

    About chain-of-thought length. A common misconception is that “longer and more complex” chain-of-thought means “stronger.” In practice, chain-of-thought length and accuracy are not monotonically related. Overly long reasoning chains can introduce repeated derivations, mid-course self-negation, condition drift, and therefore higher error rates. A more engineering-relevant metric is the accuracy-cost frontier: at a given accuracy level, who uses fewer tokens and has lower end-to-end latency; or, at a given token budget, who achieves higher accuracy. Concise reasoning often does not indicate weaker capability, but higher search efficiency. In batch processing and high-concurrency scenarios, this directly translates into cost advantages.


    2. Benchmark design: the first source of credibility

    The credibility of an evaluation conclusion is bounded by benchmark quality, not model capability alone.

    1. Contamination resistance. Once public benchmark sets enter training corpora, their scores lose discriminative meaning. Practical measures include: running n-gram overlap and semantic nearest-neighbor checks on candidate questions, removing samples highly similar to known public questions; keeping a private holdout set that is never released for final acceptance; and splitting by time, using only questions after the model’s knowledge cutoff.

    2. Shortcut resistance / robust perturbation. Multiple variants of the same question can reveal whether a model “understands” or has “memorized” the problem. Common perturbations include: rewriting numbers and entities, adding or removing irrelevant conditions, changing units, shuffling the order of conditions, or converting a direct problem into an inverse problem, such as giving a conclusion and asking for a parameter. If accuracy drops sharply after perturbation, the model’s capability has likely been overestimated.

    3. Difficulty and labeling system. Each sample should be labeled at least with: knowledge tag, number of required key steps, whether constructive proof is required, and whether the answer is unique. Without labels, only an average number can be produced, which is often meaningless. Labels enable stratified analysis.

    4. Grading-friendly output. The prompt should specify a unified output marker for the final answer, so answer extraction does not depend on fragile natural-language regex. The gain from this step is usually larger than switching models.


    3. Scoring: both answer correctness and reasoning validity

    Grading only the final answer misses cases where the answer is correct but the reasoning is invalid, such as lucky cancellation or circular argument. Grading only the reasoning process is too expensive for manual review. In engineering practice, the recommended approach is: fully automatic answer grading + process spot-checks + failure attribution.

    Three-level answer judging:

    1. Exact matching: suitable for discrete values or sets.
    2. Numerical tolerance checking: accounts for floating-point behavior, rounding, and units.
    3. Symbolic equivalence checking: normalizes algebraic expressions, equations, and intervals before comparison, avoiding false negatives such as 1/2 vs. 0.5 or (x+1)^2 vs. x^2+2x+1. Symbolic computation capability is best wrapped as an independent service, making it easier to replace and validate separately.

    Process judging: Annotate several “key intermediate quantities” for each question, then automatically check whether they appear in the model output and whether they are correct. This helps identify samples where the final answer was guessed and also reflects the reliability of the reasoning chain.

    Failure attribution categories are a core output of evaluation: parsing failure / prompt understanding deviation / calculation error / reasoning jump, such as missing a key argument / format violation / fabricating nonexistent theorems or conditions. The distribution of failure reasons is often more useful than a single total score for guiding improvement. If failures concentrate on formatting and parsing, the issue lies in prompts and post-processing. If failures concentrate on reasoning jumps, the issue lies in the model’s reasoning capability.

    A minimal evaluation skeleton, using only the standard library:

    1. from dataclasses import dataclass, field
    2. import math
    3. @dataclass
    4. class EvalItem:
    5. id: str
    6. prompt: str
    7. answer: str # normalized reference answer
    8. tags: list = field(default_factory=list)
    9. @dataclass
    10. class CaseResult:
    11. item_id: str
    12. final_answer: str
    13. correct: bool
    14. fail_reason: str = "" # parse_error / wrong_answer / format_violation
    15. latency_ms: int = 0
    16. tokens: int = 0
    17. ANSWER_MARK = "####"
    18. def extract_answer(text: str) -> str:
    19. """Extract the final answer by the agreed marker; return an empty string if parsing fails so it can be attributed to parse_error"""
    20. if ANSWER_MARK not in text:
    21. return ""
    22. return text.split(ANSWER_MARK, 1)[1].strip()
    23. def is_equivalent(pred: str, ref: str, rel_tol: float = 1e-6) -> bool:
    24. """Numerical tolerance checking; symbolic equivalence is handled by a separately encapsulated computation service"""
    25. try:
    26. return math.isclose(float(pred), float(ref), rel_tol=rel_tol)
    27. except ValueError:
    28. return symbolic_equal(pred, ref) # external symbolic computation capability; replaceable implementation
    29. def run_case(item: EvalItem, raw_output: str, latency_ms: int, tokens: int) -> CaseResult:
    30. pred = extract_answer(raw_output)
    31. if not pred:
    32. return CaseResult(item.id, "", False, "parse_error", latency_ms, tokens)
    33. ok = is_equivalent(pred, item.answer)
    34. return CaseResult(item.id, pred, ok, "" if ok else "wrong_answer", latency_ms, tokens)

    Note that retaining fail_reason, latency_ms, and tokens determines whether evaluation results can further support cost analysis and regression comparison.


    4. Statistical treatment: do not let twenty questions decide technology selection

    This is the most easily overlooked step in practice, and also the easiest place to reach the wrong conclusion.

    • Confidence intervals must be reported. Accuracy is a proportion estimate, and small sample sizes can produce large fluctuations. It is recommended to report intervals rather than bare point estimates:
    1. import math
    2. def wilson_interval(k: int, n: int, z: float = 1.96) -> tuple[float, float]:
    3. if n == 0:
    4. return (0.0, 1.0)
    5. p = k / n
    6. d = 1 + z * z / n
    7. center = (p + z * z / (2 * n)) / d
    8. half = z * math.sqrt(p * (1 - p) / n + z * z / (4 * n * n)) / d
    9. return (max(0.0, center - half), min(1.0, center + half))
    • Prefer paired comparisons. Comparing two models on the same question set, with the same prompt template and the same parser, is far more reliable than citing public scores produced under different methodologies.
    • Separate pass@1 and pass@k. pass@k reflects upper-bound potential and is suitable for scenarios with multiple retries. pass@1 reflects single-shot usability and is suitable for automated pipelines. Majority voting, or self-consistency voting, can improve accuracy, but cost grows linearly with the number of samples and must be weighed against business value.
    • Fix and record sampling parameters. Temperature, number of samples, maximum output length, and prompt version should all be recorded. Changing any of them makes historical scores incomparable.

    5. Engineering: turn evaluation into a repeatable regression pipeline

    A one-time benchmark run has limited value. What is truly useful is continuous regression. The evaluation system can be decoupled into three layers: data, execution, and metrics.

    • Data layer: Benchmark sets carry version numbers, results are stored in object storage, and raw outputs are retained in full for reproduction and secondary attribution.
    • Execution layer: Evaluation jobs run in containers, while parsers and symbolic judging services are deployed independently, making unit testing easier.
    • Metric layer: Accuracy, average latency, average tokens, and the distribution of failure reasons are unified into a logging service, with monitoring and alerting thresholds for “accuracy decline on key benchmark sets.”

    Triple binding is the key to governance: benchmark version + prompt version + model version together define one evaluation run. Missing any one of them makes historical data unusable for regression judgment.

    In a cloud environment such as Baidu AI Cloud, these three layers can directly map to general capabilities such as object storage, container platforms, logging services, and monitoring and alerting, without building the entire infrastructure from scratch. Symbolic computation and answer parsing can be deployed as stateless services and scaled independently.

    Configuration example:

    1. suite: math-eval-v3 # benchmark version
    2. prompt: cot-zh-v5 # prompt version
    3. target: model-snapshot-a # model snapshot under test
    4. params:
    5. temperature: 0.0
    6. samples: 1
    7. max_output_tokens: 8192
    8. grading:
    9. numeric_rel_tol: 1.0e-6
    10. symbolic_equal: true
    11. process_check_sample_rate: 0.1 # process spot-check rate
    12. report:
    13. include_confidence_interval: true
    14. group_by: [knowledge_tag, difficulty]

    6. Scenario-based selection recommendations

    Scenario Key metric Evaluation setup
    Customer service, general Q&A Format compliance, latency, cost Focus on L1, large sample size, monitor P95 latency
    Code and data-processing assistance First-attempt pass rate, executability Sandbox execution judgment, prioritize pass@1
    Education, question-bank construction Answer correctness + parsing completeness Full L2 coverage + process spot-checks
    Research, open-ended problem exploration Construction ability, rigor of assumptions L3 private benchmark, small-sample human review, observe upper bound with pass@k
    High-concurrency batch processing Accuracy-cost frontier Compare accuracy under a fixed budget rather than fixed-set scores

    7. Common pitfalls

    1. Treating public benchmark sets as acceptance standards—scores are already saturated, and discrimination approaches zero.
    2. Looking only at accuracy without latency and token cost, leading to uncontrolled costs after deployment.
    3. Judging capability by chain-of-thought length while ignoring search efficiency and self-interference.
    4. Failing to validate the evaluator itself—automatic judgments should be spot-checked manually. Evaluator errors will directly contaminate model conclusions.
    5. Mixing benchmark versions and prompt templates across runs, turning historical curves into noise.
    6. Making major selection decisions from single small-sample benchmark runs without reporting confidence intervals.

    8. Summary

    Differences in model math capability are real, especially for open-ended problems that require conjecture and construction. However, such differences become actionable information only when four elements are present: layered benchmarks + rigorous scoring + sound statistical treatment + a repeatable regression pipeline. Evaluation should be treated as a continuous measurement engineering practice, not a one-off comparison showcase. Only then can teams avoid being misled by a single score, a single dimension, or chain-of-thought length.