Designing and Implementing a Math Reasoning Evaluation System for Large Language Models

    Intuition-based comparisons of LLM math ability — "Model X just feels stronger" — don't survive scrutiny because of sample bias, fatigue, non-reproducibility, and mismatched reasoning budgets. This article lays out an engineering-grade, six-step framework for evaluating the math reasoning ability of large models: decomposing ability into measurable dimensions, building a layered and decontaminated question bank, controlling the evaluation protocol for comparability, moving from answer-only checks to auditable rubrics, interpreting results with statistics such as confidence intervals and paired tests, and extending evaluation from offline model selection to online regression monitoring. It includes schematic code for the evaluation pipeline and bootstrap confidence intervals, a rubric example, a standardized result card, and a table of common pitfalls with mitigations, so teams can turn model-selection decisions into repeatable, auditable, and business-aligned engineering practice.

    1. Why “it just feels stronger” doesn’t hold up

    In discussions about model selection, you constantly run into head-to-head comparisons like “Model X is better suited to advanced math problems,” “Model Y reasons more concisely and reads closer to natural writing,” or “this new version is roughly on par with such-and-such tier.” These judgments often come from genuine hands-on experience and carry some value, but they don’t withstand scrutiny:

    • Sample bias: The question types, difficulty levels, and prompting styles an evaluator tends to test happen to be the areas they’re most familiar with. Doing well on familiar ground doesn’t mean covering everything, and strong performance on one category of problem can mask weaknesses elsewhere.
    • Order and fatigue effects: When testing several models back to back, the first ones tested earn an impression bonus, the later ones suffer from fatigue, and memory tends to amplify the most recent experience.
    • Cognitive bias: Brand preference, whether a version is new or old, and UI smoothness all sway the judgment. The same model with a different prompt or a different reasoning mode can produce a noticeably different experience.
    • No reproducibility: Without recorded questions, parameters, and raw outputs, nobody can verify the conclusion afterward.
    • Comparability: Different models ship with different default reasoning budgets, sampling parameters, and output-truncation strategies. Comparing them directly is like comparing exam scores taken under different conditions.

    So evaluating a large model’s math reasoning ability calls for an engineering-grade evaluation system: one that can be run repeatedly, produces auditable results, and yields interpretable metrics. The rest of this article unfolds in six steps: define the dimensions, build the question bank, control the protocol, do the judging, read the statistics, and connect to production.

    2. Step 1: Break “math ability” into measurable dimensions

    “Strong at math” is a vague claim. It should at least be decomposed into the following dimensions and measured separately.

    1. Final-answer accuracy

    This includes single-turn accuracy and pass@k across multiple samples. Single-turn accuracy reflects the probability of getting it right on the first attempt; pass@k reflects the ability to find a correct answer when multiple attempts are allowed.

    2. Effectiveness of the reasoning process

    Looking only at the final answer distorts things in two ways: first, a model can reach the right conclusion through faulty steps — it “guesses right”; second, complex problems often admit several reasonable solution paths, and answers alone can’t tell apart levels of quality. Key points to examine:

    • Are the key steps valid, and are definitions, boundaries, and assumptions stated clearly?
    • Can intermediate calculations be checked, such as algebraic identities, combinatorial counting, and geometric relationships?
    • Is the chain of argument complete — does the conclusion genuinely follow from the reasoning rather than being “guessed” from the phrasing of the problem?
    • Is there any “self-persuasion” — an answer stated first and then a plausible-looking sequence of steps reverse-engineered to fit it?

    3. Reasoning efficiency

    When accuracy is comparable, compare the reasoning length and number of tool calls needed to solve problems of equivalent difficulty, to measure output per unit of cost. Avoid one pitfall here: longer reasoning does not mean stronger ability. The relationship between reasoning length and accuracy is not monotonic — when a model second-guesses itself repeatedly on easy problems or converges too early on hard ones, longer reasoning just means wasted compute.

    4. Stability

    Does the same problem produce consistent results across repeated attempts? Does performance hold up across different prompt styles and phrasings? For agent tasks that must run unattended over long periods (batch data processing, continuous monitoring and operations), stability often matters more than peak accuracy.

    5. Process interpretability and quality of expression

    Is the reasoning laid out clearly and easy to review? Is the written expression natural and consistent with the target style? In many real settings the intermediate reasoning gets recorded and handed to human reviewers or a downstream step, so auditability is itself a capability.

    6. Cost and latency

    The reasoning resources consumed per successful answer, the wait time for a response, and the rework cost caused by wrong results together determine whether a capability can scale into production.

    Once the dimensions are set, also pin down what “math reasoning ability” specifically means in the target scenario. Is it problem-solving for competitions and admissions tutoring, quantitative modeling for research and finance, or a “reasoning plus code execution” capability for data engineering and algorithm-implementation work? Different positioning maps to different question banks and weights.

    3. Step 2: Build the question bank — layered, decontaminated, judgeable

    Layered design

    Layer by subject (algebra, number theory, combinatorics, geometry, probability and statistics, calculus, and so on) and by difficulty, with enough questions in each layer. The question format should cover several types:

    Question type Response form Judging method
    Solve A number, expression, or set Equivalence check (e.g., compare after symbolic simplification)
    Prove A complete argument Expert review or graded scoring against a rubric
    Open-ended exploration A conclusion plus argument, possibly requiring you to propose the claim yourself Expert review, focused on novelty of ideas and rigor of argument
    Multiple choice A single option Exact match
    Modeling and application Model setup, solution process, and conclusion Compare the conclusion plus review the process

    Question metadata

    Each question should carry a unique identifier, its subject, knowledge-point tags, difficulty, a reference answer or solution, a judging method (programmatic, human, or hybrid), a source note, and the record of the most recent edit, for traceability.

    Difficulty calibration

    Borrow from Item Response Theory (IRT) in classical measurement theory: how well a question discriminates model ability depends on the current population’s performance distribution on that question. As models improve overall, questions that once separated them may become too easy, so they should be replaced or upgraded in time to keep the evaluation’s “resolving power.”

    Decontamination

    Public question banks gradually seep into training corpora. A model that has “memorized” the questions will post inflated scores, and the evaluation loses its discriminating power. Practical measures include:

    1. Maintaining a portion of private questions internally that are never made public;
    2. Generating variants by changing numbers, swapping conditions, or shifting the angle of the question;
    3. Regularly adding new questions and maintaining a timeline of when each was added to the bank.

    Judging anchors

    For questions that can’t be fully judged automatically, such as proofs and open-ended problems, write out a reference solution and rubric in advance for each question, spelling out the key steps, the cases that must be covered, and common mistakes with how they lose points. The more specific the rubric, the smaller the divergence between different reviewers.

    4. Step 3: Control the evaluation protocol to ensure comparability

    Once the question bank is ready, differences in results may come from the protocol rather than the model itself. The following must be unified and recorded:

    • Prompts: Use a unified prompt template that specifies the required response format, such as reasoning first and then the final answer, along with conventions for how to write the answer;
    • Sampling parameters: Unify temperature, maximum output length, and reasoning budget; record whether reasoning mode is enabled and the cap on reasoning length, because results under different budgets can’t be compared directly;
    • Retry and tool strategy: Whether retries are allowed and whether tools such as calculators or code-execution environments may be called must stay consistent. If comparing “bare reasoning” against “reasoning with tools,” design controlled experiments for each rather than mixing them into one ranking;
    • Answer-extraction rules: The rule for pulling the final answer out of a model’s output (such as reading the content after a specified marker) should be fixed in advance and written into a script to avoid ad hoc adjustments;
    • Judging configuration: The version, rubric, reviewers, and review process should all be archived together;
    • Execution environment: Fix the dependency versions, images, and hardware configuration of the evaluation jobs to keep results reproducible.

    The following schematic code shows how to harden the protocol into an executable pipeline:

    1. # Evaluation pipeline (schematic): all protocol parameters are configured centrally,
    2. # and evaluation logic is kept separate from judging logic.
    3. CONFIG = {
    4. "prompt_template": "standard_v1", # unified prompt template
    5. "temperature": 0.0, # greedy decoding for single-sample reproducibility
    6. "max_output_tokens": 8192,
    7. "reasoning_budget": "default", # reasoning budget must stay consistent across models
    8. "samples_per_question": 5, # multiple samples to assess stability
    9. "answer_extractor": "extract_after_marker",
    10. }
    11. def evaluate(model_client, questions, judge):
    12. records = []
    13. for q in questions:
    14. sample_results = []
    15. for _ in range(CONFIG["samples_per_question"]):
    16. resp = model_client.generate(
    17. prompt=render(q, CONFIG["prompt_template"]),
    18. temperature=CONFIG["temperature"],
    19. max_output_tokens=CONFIG["max_output_tokens"],
    20. )
    21. sample_results.append({
    22. "answer": CONFIG["answer_extractor"](resp),
    23. "reasoning_trace": resp.get("reasoning", ""),
    24. "reasoning_tokens": resp.get("reasoning_tokens", 0),
    25. "latency_ms": resp.get("latency_ms", 0),
    26. "status": resp.get("status", "ok"), # ok / truncated / error
    27. })
    28. valid = [s for s in sample_results if s["status"] == "ok"]
    29. records.append({
    30. "question_id": q["id"],
    31. # score only valid samples; track anomalies separately so that
    32. # truncation or errors are not counted as wrong answers
    33. "verdict": judge(q, valid),
    34. "majority_agreement": majority_agreement([v["answer"] for v in valid]),
    35. "reasoning_tokens": avg([v["reasoning_tokens"] for v in valid]),
    36. })
    37. return records

    5. Step 4: Judging — from “answers only” to “auditable rubrics”

    1. Answer-level judging

    For questions whose results can be expressed in a standardized form, prefer programmatic judging: first apply an equivalence transformation, such as numeric evaluation or symbolic simplification, then compare against the reference answer. This avoids wrongly failing an answer that is “equivalent but written differently.”

    2. Process-level judging

    For proofs and open-ended problems, use a structured rubric. A rubric breaks “rigor” into a set of checkable items, for example:

    1. {
    2. "score_levels": [
    3. {"level": 5, "description": "All key steps are valid, all boundaries and special cases are fully covered, and the argument is complete"},
    4. {"level": 3, "description": "The main line of reasoning is correct, with a few minor flaws or missed cases that do not affect the conclusion"},
    5. {"level": 1, "description": "The direction is roughly reasonable, but key steps are missing, or the conclusion does not match the problem"}
    6. ],
    7. "checkpoints": [
    8. "Are the definitions used and the given conditions stated clearly?",
    9. "Can the key derivation steps be independently verified?",
    10. "Are degenerate, boundary, or special-value cases discussed?",
    11. "Does the conclusion match the required form of the problem?",
    12. "Are there any implicit assumptions that were not stated?"
    13. ],
    14. "evidence_required": true
    15. }

    A rubric paired with the requirement to “cite the original text as evidence” can significantly reduce a reviewer’s subjective drift.

    3. Consistency between automated judging and reviewers

    If a model is used to judge automatically, periodically draw a certain proportion (say 10%–20%) of samples for expert review and compute an agreement metric between reviewers and the automated results (such as Cohen’s Kappa). When agreement is low, revise the rubric or prompt rather than simply trusting the automated score.

    4. Reducing judging bias

    • Anonymize model names and versions and use blind review with anonymous IDs;
    • For comparison questions, use paired preference (A better than B, comparable, B better than A) and randomly swap the left/right positions to cancel out position preference;
    • Have multiple reviewers score independently, then compute an aggregate result;
    • Cap the volume each reviewer handles per day to avoid quality degradation from fatigue.

    6. Step 5: Interpret results with statistics

    1. Report confidence intervals, not a single score

    An accuracy of 85% over 300 questions has, by the normal approximation to the binomial distribution, a 95% confidence interval of roughly 80%–89%. If two models score 85% and 83%, those 2 percentage points most likely come from random variation and are far from enough to support a “which is stronger” conclusion.

    2. Paired comparison

    When evaluating multiple models on the same batch of questions under the same protocol, use a paired comparison (such as McNemar’s test). It focuses on the questions where the two models differ, making it more sensitive than comparing two independent proportions.

    3. Repeated evaluation and bootstrap estimation

    For sampling scenarios where temperature is nonzero, run the full evaluation several rounds for each model, or use bootstrap resampling to estimate the metric distribution and confidence interval. Schematic code:

    1. import random
    2. def bootstrap_ci(pairs, n_boot=1000, alpha=0.05):
    3. """
    4. pairs: [(model_a_correct, model_b_correct), ...] paired results on the same questions
    5. Returns: confidence interval for A's accuracy difference over B
    6. (based on resampling the paired data)
    7. """
    8. diffs = []
    9. for _ in range(n_boot):
    10. sample = [random.choice(pairs) for _ in range(len(pairs))]
    11. diff = mean(a for a, _ in sample) - mean(b for _, b in sample)
    12. diffs.append(diff)
    13. diffs.sort()
    14. lo = diffs[int(alpha / 2 * n_boot)]
    15. hi = diffs[int((1 - alpha / 2) * n_boot) - 1]
    16. return lo, hi

    4. Composite ranking

    Results across dimensions shouldn’t simply be summed into a single total score; they should be weighted against business goals. For example, metrics for an office-automation scenario could be weighted as accuracy 40%, stability 25%, cost and latency 20%, and interpretability 15%. Weights need to be confirmed by the business side and adjusted as the scenario changes.

    5. Result presentation

    It’s advisable to use a uniform result card in reports so that every conclusion can be traced to a specific dimension and evidence:

    1. Model ID: Model-A (anonymized)
    2. Evaluation date: 2025-XX-XX
    3. Question bank version: MATH-EVAL-2025Q2 (400 questions total)
    4. Protocol version: standard_v1 (temperature=0, reasoning_budget=default)
    5. Samples: 5 samples per question
    6. ────────────────────────────
    7. Metric Value (95% confidence interval)
    8. Overall accuracy xx.x% (xx.x% ~ xx.x%)
    9. Average proof score (1-5) x.x (x.x ~ x.x)
    10. Multi-sample agreement rate xx.x%
    11. Average reasoning tokens xxxx
    12. Single-call latency P95 xx.x s
    13. Truncation / error rate x.x%
    14. ────────────────────────────
    15. Paired comparison (vs Model-B): difference +x.x percentage points, p = 0.xx
    16. Conclusion: whether the difference between the two is significant on this question bank and protocol

    7. Step 6: From offline evaluation to online regression

    Offline evaluation addresses “how to choose among candidate models”; running in production addresses “whether quality has regressed.”

    1. Establish a baseline

    Before switching model versions, adjusting reasoning parameters, or editing the system prompt, run the full evaluation once and record it as the baseline; every change should be compared against this baseline.

    2. Online sampling evaluation

    Sample from real requests at some proportion and score automatically with the rubric. For high-stakes scenarios such as financial calculation and compliance review, bring in code execution or symbolic computation for cross-validation.

    3. Shadow traffic

    Have a new model receive production traffic in shadow mode first — compute but don’t return results — and compare against the model in production. Decide whether to switch after observing the distribution of differences.

    4. Online metrics

    Beyond accuracy, also monitor call success rate, truncation rate, average reasoning length, P95 latency, and unit cost. An abnormally rising truncation rate often signals an unreasonable reasoning-budget setting or a model stuck in repetitive reasoning.

    5. User-feedback loop

    Fold signals such as human corrections and upvotes/downvotes into the evaluation system as an important complement to the automated metrics.

    At the implementation level, an evaluation system typically consists of four parts: the question bank and rubrics, sampling tasks, judging tasks, and a reporting service. These can be built from general cloud product capabilities (the following are common category examples; actual selection should follow each platform’s current offerings):

    • Use object storage to hold the question bank, rubrics, and raw outputs for versioning and traceability;
    • Use a message queue to split and schedule sampling and judging tasks, supporting large-scale concurrent evaluation;
    • Use a container platform to run judging tasks and unify the runtime environment;
    • Use log services and monitoring/alerting to track evaluation-job status and fluctuations in online metrics.

    Baidu AI Cloud provides all of these general cloud product capabilities. Teams can combine them to fit their own tech stack and build a complete pipeline from offline evaluation to online monitoring.

    8. Common pitfalls and how to avoid them

    Pitfall Symptom Mitigation
    Public benchmark contamination Inflated scores that don’t reflect real ability Use a private question bank and variant questions, and rotate questions regularly
    Judging on final answers only Faulty reasoning judged as correct Introduce process scoring and key-step verification
    Bias toward longer reasoning Mistaking longer reasoning for greater intelligence Cap the reasoning budget, and assess accuracy and cost together
    Comparing across different budgets Conclusions change with parameter settings Standardize and disclose the sampling and reasoning-budget configuration
    Drawing conclusions from a single sample Conclusions swayed by random variation Use multiple samples plus confidence intervals and paired tests
    Repeatedly tuning on the eval set The question bank gets “gamed” and generalization drops Split development and held-out test sets, and rotate the held-out set periodically
    No human validation Divergence between reviewers and automated judging goes unnoticed Periodically spot-check and compute agreement metrics
    Conclusions that ignore the scenario One ranking treated as an all-around verdict Report results by scenario and by dimension, and state the scope each conclusion applies to

    9. Conclusion

    “Can this model handle advanced math problems?” and “Which one is stronger, it or that other model?” are questions that shouldn’t be answered from impression in engineering. The more reliable approach is to answer four more concrete questions:

    1. Under a clearly defined question bank and protocol, what is its accuracy?
    2. Does its reasoning hold up to step-by-step verification?
    3. Under cost and latency constraints, can it reliably serve the business?
    4. After moving to production, can the quality be sustained?

    The core of math reasoning evaluation is not to produce a “one-size-fits-all leaderboard” for models, but to build an evaluation capability aligned with your own business goals and capable of continuous iteration. Question banks will age, models will improve, and protocols will change — but a rigorous evaluation method is always the foundation for turning technical judgment into business decisions.