Evaluating the Math Capabilities of Large Language Models: Methodology and Engineering Practice

    Evaluating whether one large language model is "better at math" than another is harder than a single conversation or a single leaderboard can settle. This article lays out a systematic, reproducible approach to benchmarking the math capabilities of LLMs, covering how to define capability across multiple dimensions, how to design and calibrate problem sets, how to score reasoning steps rather than only final answers, how to apply sampling and statistics such as pass@k, how to handle open-ended problems, and how to operationalize the whole process as an engineering pipeline. It includes reference code and a practical checklist that teams can adapt for model selection and iteration.

    Math capability is often a top consideration when teams select a large language model, yet “who is stronger” is hard to determine through one conversation or one leaderboard. This article systematically lays out how to evaluate the math capabilities of LLMs in a scientific, reproducible way, spanning evaluation dimensions, problem-set design, scoring criteria, sampling statistics, open-ended problems, and engineering implementation.

    1. Why “who is stronger” can’t be answered in one sentence

    In model communities you often see discussions where a user ranks different models based on a batch of problems they personally tested. Such evaluations usually reflect genuine hands-on experience, but turning individual impressions into a reliable conclusion faces at least three difficulties.

    First, sample bias. The number of problems an individual tests is limited, and personal preferences differ. Whether a user favors proof problems or computation problems can lead to very different results.

    Second, problem quality and difficulty. Problems collected from the web often lack answers, are ambiguously worded, or even contain errors. Difficulty is usually uncalibrated, which leads to poor discrimination between models.

    Third, run-to-run randomness. Reasoning models typically produce stochastic chains of thought; even with the same model and the same problem, multiple runs can yield different intermediate steps and answers.

    So answering “what level is a given model’s math ability at” is closer to an engineering problem that requires controlling variables, sampling repeatedly, and applying statistics, rather than ranking by impression.

    2. First define “math capability”: from vague judgment to multi-dimensional characterization

    The first step is not finding problems but clarifying what to measure. LLM math performance can be broken down into at least these dimensions:

    1. Computation and algebra: multi-step arithmetic, solving equations and inequalities, algebraic manipulation. Answers are unambiguous, making these good for basic capability checks.
    2. Modeling and application: translating a real-world problem described in natural language into math expressions and solving it, testing whether the model identifies implicit conditions and makes reasonable assumptions.
    3. Reasoning and proof: including inequality proofs, number theory, and combinatorial reasoning, with emphasis on whether the logical chain is complete and whether it relies on unproven assertions.
    4. Geometry and chart understanding: spatial relationships, geometric properties, and reading data from images; this demands more of multimodal input and visual parsing.
    5. Stability on multi-step problems: the more steps, the higher the chance an intermediate error propagates into the final answer; measure how accuracy decays as the chain lengthens.
    6. Open-ended exploration: given a problem with no mature solution or no clear answer, can the model propose a reasonable analytical framework, point to feasible directions, and honestly state its limitations? This is especially valuable for research assistance.

    The point of the dimensions is this: two models may trade wins on computation problems but differ markedly in proof rigor. Reporting a single aggregate score hides this structural difference.

    3. Problems and problem-set design: the ceiling of evaluation quality

    The problem set is the foundation of the evaluation system. Design should focus on the following.

    1. Difficulty tiers. Divide problems into basic, intermediate, advanced, and open levels, and observe each model’s pass rate per tier. Looking only at the overall score tends to yield a vague “they’re about the same” conclusion; tiered statistics reveal whether a model only leads on easy problems.

    2. Transparent problem-type mix. The ratio of proof to computation problems strongly affects conclusions and must be stated in the report. Tilting toward harder problem types better reflects real business needs.

    3. Preventing data contamination. Public problems may already be in some models’ training data, so they “memorize the answer” rather than “learn to reason.” Countermeasures:

    • Construct or rewrite problems, changing both numbers and phrasing;
    • Build a private, non-public validation set to double-check key conclusions;
    • Periodically add new problems to maintain a sustainable dynamic problem set.

    4. Answer validation and uniqueness. Prefer problems with a single definitive answer, and record the reference answer and its equivalent forms (e.g., fraction vs. decimal). For problems with multiple valid solution paths, answer extraction and judging logic must cover all equivalent forms, otherwise false negatives arise.

    5. Difficulty calibration. Before putting new problems into use, run them through several baseline models and estimate difficulty roughly from pass rates, then adjust the tiers accordingly.

    4. Chain-of-thought and scoring criteria: the process matters more than the answer

    Math evaluation can’t look only at whether the final answer is correct.

    Process quality also needs evaluation. A model may “guess right” through flawed reasoning, or make an arithmetic slip despite a correct approach. Counting only answer accuracy can’t distinguish these. A more robust approach is to score the solution process point by point:

    • Whether key steps are complete and whether any steps are skipped;
    • Whether the theorems and formulas used are valid and their applicability conditions are met;
    • Whether case analysis is complete and boundary cases are handled;
    • Whether symbols and notation are standard and clear.

    Open-ended problems need a rubric. For problems without a single reference answer, define a rubric in advance, for example scoring in tiers across five areas: “problem understanding and reasonableness of assumptions,” “coherence of reasoning,” “key insights and creativity,” “correctness of conclusions and boundary handling,” and “clarity of presentation.”

    Scorers should be diverse. Using a model as judge is cheap and scalable, but it may carry biases such as favoring its own style or preferring longer answers, so sample-based human review is needed and scoring order should be randomized. High-value problems should use two independent scorers, with agreement as a trust indicator.

    Chains of thought should be readable. For reasoning models, the conciseness and information density of the full chain of thought are part of readability: a long, repetitive process of self-doubt is not necessarily better than a concise derivation that captures the key point.

    5. Sampling and statistics: making conclusions survive repeated testing

    A single run isn’t enough to support a selection decision; the design needs statistics.

    1. Multiple samples. Run each problem at least 5–8 times. When calling a reasoning model, even with temperature set to 0, output can fluctuate, so multiple sampling is necessary, not optional.

    2. Report pass@k. Reporting only a single-run accuracy can’t show whether a model can “get it right with a few tries.” pass@k is the fraction of problems where at least one of k samples passes, commonly estimated unbiasedly by:

    1. pass@k = 1 - C(n - c, k) / C(n, k)

    where n is the number of samples and c is the number of passes. In practice, k shouldn’t be too large, otherwise it mainly measures “can it eventually hit,” diverging from the real-world feel of “usually gets it right the first time.”

    3. Report dispersion. Beyond the mean, also report the standard deviation or confidence interval of pass rates across problems. With few problems, a gap of a few percentage points may come entirely from random fluctuation and isn’t enough to claim superiority.

    4. Set termination conditions. Define what counts as “done,” for example the model says it can’t continue, gives a final answer, reaches the maximum number of turns, or exceeds budget. Inconsistent termination conditions make turn statistics incomparable.

    5. Fix experimental conditions. Sampling count, temperature, maximum generation length, system prompt, whether tools are available, and so on must be kept consistent and recorded with the report.

    The reference code below shows a basic evaluation loop, with the model call wrapped behind a unified interface so implementations are easy to swap:

    1. import random
    2. from dataclasses import dataclass, field
    3. @dataclass
    4. class EvalConfig:
    5. samples: int = 5 # Samples per problem
    6. k: int = 3 # k in pass@k; must be <= samples
    7. temperature: float = 0.6 # Generation temperature
    8. max_turns: int = 20 # Max dialogue turns for interactive solving
    9. seed: int = 42 # Fixed random seed for reproducibility
    10. def evaluate(model_client, problems, judge, cfg: EvalConfig):
    11. """Sample each problem multiple times and compute pass rate."""
    12. per_problem = []
    13. for p in problems:
    14. passes, turns, tokens = 0, [], []
    15. for _ in range(cfg.samples):
    16. # Build a session and call the model under a reproducible RNG
    17. rng = random.Random(cfg.seed)
    18. session = model_client.new_session(
    19. system=SYSTEM_PROMPT, temperature=cfg.temperature, rng=rng
    20. )
    21. result = solve_interactively(session, p, cfg.max_turns)
    22. tokens.append(result.total_tokens)
    23. if judge.is_correct(p, result):
    24. passes += 1
    25. turns.append(result.turns)
    26. per_problem.append({
    27. "id": p.id,
    28. "difficulty": p.level,
    29. "pass_rate": passes / cfg.samples,
    30. "avg_turns_on_success": sum(turns) / len(turns) if turns else None,
    31. "avg_tokens": sum(tokens) / len(tokens),
    32. })
    33. return per_problem
    34. def pass_at_k(n: int, c: int, k: int) -> float:
    35. """Unbiased estimate of pass@k."""
    36. if n - c < k:
    37. return 1.0
    38. return 1.0 - comb(n - c, k) / comb(n, k)

    6. Interactive evaluation: closer to real usage

    In real use, models rarely settle things in one shot; they converse over multiple turns with the user, use tools, and correct step by step. The evaluation approach should adapt accordingly.

    Follow-up and correction in multi-turn dialogue. For problems that can’t be solved under a standard prompt, design a uniform two-step follow-up: the first gives a hint, the second points out the error location. Comparing success rates across the two stages reveals how well the model uses hints and how well it corrects its own mistakes.

    Tool-calling ability. In math tasks, using calculation or symbolic tools often sharply reduces multi-step arithmetic errors. Evaluation should state clearly whether tools are available, and record “model-only mode” and “model + tools mode” separately, to avoid conflating the two capabilities.

    Error attribution. Attribute each failed sample, distinguishing these types:

    • Misunderstanding;
    • Formula misuse;
    • Arithmetic error;
    • Reasoning step skipped;
    • Problem with expressing the conclusion;
    • Tool-call failure.

    The proportions of each type guide optimization better than pass rate alone.

    Token consumption and cost. Multi-turn interaction and tool calls increase token usage, so average turns, average tokens, and cost should be included in the report, to prevent “higher pass rate but too costly” from being overlooked.

    7. Common evaluation biases and countermeasures

    Bias type Symptom Countermeasure
    Sample bias Few problems, personalized preferences, conclusions shift with who writes the problems Increase problem count, involve multiple authors, specify the type mix
    Random fluctuation Rankings change across runs Multiple samples; report mean, standard deviation, and confidence interval
    Data contamination Inflated scores on public problems, drop on private new ones Construct and rewrite problems; use a dynamically maintained private set
    Inconsistent scoring Subjective scoring varies by person and over time Define the rubric in advance, two independent scorers, randomize scoring order
    Missing cost dimension Looking only at accuracy, ignoring turns and token cost Also track turns, token count, latency, and failure rate

    8. Engineering implementation of the evaluation pipeline

    Evaluations often involve many problems, multiple samples, and multi-turn calls, so they need a stable pipeline to carry the load:

    1. Task splitting and dispatch: split “problem × sample count × model” into independent tasks, distributed via a message queue, ensuring tasks are retryable and traceable.
    2. Elastic inference: run the evaluation programs and model inference services on a container platform, scaling in and out elastically with concurrency.
    3. Data archiving: store the original problem set and the full conversation records and model outputs of every run together in object storage, so any conclusion can be traced back to raw data.
    4. Result storage and metric computation: write structured results to a database, aggregate by problem type, difficulty, and error type, and generate comparison reports and visualizations.
    5. Monitoring and alerting: set monitoring and alerts for call failure rate, average latency, and token consumption, to prevent network fluctuation, rate limiting, or service anomalies from contaminating evaluation data.

    For teams already running business in the cloud, all of the above can be implemented using Baidu AI Cloud’s object storage, message queue, container platform, log service, monitoring and alerting, and similar general-purpose technology components.

    The pipeline skeleton below is for reference:

    1. def run_pipeline(tasks, enqueue, process_task, save_result):
    2. """Pipeline skeleton: message queue dispatches tasks; results archived together."""
    3. for task in tasks:
    4. enqueue(task) # Split into "problem × sample count × model" granularity
    5. while True:
    6. task = dequeue() # Fetch a pending task; failed tasks auto-retry
    7. if task is None:
    8. break
    9. try:
    10. result = process_task(task) # Call model, solve interactively, run judging
    11. result["trace_id"] = task.id # Keep full conversation records for reproducibility
    12. save_result(result) # Archive to object storage and write to database
    13. except Exception as exc:
    14. log_failure(task, exc) # Log failed samples to avoid polluting statistics

    9. The evaluation report: making conclusions trustworthy

    A qualified math capability evaluation report should include at least:

    • Evaluation goals and scenario assumptions: research assistance, tutoring, or business computation — different scenarios emphasize different capabilities;
    • Model information and call conditions: version identifier, evaluation date, temperature, maximum generation length, whether tools are allowed;
    • Problem-set description: number of problems, type mix, difficulty tiers, data sources, and decontamination methods;
    • Statistical results: overall and per-difficulty, per-type pass rates, pass@k, average turns, and token consumption;
    • Open-ended scoring: rubric, scorer agreement, and sampled human review results;
    • Failure case analysis: typical errors and the distribution of attributions;
    • Conclusions and scope of applicability: clearly answer “which model is more suitable, on which problem types, under which budget.”

    10. Conclusion

    Evaluating the math capabilities of large language models is essentially a data-driven, statistics-backed systems engineering task. Judging whether a model is “good at math” shouldn’t rest on wins and losses on individual problems, but should return to the fundamentals: which problems to use, how to score, how many times to sample, and how to do the statistics.

    Only by building a tiered problem set, defining scoring dimensions, sampling multiple times, controlling variables, and ensuring full-pipeline traceability can we turn “gut-feel judgment” into reproducible, comparable engineering conclusions. This is precisely what lets an evaluation system continuously serve model selection and iterative optimization.