Evaluating LLM Math Capabilities: A Reproducible Approach Beyond Gut-Feel Comparisons
Claims about which model is "best at math" usually come from single-point experience, yet hands-on impressions are easily distorted by sample bias, sampling randomness, and prompt sensitivity. This article presents a reproducible methodology for evaluating large language models' math capabilities instead of relying on gut feel. It first breaks math ability into six dimensions—correctness, process rigor, difficulty ceiling, stability, resistance to misleading premises, and cost—then shows how to build a clean, tiered question bank, score both answers and reasoning, and apply statistical methods such as repeated sampling, paired comparison, significance testing, and confidence intervals. It also covers plotting a cost-effectiveness frontier, automating evaluation as a versioned pipeline with regression gates, and using tiered routing to match models to business scenarios. Readers get a concrete, repeatable framework for grounded model selection, controllable iteration, and computable cost.
Experiential discussions frequently feature claims like these: a flagship model from a certain generation clearly leads on a hard problem; between adjacent generations, or between versions of the same generation, the gap is limited and highly dependent on the problem type; some models produce long-winded reasoning while others are more concise, and the more concise style “feels more like a person thinking.”
These judgments come from real hands-on use, and many of them do reflect genuine differences in model capability. But if we take them directly as conclusions, we slide from “single-point experience” to “a reproducible method”—and there is a considerable distance between the two.
1. Why gut-feel comparisons get distorted
Intuitive evaluation is common because it has a low barrier and gives fast feedback. A user hits a hard problem, runs two models once each, and whichever returns the correct answer first “wins.” This approach has at least three types of bias:
- Sample bias: One or a few comparisons cover only a very narrow range of problem types and difficulty. Most differences between models are conditional—stronger on a certain category, more stable at a certain difficulty band. A single-scenario result is easily over-generalized into an overall conclusion.
- Randomness bias: Mainstream models generally sample with randomness. The same model, same problem, and same prompt can yield different results across two runs. A single failure may just be luck, and a single success may not be reproducible.
- Prompt bias: How you ask strongly affects the output. Adding a “first list the known conditions, then derive step by step” instruction to a model without chain-of-thought, or limiting output length for a long-chain-of-thought model, can flip the ranking.
2. First, define: math capability is not a single number
To evaluate rigorously, the first step is not choosing models but clarifying “what we are comparing.” Capability on math tasks can be broken into at least six independent dimensions:
| Dimension | Meaning | How to observe |
|---|---|---|
| Correctness | Whether the final answer is correct | Comparison with reference answers, equivalence checks |
| Process rigor | Whether reasoning is complete, free of skipped steps, circular argument, or “guessing the answer to fit the result” | Step verification, reasoning statistics |
| Difficulty ceiling | The hardest problem type that can be handled reliably | Tiered pass rates on a layered question bank |
| Stability | Consistency across repeated attempts on the same problem | Repeated-sampling variance, self-consistency rate |
| Resistance to misleading premises | Whether it holds a correct judgment against a false premise | Adversarial samples, false-premise questions |
| Cost | Output length, latency, and per-call cost incurred to reach the correct answer | Log statistics |
In practice, people often conflate “process rigor” with “difficulty ceiling,” using “does the reasoning feel human” to infer “is the solving stronger.” The two are correlated but not equivalent: a model may reason verbosely yet end up correct, or be concise yet occasionally skip steps. Evaluation should re-separate these entangled things.
3. Step 1: Build a reproducible, tiered question bank
3.1 Layer by capability
It helps to split math problems into five levels, each corresponding to a different capability requirement:
- L1 Basic computation: Arithmetic, solving equations, simple geometry. These can be generated programmatically in bulk with automatically computed reference answers.
- L2 Multi-step word problems: Requires 2 to 5 reasoning steps, involving ratios, percentages, motion (distance-rate-time), and similar problems.
- L3 Competition-level reasoning: Combinatorial counting, number theory, inequality proofs, and the like, requiring non-routine construction or insight.
- L4 Integrated modeling: Turning a business description into a mathematical model and solving it, e.g., capacity planning, cost optimization, scheduling constraints.
- L5 Open problems: Research-style problems with no accepted answer, mainly assessing the exploration process rather than the final result.
The value of layering is that overall accuracy is only a weighted average, and performance across tiers is what exposes the capability structure. For example, two models with the same overall accuracy of 70% may differ sharply: one might score near-perfect on L1–L3 but fail badly on L4, while another might sit at roughly 70% across all tiers. The former suits standardized, controlled-difficulty workflows; the latter is more robust when difficulty fluctuates.
3.2 Keep the question bank “clean”
Public question banks have a structural problem: they may already be in a model’s training data, so the model memorizes answers instead of learning to reason, and scores are systematically inflated. Mitigations include:
- Private held-out questions: Build your own, keep them unpublished and out of training data.
- Parameter randomization: Generate random values and conditions from the same template so every instance has a different answer.
- Time slicing: Periodically add recently emerged problems and retire those suspected of being “memorized.”
- Contamination detection: Re-ask the same problem with a different numeric phrasing or narration order; a large drop in accuracy suggests the problem has been memorized.
3.3 Fix prompt templates
Once the bank is set, prompts should be templated and version-controlled. The template makes three things explicit: role and goal, output format, and chain-of-thought requirement. Example:
You are a rigorous math problem-solving assistant. Solve the problem below:[Problem] {question}Requirements:1. First list the known conditions and the target to find;2. Derive step by step, giving the basis for each step;3. Output "Answer: X" on the last line, where X is the final result only.4. If the problem's conditions are self-contradictory or the information is insufficient, state this clearly and explain why; do not force an answer.
Requirement 3 constrains the output format, making it easy to extract answers automatically; requirement 4 supports the later resistance-to-misleading tests. The same template must be applied to every model under comparison, otherwise the comparison is unfair.
4. Step 2: Scoring—from “is the answer right” to “is the process sound”
4.1 Result correctness
Answer comparison looks simple but is actually the most error-prone. Common model output forms include fractions, radicals, intervals, sets, and multi-solution statements like “x = 2 or x = 3.” Direct string comparison produces many false judgments. The correct approach has three steps:
- Extract the content after “Answer:” from the response;
- Normalize the format: remove spaces, unify case, strip markers wrapping the answer;
- Judge equivalence: for numeric problems use tolerance comparison, convert fraction strings to numbers before comparing, and for expression problems use symbolic computation or random substitution to verify identity.
Illustrative code:
import mathdef extract_answer(text: str) -> str:"""Extract the content after "Answer:"; if there are multiple lines, take the last marker to avoid interference from intermediate steps"""marker = "Answer:"if marker not in text:return ""return text.split(marker)[-1].strip().splitlines()[0]def normalize(ans: str) -> str:return ans.strip().lower().replace(" ", "").rstrip(".")def to_number(token: str):"""Supports integers, decimals, and fractions; returns None on failure"""try:return float(token)except ValueError:if "/" in token:a, b = token.split("/", 1)try:return float(a) / float(b)except (ValueError, ZeroDivisionError):return Nonereturn Nonedef judge(pred: str, gold: str) -> bool:p, g = normalize(extract_answer(pred)), normalize(gold)if p == g:return Truevp, vg = to_number(p), to_number(g)return (vp is not None and vg is not Noneand math.isclose(vp, vg, rel_tol=1e-6, abs_tol=1e-9))
This handles only numeric answers. Proofs and expression problems need more complex judging logic; in engineering it is usually layered: first rule-based matching, then symbolic equivalence, and finally routing samples that cannot be judged automatically to human review.
4.2 Process rigor
A correct answer does not mean a correct process. Common “process distortions” include: circular argument (using the conclusion to prove the conclusion), skipped steps (giving the result of a key step directly), answer-fitting (guessing the result first and reverse-engineering the steps), and misuse of conditions (ignoring constraints in the problem). There are three feasible checking methods:
- Rule checking: Parse reasoning steps, check whether conditions not given in the problem are cited, and whether derivations are missing between key steps.
- Model review: Have another model score against the reference solution item by item, focusing on the issues above. See section 4.3 for bias control.
- Human spot-check: Sample by a certain proportion and have domain experts review; this is the most reliable step when sample sizes are small.
You can also quantify the reasoning process, turning “concise” and “verbose” into measurable metrics:
import reTOKEN_RE = re.compile(r"\w+|[^\w\s]", re.UNICODE)def trace_stats(text: str) -> dict:"""Extract quantifiable process metrics from raw output (including reasoning)"""tokens = TOKEN_RE.findall(text)markers = ["wait", "hmm", "but wait", "recompute", "let me recheck", "actually"]return {"token_count": len(tokens),"digit_ratio": round(sum(t.isdigit() for t in tokens) / max(len(tokens), 1), 3),"uncertain_markers": sum(text.count(m) for m in markers),"char_count": len(text),}
Longer reasoning is not necessarily better. As an illustrative rule of thumb, the number of times the same model doubts itself and re-verifies tends to rise with difficulty, and so does the probability of error accumulation. So “concise reasoning” itself can be a quality signal: given comparable accuracy, a model with fewer reasoning steps and fewer re-verifications is usually more stable and cheaper.
4.3 Scoring open-ended problems
For problems without a reference answer, scoring is closer to “peer review,” rated on five aspects (1 to 5 each):
- Problem understanding: whether it accurately grasps the intent and implicit constraints;
- Method selection: whether the tools and approach are appropriate;
- Derivation completeness: whether each step has a basis and whether steps are skipped;
- Conclusion reasonableness: whether the conclusion holds up and whether its conditions of applicability are stated;
- Expression quality: whether the structure is clear and the terminology accurate.
Three practical rules are worth following:
- Anonymity and shuffling: Hide model identity during review and randomize the presentation order of different responses to the same problem.
- Calibrate reviewers: If a model acts as reviewer, first calibrate it on a batch of human-annotated samples and check whether it is sensitive to response length and formatting.
- Multiple reviewers, consensus: Have at least two reviewers score independently, compute agreement, and send samples with large disagreement into a review process.
5. Step 3: Replace single-shot wins with statistics
5.1 Multiple sampling and pass rate
Sampling the same model multiple times on the same problem lets you estimate its true pass rate. Two metrics are common in engineering:
- Single-attempt pass rate: the proportion correct on a single attempt, representing average performance;
- At-least-one-pass rate across multiple samples: the proportion correct at least once across several samples, representing the capability ceiling.
Looking at both distinguishes “insufficient capability” from “unstable performance.” For example, model A has a 40% single-attempt pass rate and a 90% multi-sample pass rate, meaning it can solve the problem but its output is unstable; this can be improved via more sampling, best-of-N selection, or self-consistency voting. Model B has a 60% single-attempt pass rate and a 65% multi-sample pass rate, meaning it is stable but has a low ceiling.
Self-consistency voting works by sampling the same problem multiple times, normalizing the answers and grouping them, and taking the most frequent answer as the final output. Illustrative code:
from collections import Counterdef majority_vote(answers: list[str], judge) -> str:"""judge(a, b) checks whether two answers are equivalent, used to group "1/2" and "0.5" together"""groups: list[tuple[str, list[str]]] = []for a in answers:for rep, members in groups:if judge(a, rep):members.append(a)breakelse:groups.append((a, [a]))return max(groups, key=lambda g: len(g[1]))[0]
5.2 Paired comparison and significance testing
When comparing two models, use the same set of problems and the same prompt, record each problem’s result (correct/incorrect), and do a paired analysis.
Why must it be paired? Suppose two models have true pass rates of 70% and 65%, each tested on 100 problems. With independent samples, the standard error of the difference is about 6.6%, so even a real 5-point gap can produce a reversed ranking in a single experiment. The advantage of a paired design is that it removes “problem difficulty”—the largest source of variance—and lets you use the counts of “only A correct” and “only B correct” problems with a sign test or McNemar’s test to judge whether the difference is significant.
A practical rule of thumb you can apply directly: only consider a difference credible when the number of problems a model clearly wins in the paired comparison exceeds an empirical threshold. With small samples, prefer interval estimation (e.g., bootstrap confidence intervals) to express uncertainty rather than issuing an either/or conclusion.
import randomimport statisticsdef bootstrap_ci(values: list[float], reps: int = 2000,alpha: float = 0.05, seed: int = 0):"""Bootstrap-resample per-problem scores (0/1) and return the 95% confidence interval of the mean"""rng = random.Random(seed)n = len(values)means = []for _ in range(reps):sample = [values[rng.randrange(n)] for _ in range(n)]means.append(statistics.mean(sample))means.sort()lo = means[int(alpha / 2 * reps)]hi = means[int((1 - alpha / 2) * reps) - 1]return statistics.mean(values), lo, hi
6. Step 4: Look beyond one ranking to the cost-effectiveness frontier
Between two models with similar math ability, the choice often depends not on “who scores a few points higher” but on cost. Cost metrics to record include:
- Average output length (strongly correlated with reasoning length);
- End-to-end latency;
- Per-call cost and cost per thousand problems;
- Retry overhead after failures.
Plotting these alongside accuracy gives a cost-effectiveness frontier. Models on the frontier are the best “value” choice under a given budget. Also watch marginal returns: when the budget moves from a low tier to a high tier, does accuracy still improve materially? If the gain is limited, a higher-tier model mainly adds latency and cost.
7. Step 5: Turn evaluation into a sustainable pipeline
Evaluation is not a one-off activity. Models iterate, prompts change, and business problem types evolve. It helps to build evaluation as an automated pipeline with five stages:
- Problem management: version-store the question bank, prompt templates, reference answers, and review rules together.
- Batch inference: wrap model-calling interfaces uniformly, supporting concurrency, retries, and rate limiting.
- Automatic scoring: judge automatically per section 4.1, routing undeterminable cases into a human queue.
- Statistical aggregation: output tiered pass rates, confidence intervals, process metrics, and cost metrics.
- Regression gate: after each model or prompt change, automatically re-run and compare against the baseline; block release when significant degradation appears.
Illustrative code (unified evaluation entry point):
import asyncioasync def call_model(model: str, prompt: str, temperature: float,max_tokens: int) -> str:"""Unified model-calling interface: implement auth, request, and retry per your service's docs.This is a placeholder and does not correspond to any specific vendor's API."""raise NotImplementedErrorasync def evaluate(model: str, tasks: list[dict], repeats: int,concurrency: int = 8) -> list[dict]:sem = asyncio.Semaphore(concurrency)async def one(task: dict, k: int) -> dict:prompt = task["prompt_template"].format(question=task["question"])async with sem:raw = await call_model(model, prompt, task.get("temperature", 0.0),task.get("max_tokens", 8192))return {"task_id": task["id"], "level": task["level"],"sample": k, "raw": raw,"stats": trace_stats(raw),# Scoring logic (numeric comparison / symbolic equivalence / human review) omitted}jobs = [one(t, k) for t in tasks for k in range(repeats)]return await asyncio.gather(*jobs)
In implementation, note three points:
- Temperature: for correctness-testing, set temperature to 0 or near 0 to reduce randomness; raise it only when studying sampling stability. Within a comparison round, all models must use the same sampling parameters.
- Retain raw output: fully preserve raw outputs, latency, usage, and version numbers so every conclusion can be traced back to raw data.
- Versioned records: record the model version, prompt version, question-bank version, and scorer version for each evaluation; any change to these four affects result comparability.
If evaluation must run long-term and serve multiple teams, you can build on general cloud capabilities, for example:
- Use object storage to keep the question bank, raw outputs, and historical results;
- Use a message queue to decouple task submission from execution and handle batch-evaluation peaks;
- Use a container platform to run evaluation workloads and scale on demand;
- Use logging services and monitoring/alerting to track trends in accuracy, latency, and cost.
On Baidu AI Cloud, model services can be combined with these data, scheduling, and observability products to support the full path from evaluation to release.
8. Applying results to business: scenario-based selection
Different businesses weight math capability differently, and evaluation dimensions should follow:
- Streamlined computation (report reconciliation, metric accounting): emphasis on correctness and stability; L1 and L2 problem types dominate; prioritize models with high single-attempt pass rates and stable output formats.
- Complex analysis (business forecasting, resource planning): emphasis on difficulty ceiling and process rigor; focus on L3 and L4 problem types and bring reasoning into human spot-checks.
- Exploratory research (algorithm derivation, solution argumentation): emphasis on the exploration quality of open problems; for L5 problem types, review matters more than automatic scoring.
In engineering practice, a tiered routing strategy is recommended: simple tasks go to a low-cost model directly; when the system judges a problem to be hard, or a low-confidence result triggers secondary verification, escalate to a stronger model. Illustrative code:
def route(difficulty_level: int, low_conf: bool, tier_map: dict[int, str]) -> str:"""tier_map: capability tier -> model identifier; select a model by difficulty and confidence"""if difficulty_level <= 2 and not low_conf:return tier_map[1] # Easy: low-cost modelif difficulty_level <= 3:return tier_map[2] # Medium: standard modelreturn tier_map[3] # Hard or low confidence: high-capability model
The effect of tiered routing should also be validated with the evaluation method from section 4: check not only whether overall cost drops but also whether routing logic lowers accuracy on the hard tier.
9. Conclusion: make evaluation reproducible
Back to the opening question: how do you evaluate a model’s math capability? The answer is not who won a single conversation, but a reproducible, comparable, sustainable process. It can be summarized in five points:
- Define dimensions first: break “math capability” into six observable items—correctness, rigor, difficulty ceiling, stability, resistance to misleading premises, and cost.
- Then build the question bank: layered, private, parameter-randomized, avoiding the inflation from memorized public banks.
- Judge rigorously: answer equivalence checks, process-quality checks, and multi-reviewer or calibrated review for open problems.
- Decide statistically: multiple sampling, paired comparison, significance testing, and confidence intervals replace single-shot wins.
- Settle it in engineering: automated pipelines, versioned records, and regression gates make every model iteration evidence-based.
For users, the payoff is direct: grounded selection, controllable iteration, and computable cost. For teams, it turns math-capability evaluation from “personal experience” into an “organizational asset.” While others are still arguing over “who’s stronger,” you already have a reproducible, comparable capability profile that evolves with the business.
