How to Scientifically Evaluate the Math Capabilities of Large Language Models

    When users say one model "feels stronger at math" than another, or that a certain model produces chains of thought closer to human reasoning, they are drawing on real-world experience—but subjective impressions have two persistent weaknesses: the conditions are opaque, and the results cannot be reproduced. Change the prompt, adjust the temperature, or drop a piece of context, and the same model can behave noticeably differently. This article argues for turning intuition into measurable, reproducible experiments. It lays out a complete evaluation framework covering capability layering, problem-set design, metrics, engineering practices, and common pitfalls—so you can replace "who feels stronger" with conclusions you can defend and revisit.

    1. The Evaluation Problem: From “Gut Feeling” to Reproducible Conclusions

    Judgments like “this new version is roughly somewhere between two other versions at math” or “this model produces chains of thought closer to human reasoning” usually come from people using the models in real tasks. They carry real reference value. But “feeling stronger” has two problems: the conditions are not transparent, and the conclusions cannot be reproduced.

    Change the prompt set, adjust the temperature once, or supply less context, and the same model can perform noticeably differently. So a conclusion like “this version sits between the other two” should be treated as a hypothesis for model selection, not as final evidence. The more reasonable path is to turn subjective impressions into measurable metrics, and turn side-by-side comparisons into reproducible experiments. This article follows that idea and lays out a complete evaluation framework, from capability layering and problem design through to engineering.

    2. First, Layer “Math Capability”: Different Difficulty Tests Different Skills

    Scoring a math model with a single aggregate number tends to hide the real differences. A wrong answer on a competition problem might come from a calculation slip, a jump in reasoning, choosing the wrong solution direction, or even giving up midway because the model mistakenly believes “the problem setter made an error.” A more reasonable approach is to layer first, then measure layer by layer.

    Layer Typical Tasks Core Capability How to Judge
    L1 Calculation and formal transformation Numerical operations, equation solving, algebraic simplification Calculation accuracy and format adherence Automated
    L2 Multi-step reasoning Classic competition problems, word problems Step coherence and strategy selection Automated or rule-based
    L3 Requiring construction and insight Proofs, construction problems Strategic innovation and self-correction Human scoring
    L4 Open math problems Hard problems with no known answer or solution path today Long-range exploration, verification, and self-critique Human review plus external verification

    A few caveats. There are no hard boundaries between the layers, and the same problem can land in different layers for different models. The point of layering is that companies using models for data accounting or report validation should focus on L1 and L2; those using models for research support must look at L3 and L4. After layering, you should also assign an independent weight to each layer, so that a large volume of easy problems cannot inflate the total score.

    L4 is where this article draws the most observations. Some users note that a few models are among the minority that can handle open math problems, while certain models whose writing style is closer to human are actually slightly weaker here. This kind of information is valuable for understanding a model’s “capability ceiling,” but it depends on two things. First, the evaluator must be able to judge whether an answer is correct, or cross-check it with verifiable means. Second, the openness of the problem must not be misread. Real-world math problems fall into two categories:

    • Open-answer problems: for example, “find all positive integer solutions that satisfy the condition.” The conclusion is unique, but the process is open, and the model must construct its own solution method.
    • Research-level open problems: genuinely unsolved math problems with no known answer.

    Both require the model to build a solution path from scratch, but the second is far more expensive to verify and needs a professional background or formal tools to back it up.

    3. Building the Evaluation Set: Stratified Sampling, Contamination Control, and Scoring Criteria

    Once the layers are defined, the quality of the evaluation set directly determines how much you can trust the conclusions. Four points deserve attention:

    1. Stratified sampling: Each layer needs enough problems to support statistical conclusions, and the sources should span multiple domains. If the sample contains many near-duplicate problem types, accuracy gets inflated by a single pattern.
    2. Strict scoring criteria: In competition math, writing only the final answer without the process is a common mistake and a frequent point of loss. The criteria should make clear “whether key derivation steps are given” and “whether the answer is decidable.” Math answers are usually unique, so automated judging is relatively easy; the difficulty lies in recognizing equivalent transformations and judging proof rigor.
    3. Contamination control: Evaluation problems must not be taken verbatim from public leaderboards, or you will easily get inflated scores. Common measures include:
      • Building a private problem bank;
      • Varying the numbers, conditions, or the way the question is asked based on original problems;
      • Rotating problems on a schedule and reserving a set of “held-out” problems for final verification.
    4. Version and prompt management: Different versions of the same model need to be evaluated separately. Prompt templates, temperature, maximum generation length, and similar parameters should be fixed and recorded, otherwise cross-model comparison loses its validity.

    You also need to be explicit about which dimensions the evaluation covers: math capability, writing readability, cost, or all three.

    4. Metric Design: Beyond Accuracy—Measure Output per Token

    Reporting only “accuracy” is not enough. A model that gets a problem right on one sample can fail once you change the temperature; some models generate several times the necessary length of scratch work per problem, and token consumption is noticeably higher. At a minimum, set up three groups of metrics:

    • Accuracy: the pass rate per layer under a fixed number of samples, such as pass@1.
    • Stability: the degree of fluctuation across multiple samples of the same problem.
    • Cost efficiency: average token consumption, and the “number of tokens needed to obtain one correct answer.”

    Sample code:

    1. import json
    2. import statistics
    3. from collections import defaultdict
    4. # Minimal record structure per sample (illustrative)
    5. # {
    6. # "model": "A", "task_id": "algebra_04", "level": "L2",
    7. # "sample_id": 1, "response": "...", "is_correct": True,
    8. # "prompt_tokens": 820, "completion_tokens": 2140
    9. # }
    10. def build_report(results, confidence=0.95):
    11. """Aggregate by model and difficulty layer, output accuracy and cost efficiency"""
    12. by_level = defaultdict(list)
    13. for r in results:
    14. by_level[(r["model"], r["level"])].append(r)
    15. report = {}
    16. for (model, level), items in by_level.items():
    17. n = len(items)
    18. correct = sum(r["is_correct"] for r in items)
    19. acc = correct / n
    20. # 95% confidence interval (normal approximation), to avoid being
    21. # misled by small-sample fluctuation
    22. se = statistics.pstdev([r["is_correct"] for r in items]) / n ** 0.5
    23. half = 1.96 * se
    24. # Total consumption = input tokens + output tokens
    25. tokens = [r["prompt_tokens"] + r["completion_tokens"] for r in items]
    26. avg_tokens = statistics.mean(tokens)
    27. report.setdefault(model, {})[level] = {
    28. "samples": n,
    29. "accuracy": round(acc, 3),
    30. "ci95": [round(acc - half, 3), round(acc + half, 3)],
    31. "avg_total_tokens": round(avg_tokens, 1),
    32. "tokens_per_correct": round(avg_tokens / acc, 1) if acc else None,
    33. }
    34. print(json.dumps(report, ensure_ascii=False, indent=2))
    35. return report

    This script has three key points:

    • Stratified statistics: Difficulty layers are computed separately, so easy problems cannot dilute the performance on hard ones.
    • Confidence intervals: Accuracy fluctuates a lot under small samples, and directly comparing two percentages can easily lead to the wrong conclusion.
    • Cost per correct answer: It reveals the “high accuracy but enormous consumption” case, which matters especially for cost-sensitive companies.

    For L3 and L4, you can also introduce a majority-voting strategy: sample the same problem multiple times and aggregate by answer agreement. High agreement does not equal correctness, but it does reflect output stability. If multiple samples give different answers, it usually means the problem’s difficulty already exceeds the model’s current range of stable capability.

    5. Open Math Problems vs. Models with “Human-Like Writing Style”

    Some users have pointed out an interesting phenomenon: certain models can handle open math problems, while others whose writing style is closer to human cannot. This suggests that in math evaluation, “whether it can” and “how well it expresses itself” are two relatively independent dimensions.

    • Expression level: You can examine structure and organization, whether the derivation steps are complete, and whether there is redundant repetition. Concise does not mean weak, and verbose does not mean rigorous. When scoring, treat “is the chain of thought too long” and “does it repeatedly second-guess itself” as quality items, rather than assuming that longer is always better.
    • Math level: You can build an “open-ended problem set” that asks the model to give a solution direction, key lemmas, and a verification approach. The evaluation focus is “whether it proposes verifiable intermediate steps,” not whether it ultimately produces a perfect answer.

    6. Engineering the Evaluation: Making It a Repeatable Experiment

    Whether evaluation conclusions are trustworthy depends largely on whether the process is disciplined. At a minimum:

    1. Unified interface and parameters: All models go through the same calling wrapper, with prompt templates, temperature, and maximum generation length fixed, and parameters archived along with the results.
    2. Multiple samples: Sample each problem at least 3 to 5 times, and record every output and its judgment, rather than drawing conclusions from a single result.
    3. Result archiving: Raw responses, token statistics, and scoring records are all persisted to disk, making review and retrospection easy.
    4. Dynamic model version management: Models iterate fast, so the evaluation set should support version numbers and a configurable model list, making it easy to run quick regressions after a model upgrade.
    5. Rely on cloud capabilities: You can use Baidu AI Cloud object storage to save raw responses, a logging service to record the call chain, and monitoring and alerting to track the duration and failure rate of evaluation tasks.

    For the regression flow, you can fix the parameters, run the script, and then compare the reports:

    1. # Evaluation entry (illustrative): problem file, model list, and sample count as inputs
    2. python evaluate_math.py \
    3. --tasks ./tasks_layered.jsonl \
    4. --models ./models.txt \
    5. --samples 5 \
    6. --temperature 0.7 \
    7. --report ./reports/round_001.json

    Fixing the parameters and sample count is a prerequisite for comparing across rounds. If the model list changes between rounds, the report should explicitly mark the changes.

    After the evaluation is done, you can build a “selection matrix”: the horizontal axis is difficulty layers and cost constraints, and the vertical axis is each model’s performance. Every cell in the matrix should carry a sample count and a confidence interval, avoiding single-point conclusions like “who is strongest.” The final recommendations are:

    • Core capability: primarily the accuracy and cost efficiency of L1 and L2;
    • High-difficulty tasks: primarily the exploration capability and verification quality of L3 and L4;
    • Writing needs: evaluate expression quality separately, and do not conflate it with math accuracy.

    7. Common Pitfalls

    • Overgeneralizing: Drawing the conclusion “a certain model is bad at math” from a small number of samples.
    • Only looking at the answer: Ignoring process scoring, so that “guessing right” gets counted as “knowing how to do it.”
    • Ignoring cost: A model with high accuracy but extremely high token consumption may not be usable in production.
    • Problem leakage: Public leaderboard problems entering the private evaluation set, causing inflated scores.
    • Inconsistent parameters: Using different prompts or temperatures for different models, which makes the comparison meaningless.
    • Capability labeling: Conflating “good at math” with “good at writing,” leading to poor selection decisions.

    8. Conclusion

    Evaluating math capability is essentially running a reproducible experiment: define tasks by layer, manage scoring strictly, measure multiple metrics at once, and codify the process through engineering. The contrast between open math problems and models with human-like writing styles reminds us that math capability and expression style are two dimensions that cannot substitute for each other. For developers, turning “feeling stronger” into conclusions that are measurable, reproducible, and regression-testable is the right starting point for selecting large language models.