Evaluating Mathematical Reasoning in Large Language Models: Dimensions, Methods, and Practice

1. Why Mathematical Reasoning Is the Litmus Test for LLMs

Among the many capability dimensions used to evaluate large language models, mathematical reasoning is widely regarded as a reliable indicator of real model quality for three reasons.

First, strong verifiability. The final answer and the derivation process for a math problem can be checked step by step. Evaluation does not depend on subjective scoring, and conclusions are not influenced by how polished or persuasive the model sounds. Compared with writing tasks, where scores often vary from reviewer to reviewer, math evaluation is more objective and more reproducible.

Second, naturally well-defined difficulty levels. From basic arithmetic and standard competition problems to open-ended math questions without a standard answer, the difficulty hierarchy is clear. This makes it possible to separate models into distinct capability levels.

Third, high sensitivity to genuine reasoning. Pattern memorization can help a model achieve good scores on closed test sets. However, when faced with open-ended math problems—where the model must propose a lemma, construct a counterexample, or build a new proof path—it has to demonstrate real reasoning organization. A widely shared observation in the field is that very few models can consistently handle open math problems, making this metric a dividing line between leading models and the rest of the first tier.

Therefore, a sound evaluation framework should treat open-ended problem solving as the highest-weight dimension, rather than focusing only on pass rates on standard benchmarks.

2. Evaluation Discipline: Only Speak to What You Have Verified Yourself

The most common source of noise in model comparisons comes from secondhand conclusions: repeating other people’s test results, citing leaderboards that may be contaminated by training data, or making absolute judgments based on one or two interactions. To arrive at trustworthy personal conclusions, three disciplines are recommended.

1. Only evaluate models and tasks you have run yourself. For anything you have not tested directly, explicitly mark it as “no comment.” An honest blank is more informative than a vague guess.
2. Separate conclusions for closed problems from those for open problems. The same model may perform completely differently on these two types of tasks, and conclusions should not be mixed or transferred.
3. Keep evaluation records. Record the prompts, sampling temperature, test date, and model version. Models iterate quickly, and a conclusion from three months ago may already be outdated. Keeping records allows evaluations from different periods to be compared.

3. Four Core Evaluation Dimensions

3.1 Upper Bound: Open-Ended Problem Solving

Closed problems test whether a model “knows how.” Open problems test “how far it can go.” For evaluation, prepare several open-ended tasks, such as giving a conjecture with an incomplete proof and observing whether the model can propose a feasible attack strategy, or asking it to construct a counterexample that satisfies specific constraints. The focus is not whether the answer is correct, but whether the reasoning is internally consistent, whether the model can identify gaps in its own argument, and whether it can effectively self-correct after errors are pointed out.

3.2 Process: Chain-of-Thought Quality and Cost

Two models may achieve the same accuracy while differing greatly in chain-of-thought quality. Evaluation should focus on three sub-items:

• Conciseness: Long chain-of-thought does not equal strong capability. Some models produce extremely long reasoning with many branches without necessarily reaching better conclusions, while the cost of reading and debugging rises significantly. Models with concise and clear reasoning chains are generally preferred for engineering usability.
• Self-correction: Whether the model can detect and fix its own mistakes during reasoning is a key signal that distinguishes “reciting answers” from “real reasoning.”
• Readability: Whether the reasoning steps have a clear logical structure that makes human review easier.

3.3 Delivery: Naturalness of the Final Output

Even when the reasoning is correct, the presentation quality of the final answer directly affects usability: whether the structure is clear, whether the wording is natural, and whether the output can be used directly without further editing. In scenarios where model output is delivered to non-technical readers, the weight of this dimension should be increased.

3.4 Stability: Variance Across Multiple Samples

A single run may be affected by sampling randomness. By sampling the same set of tasks multiple times and calculating the variance in pass rates, you can determine whether the performance reflects a stable level or an occasional burst.

Combining these dimensions yields the following reference weight table:

Dimension	Reference Weight	Key Observations
Open problem solving	40%	Internally consistent reasoning, verifiable constructions, self-correction
Chain-of-thought quality	25%	Conciseness, mid-course correction, logical structure
Stability	20%	Pass-rate variance across multiple samples
Output naturalness	15%	Clear structure, ready to use directly

Weights should be adjusted according to the use case. For algorithm research, increase the weight of “upper bound.” For user-facing products, increase the weight of “delivery.”

4. Layered Positioning: Using Reference Models Instead of Absolute Scores

Assigning absolute scores to models is both difficult and unstable. A more practical approach is layered positioning: describe a new model on your own anchor task set as “falling between two known reference models.” For example, “the new model’s performance falls between reference model A and reference model B that I am familiar with, and is closer to A.” Such conclusions are less precise than scores, but more robust and more transferable.

The steps are as follows:

1. Build an anchor set: Select 10–20 problems covering different difficulty levels, and run them repeatedly with the models you know best to form “anchor profiles.”
2. Same-problem comparison: Test every new model on the same anchor set to ensure comparability.
3. Output positioning, not ranking: Write conclusions as “between X and Y, closer to X,” and attach comparison records for key problems.

Using this method, models on the market can be roughly divided into capability tiers. At the top are the few models that can handle open math problems. Below them are models that reliably pass high-difficulty competition problems but show limited thinking on open problems. Further down are models that handle standard problems consistently but clearly depend on prompting for harder ones. Tier divisions are dynamic and should be recalibrated after every major version update.

5. Engineering Practices for Long-Running Evaluations: Quotas, Rate Limits, and Checkpointing

When evaluating advanced math problems, a single task can consume a large number of inference tokens. Combined with batch execution across many tasks, this can easily hit platform rate limits and quota mechanisms. A typical scenario: you set up a batch job overnight, only to discover several hours later that it hit an hourly rate limit. After the system recovers, the job continues, but then triggers a longer-period usage cap, forcing the entire batch to stop.

These lessons point to three engineering practices:

First, estimate the budget before starting. Based on historical data, estimate the average token consumption and duration per task, multiply by the number of tasks, and compare against the platform’s rate-limit windows to judge whether a batch can complete within one window.

Second, save checkpoints. Persist state immediately after each task completes so that any interruption can be resumed from the checkpoint rather than restarting the entire batch.

Third, decompose tasks and stagger scheduling. Split large batches into smaller ones and distribute them across multiple time windows. Prioritize critical tasks and downgrade postponable ones to a queue.

A minimal checkpoint-and-resume framework is shown below as pseudocode. The interfaces are illustrative only:

import json
from pathlib import Path
CHECKPOINT = Path("eval_state.json")
def load_state():
    if CHECKPOINT.exists():
        return json.loads(CHECKPOINT.read_text(encoding="utf-8"))
    return {"done": [], "results": {}}
def run_eval(tasks, quota):
    state = load_state()
    for task in tasks:
        if task["id"] in state["done"]:
            continue                      # already completed, skip
        if quota.remaining() <= 0:
            break                         # budget exhausted, wait for next window
        answer = call_model(task.prompt)  # model call (illustrative)
        state["results"][task["id"]] = grade(task, answer)
        state["done"].append(task["id"])
        CHECKPOINT.write_text(            # persist immediately
            json.dumps(state, ensure_ascii=False),
            encoding="utf-8",
        )
    return state

If evaluation tasks are hosted in the cloud—for example, deployed on a container service such as Baidu AI Cloud—you can also use logging services to collect usage records for each call. Set quota watermarks through monitoring and alerting so that tasks are automatically alerted or paused when approaching thresholds, turning “waking up to discover you hit a limit” into “receiving advance notice.”

6. Common Evaluation Pitfalls

• Drawing conclusions from a single problem: Performance on one problem does not represent a model’s level. At minimum, draw statistical conclusions on a small task set.
• Ignoring prompt sensitivity: The same task with a different prompt may produce very different results. Fix the prompt template during evaluation, or try 2–3 prompt variants per task and take the best result.
• Data contamination: Public problem sets may already be included in training data, leading to inflated scores. When possible, create or modify problems yourself—change numbers, conditions, or wording.
• Survivorship bias: Sharing only success stories systematically overestimates model capability. Failure cases are equally worth recording.

7. Conclusion: Evaluation Is a Continuously Accumulating Personal Infrastructure

There is no one-time leaderboard for LLM capability evaluation. What matters is a continuously maintained personal evaluation system: a stable anchor task set, a discipline of “only speaking to what you have tested yourself,” and a batch task framework that supports checkpointing and resumption. When a new model version is released, what you need is not to start from scratch, but to perform an incremental calibration on top of your existing system. This is precisely the core value of a personal evaluation system compared with public leaderboards: the conclusions fit your own use cases better and become more accurate over time.