Evaluating LLM Math Reasoning: Methodologies and API Quota Management Best Practices

I. Why Math Reasoning Deserves Dedicated Evaluation

Mathematical reasoning is one of the most revealing dimensions for assessing the “deep reasoning” capabilities of Large Language Models (LLMs). Unlike open-ended chatting or text summarization, math problems offer three inherent advantages:

1. Verifiable Results: Answers are strictly correct or incorrect, enabling automated scoring and minimizing subjective bias during evaluation.
2. Long Reasoning Chains: A comprehensive math problem often requires more than ten derivation steps. A single error in any step leads to final failure, effectively exposing a model’s stability weaknesses.
3. Scalable Difficulty: From basic arithmetic to competition-level problems and open-ended research questions, the difficulty gradient is clear, making it easier to map a model’s capability boundaries.

Because of these traits, developers often benchmark new models directly against each other using math problems. However, without a solid methodology, these informal comparisons often yield unreliable conclusions.

II. Three Common Evaluation Pitfalls

1. Blindly Trusting Leaderboards: Public benchmark scores are only good for initial screening. Benchmark datasets face the risk of training data contamination, meaning a model might have “seen” similar problems during pre-training. Furthermore, leaderboard scores are typically achieved under specific decoding parameters and prompt templates, creating a systematic gap from real-world business environments.
2. Single-Point Comparisons: Testing a model with just one or two problems leads to high variance. Math reasoning is highly sensitive to sampling randomness—the same model might solve a hard problem correctly at one temperature setting but fail at another. A robust approach is to sample the same problem 8 to 16 times and calculate the pass rate, rather than drawing conclusions from a single attempt.
3. Only Checking Final Answers: Two models might have a similar final answer accuracy, but that does not mean their capabilities are equal. One might converge in three steps, while the other takes twenty redundant reasoning steps to barely reach the conclusion. The former typically generalizes better on out-of-distribution problems, while the latter’s failure rate on high-difficulty problems amplifies as the step count increases.

III. Tiered Task Design: Mapping Model Boundaries

An actionable math capability evaluation should organize the problem set into difficulty tiers:

Tier 1: Basic Arithmetic and Symbolic Manipulation. Tests computational stability, including multi-digit operations, fraction simplification, and algebraic transformations. Point loss here is usually an engineering issue—such as numerical precision or tokenization methods—rather than a lack of “intelligence.”

Tier 2: Competition-Level Structured Problems. These have standard answers and serve as the core evaluation set. It is recommended to track scores across sub-categories like algebra, geometry, number theory, and combinatorics to create a capability profile. Two models with similar total scores might have vastly different sub-category distributions, corresponding to entirely different applicable scenarios.

Tier 3: Open-Ended Math Problems. These lack ready-made answers and require multi-step exploration or trial and error. Models that can handle this tier reliably are rare; it acts as a watershed between “top-tier reasoning models” and “excellent models.” These cannot be auto-scored and require manual review of key reasoning nodes. The evaluation cost is high, but the informational value is maximal.

IV. Chain-of-Thought Quality: Length Does Not Equal Capability

A commonly overlooked phenomenon in practice is that Chain-of-Thought (CoT) length does not necessarily correlate positively with capability.

• Long-chain models: Feature longer reasoning chains, more branches, and frequent self-correction. While this “cautious” style isn’t necessarily detrimental on medium-difficulty problems, every step in a super-long chain is a potential failure point. More redundant steps mean a higher cumulative probability of error.
• Concise models: Directly address key lemmas, offering better readability and auditability. Human reviewers can quickly locate reasoning flaws, which is crucial for troubleshooting model shortcomings.
• Note: It is vital to distinguish between “concise” and “skipping steps.” The former omits无效探索 (ineffective exploration); the latter omits necessary proofs. Correct answers derived from skipped steps should be considered unqualified in rigorous scenarios.

Recommended metrics for measurement:

For the same problem set, track three metrics:
- pass@k: The proportion of at least one pass in k samples
- Average effective steps: The mean of reasoning steps in correct samples
- Redundancy rate: The proportion of steps that can be deleted without affecting the conclusion (manually sampled and labeled)

V. Anchor Model Positioning: Turning Subjective Feelings into a Discussable Coordinate System

When evaluating a new model, a practical technique is to select two or three “anchor models” with known capability levels. Have the evaluated model compete on the same problem set as the anchors to determine its relative positioning (e.g., “between Anchor A and Anchor B”). This method offers three benefits:

• Reproducible conclusions: Others can independently verify the results following the same process.
• More precise expression: Conveys much more information than vague descriptions like “very strong” or “average.”
• Resistant to leaderboard drift: Even if public benchmark scores are distorted, the relative ranking remains valuable.

Note that relative positioning conclusions should include a confidence interval. With small sample sizes or clustered problem difficulties, the ranking of two models might completely flip. In such cases, a more rigorous statement is “no significant difference was observed between the two on the current problem set,” rather than forcing a hierarchy.

VI. Quota Mechanism Design: The Conflict Between Short Windows and Weekly Quotas

Math evaluation is rarely completed with a single API call. A 200-problem set sampled 8 times per problem amounts to 1,600 long reasoning calls. A single open-ended problem inference might span thousands of tokens and take minutes. This inevitably collides with the API quota design of model services.

A typical poor user experience: A user checks their quota in the evening, sees 60% remaining, and launches a long-running task. Late at night, a short-window limit (e.g., 5-hour level) is triggered, pausing the task. By the next morning, the weekly limit has also been triggered—yet the total weekly quota equates to only three short windows. For long-task users, such “nested limits” almost equate to permanent task interruption.

From a mechanism design perspective, four principles can be abstracted:

1. The outer layer of nested limits should not exhaust before the inner layer. The weekly quota should be significantly larger than “short-window quota × number of windows.” Otherwise, the outer layer frequently becomes the actual constraint, causing continuous user frustration.
2. Limits should be priced in consumable units, not wall-clock time. Metering by token count or request count is fairer than “usage duration”—slow inference is not the user’s fault.
3. Long tasks require clear pause and resume semantics. When paused, the context breakpoint should be saved, resuming from that point rather than starting over. The caller should also implement idempotent designs to fulfill this contract.
4. Quota status must be observable. Clients should be able to query the remaining quota for the current period and the limit reset time, allowing proactive task scheduling rather than passively hitting walls and deducing rules from error messages.

VII. Caller-side Strategies: Engineering the Evaluation Task

Given uncontrollable server-side quotas, callers can implement the following:

1. Batching and Checkpoint Resumption. Split the problem set into small batches and save results to disk per batch. Upon hitting a limit, record the breakpoint and auto-resume after the window resets.

# Pseudo-code: Batch evaluation scheduling with checkpoints
def run_eval(model, problems, samples_per_problem=8):
    state = load_checkpoint()                     # Load checkpoint
    pending = [p for p in problems if p.id not in state.done]
    for p in pending:
        while True:
            try:
                answer = model.sample(p.prompt, n=samples_per_problem)
                state.record(p.id, answer)
                state.save()                       # Save per problem to ensure checkpoint granularity
                break
            except RateLimitError as e:
                wait_until(e.reset_time)           # Wait for reset based on server-side prompt

1. Result Caching and Deduplication. Cache and reuse results for the same model, prompt, and parameter combination to avoid wasting precious quota.
2. Elastic Scheduling. To accelerate evaluation, split tasks across local compute environments for parallel execution and aggregate scoring. If utilizing cloud compute—such as creating on-demand GPU compute instances on Baidu AI Cloud to handle inference—you must design task queues and instance auto-scaling strategies to avoid cost waste from idle compute resources.
3. Control Variables. During parallel execution, strictly fix decoding parameters and prompt templates. Otherwise, differences across execution environments will directly contaminate evaluation conclusions.

VIII. Conclusion

Evaluating model capability is fundamentally a measurement problem: the problem set is the ruler, the sampling count is the measurement frequency, and CoT review is the error analysis. Transforming the intuition that “Model X is stronger” into a standardized process comprising tiered problem sets, multiple samplings, anchor comparisons, and confidence labeling gives conclusions real discussable value. Similarly, quota mechanism design should stem from the actual workflows of long-task users, making nested limits, checkpoint recovery, and quota observability a shared contract between server and caller. Measuring accurately and running to completion are the two cornerstones of transitioning large-scale model evaluation from personal experience to engineering practice.