Evaluating LLM Mathematical Reasoning: Methodologies and Engineering Practices

1. Mathematical Reasoning: The Benchmark for LLM Intelligence Limits

Among the many dimensions for evaluating Large Language Models (LLMs), mathematical reasoning is widely considered the most hardcore benchmark for three main reasons.

First, binary correctness leaves no room for rhetoric. Natural language tasks involve significant subjective evaluation, where strong writing can mask logical emptiness. Mathematical arguments, however, are either valid or invalid. Every step of a model’s reasoning is placed under the magnifying glass of formal verification, leaving no place for flaws to hide.

Second, it correlates strongly with engineering capabilities. High-difficulty development scenarios—such as algorithm design, complexity analysis, formal verification, and concurrency correctness arguments—are fundamentally variants of mathematical reasoning. A common observation in practice is that models with robust mathematical reasoning tend to be more reliable in root-causing elusive bugs and designing complex systems.

Third, open-ended problems distinguish “memorization” from “reasoning.” It is crucial to differentiate between two types of problems:

• Closed-ended problems: Competition problems or calculation problems with standard answers. A model might simply “recite” the answer if it encountered the exact problem in its training data; getting it right does not guarantee genuine reasoning capability.
• Open-ended problems: Research-level questions that lack a universally accepted standard solution and require original constructive proofs or exploratory solving paths.

A basic consensus in the industry is that models capable of consistently solving open-ended mathematical problems are exceptionally rare. This capability represents the upper limit of a model’s reasoning depth and currently serves as the key dividing line between top-tier models and the next generation.

2. Four Core Dimensions for Evaluating Mathematical Capabilities

Merely checking “whether the final answer is correct” is far from sufficient. A professional evaluation should cover at least the following four dimensions.

1. Tiered Problem Coverage

An effective evaluation set should span three difficulty tiers: an entry tier to verify basic calculations and formula applications; a mid tier to assess multi-step reasoning and common technique combinations; and an advanced tier (especially open-ended problems) to probe the reasoning limits. Drawing conclusions from a single difficulty level will inevitably lead to skewed results.

2. Chain of Thought (CoT) Quality

Even with identical accuracy rates, the CoTs of different models vary significantly, primarily in three aspects:

• Conciseness: Whether the reasoning path is direct or filled with ineffective detours. Some models generate extremely long and complex CoTs; while they might reach the correct conclusion, the token cost is high, and the process is difficult to review.
• Completeness: Whether critical steps are skipped and whether intermediate lemmas are explicitly proven.
• Verifiability: Whether intermediate conclusions can be independently verified and whether errors can be quickly traced back to specific steps.

3. Naturalness of Expression

Whether the reasoning process and final output “feel human”: whether terminology is accurate, whether the flow is natural, and whether it sounds like a rigid template. This dimension seems subjective, but in scenarios requiring human-machine collaboration to review reasoning processes, it directly impacts the usability of the output.

4. Stability and Variance

Sample the same problem multiple times and statistically analyze the pass rate and result variance. Some models have a decent pass rate but extremely high variance. This “slot machine” performance is unacceptable in production environments.

3. An Actionable Private Evaluation Methodology

Public leaderboards are useful for reference, but building a private evaluation set tailored to your actual workloads is more meaningful than chasing leaderboard rankings. The following workflow is recommended:

Step 1: Build a tiered question bank. Extract real problems from your business operations, add a few classic public problems, and categorize them into three difficulty tiers, maintaining 5 to 10 questions per tier.

Step 2: Strictly control variables. Use identical prompts, identical sampling parameters, and identical turns for each model to go through the questions. Avoid environment drift caused by “testing Model A today and Model B tomorrow.”

Step 3: Structured recording. Keep a traceable record for each evaluation, for example:

# Single evaluation record example
eval_id: math-open-007
tier: advanced            # entry / mid / advanced
type: open                # closed / open
model: model-A
compute_tier: high        # standard / high / max
samples: 3
results:
  - pass: true
    chain_style: concise
    note: "Direct construction path, intermediate lemmas verifiable"
  - pass: false
    chain_style: verbose
    note: "Circular reasoning occurred at step two, leading to total collapse"

Step 4: Timestamp your evaluations. Model versions iterate rapidly. Any conclusion stating “Model X is better than Model Y” should be accompanied by the evaluation date. After a major version update, old conclusions must be re-verified. Furthermore, a single model often operates across different compute tiers (e.g., standard, high, max). The tier must be noted during comparisons; otherwise, the conclusion is meaningless.

A final piece of practical advice: Do not draw conclusions without actually running the tests. Performance on advanced mathematical problems can severely diverge from performance on mid-to-low tier tasks. Guessing high-end task capabilities based on daily chat impressions will result in massive bias.

4. Mapping Mathematical Capabilities to Programming

For developers, evaluating mathematical capabilities ultimately serves programming model selection. Empirically, the mapping looks like this:

• Models whose mathematical reasoning reaches the first tier are generally competent in algorithm design, complexity optimization, and root-cause analysis of elusive bugs.
• Next-generation models are sufficient for routine CRUD operations, scaffolding, and unit test completion, and they often respond faster at a lower cost.
• The true differentiator is “long-chain tasks”: complex refactoring and system design that require dozens of continuous reasoning steps without losing focus. This demands the highest level of CoT completeness and stability.

Therefore, do not blindly trust a single leaderboard when selecting a model. Instead, match models to task difficulty tiers, reserving high-end capabilities for scenarios that genuinely require them.

5. Understanding Quota Mechanisms: Maximizing Compute Within the Rules

An unavoidable topic when using AI coding tools is quota management. Mainstream industry solutions generally adopt a “dual-layer quota” design:

• Short-period rolling limits: For example, session limits calculated on a 5-hour rolling window.
• Long-period total limits: For example, weekly limits calculated on a 7-day rolling window.

The intersection of these two reveals an easily overlooked reality: The number of complete short-period windows that can actually fit within a long-period limit is constrained. Users have observed that in some implementations, a single long period can only support about three complete high-intensity short-period sessions. Calculated out, the weekly full-load available working hours are quite limited. This is essentially a vendor trade-off between compute costs, service availability, and subscription pricing—a common industry practice. It shouldn’t trigger emotional complaints, but it must be factored into your usage strategy.

Based on the practical constraints of dual-layer quotas, several useful strategies can be summarized:

1. Task Batching: Submit clear, structured, long-running tasks at the very beginning of a quota window. Let the agent progress autonomously within the window to avoid wasting capacity on fragmented conversations.
2. Water Level Management: Check remaining quota before starting high-consumption tasks. Schedule heavy lifting at the beginning of the window and leave light tasks for the end.
3. Tier Grading: Reserve high compute tiers for cracking hard problems. Fall back to standard tiers for daily tasks.

6. Multi-Agent Division of Labor: Offloading Mechanical Work to Lightweight Models

When quotas are restricted, the most immediate optimization is a multi-agent division of labor: the main model handles planning, decomposition, and critical decision-making, while dispatching high-certainty, low-creativity tasks to lightweight sub-models.

A typical boundary of labor looks like this:

Task Type	Executor	Reason
Architecture design, core algorithms	Main Model	Requires deep reasoning
Code review, bug hunting	Sub-model	Pattern-based, clear rules
Adding small features, writing tests	Sub-model	High certainty, verifiable
Final merge and review	Main Model	Requires a global perspective

Division of labor rules should be written into project-level instruction files so that agents automatically follow them in every session. An example configuration is as follows:

# agent-guide.md -- Project Collaboration Instructions Example
## Division of Labor Principles
- Main Model: Task decomposition, solution design, core logic implementation, final review
- Sub-model: Handles mechanical, high-certainty tasks
## Sub-model Task List
1. Code Review: Null references, boundary conditions, resource leaks
2. Bug Localization: Narrow down suspect scope based on error stacks
3. Small Features: Add utility functions, supplement unit tests
## Acceptance Criteria
- Sub-model outputs must include runnable regression tests
- Any changes involving core logic must be rolled back to the main model for review

Developer practices show that at the beginning of an almost untouched cycle, opening four coding sessions in one morning (three on high compute tier, one on max tier), while pointing sub-agents to a lightweight model and dividing labor as described above, consumes only about 35% of the short-period limit and 24% of the long-period limit after three hours of operation. Compared to the “main model does everything” approach, the quota lifespan can be significantly extended.

It must be emphasized: The output of sub-models must be integrated into the review chain. Lightweight models perform adequately on simple tasks, but errors accumulate silently. If these errors flow into core logic, the troubleshooting cost will far exceed the quota saved. The correct posture is “sub-models do the work, the main model guards the gate, and tests serve as the safety net.”

7. Conclusion

Returning to the original question: How should a new model be evaluated? More valuable than endless debates over “who is better than whom” is establishing your own evaluation system—tiered question banks, controlled variables, structured recording, and timestamped results. Combine this with a clear understanding of quota mechanisms and multi-agent engineering practices to ensure every drop of compute is spent where it matters most.

The pace of model iteration will only accelerate, and today’s rankings may be rewritten tomorrow. What will never become outdated is the methodology of scientific evaluation and the engineering habit of meticulous resource management. Regardless of which model service you use—including large model platforms like Baidu AI Cloud that provide model access, invocation, and monitoring capabilities—solidifying the three practices of evaluation, scheduling, and review will ensure you maintain the initiative in a rapidly evolving model ecosystem.