Evaluating and Selecting LLMs for Mathematical Reasoning

Why Mathematical Reasoning Is a Key Test of Model Capability

When evaluating whether a large language model possesses “true reasoning ability,” math problems tend to be far more discriminative than everyday conversation, summarization, or code completion. The reasons are as follows:

1. Answers are highly verifiable: The final result is often unique and can be objectively checked through computation or proof.
2. High demands on the reasoning chain: Many open math problems have no fixed template. The model must organize intermediate steps itself, introduce auxiliary constructions, reason case by case, and even try multiple paths.
3. Sensitivity to hallucination: Once a logical leap or factual error appears in mathematical reasoning, it usually leads to a completely wrong conclusion that is easy to detect.
4. Differentiation across model tiers: Entry-level models may only handle standard problems, while advanced models can demonstrate stronger exploratory ability on open, competition-level, or research-oriented problems.

As a result, many technical teams treat mathematical reasoning as a standalone core metric when grading model capability, rather than relying solely on general conversational performance.

Open vs. Closed Math Problems

When assessing mathematical ability, the first step is to distinguish two categories of problems:

1. Closed Math Problems

These problems typically have well-defined concepts, standard solution methods, and determinate answers. Examples include:

• Solving basic algebraic equations
• Standard calculus computations
• Routine geometric proofs
• Counting problems in discrete mathematics

Such tasks reflect how well a model has mastered existing mathematical knowledge, but they are insufficient to fully demonstrate exploratory reasoning ability. Many models fine-tuned on large-scale instructions can perform well on closed problems.

2. Open Math Problems

Open math problems are closer to real research or competition scenarios and may include:

• Constructive problems with no specified solution path
• Problems that require trying multiple strategies to find a breakthrough
• Problems that require discovering hidden conditions or making reasonable assumptions
• Problems that are not fully formalized and require modeling before solving

These problems place higher demands on long-term planning, strategy selection, and error correction. A typical pattern is that some models score highly on standard problems but suffer broken reasoning chains when facing problems that require a flash of insight or multi-step trial and error.

Based on current industry discussion, models that can reliably handle open math problems remain a minority, which is why this capability is often used as a criterion for high-end models.

Key Technical Factors Affecting Mathematical Reasoning

1. Chain-of-Thought Length and Complexity

Chain of Thought (CoT) is the intermediate reasoning process a model unfolds before giving a final answer. Different models differ noticeably in CoT style:

• Some models prefer concise, direct reasoning paths with few steps but high information density per step.
• Others generate very long reasoning chains containing extensive probing, branching, and verification.

A longer chain does not necessarily mean stronger reasoning. Overly long chains can bring two problems:

• Increased risk of error accumulation: Each reasoning step may introduce error, and the longer the chain, the more easily overall reliability degrades.
• Higher computational cost: For engineering applications, long reasoning means higher latency and resource consumption.

Yet when handling highly complex open problems, appropriate “exploratory long reasoning” is often necessary. The ideal case is a model that can dynamically adjust its reasoning strategy according to problem difficulty, rather than applying the same unfolding pattern to every problem.

2. Self-Verification and Error Correction

High-performing mathematical reasoning models usually exhibit clear self-verification behavior. For example:

• After producing an intermediate result, proactively checking whether it satisfies the problem’s conditions
• Testing suspect steps with counterexamples
• Backtracking and trying other paths when a derived result conflicts with intuition

This ability is reflected not only in the final answer but also in whether the reasoning process shows a “pause—check—correct” pattern. During evaluation, observe whether the model can stop in time when it encounters an obvious contradiction, rather than persisting down a wrong path.

3. Symbolic Computation and Formalization Ability

Although large models typically do not rely on external symbolic computation engines, their internal understanding of mathematical symbols, formula structures, and formalization rules directly affects reasoning quality. Specific manifestations include:

• Whether the model can correctly distinguish “necessary conditions” from “sufficient conditions”
• Whether it can correctly handle quantifier order
• Whether it can identify implicit mathematical structures such as symmetry, monotonicity, periodicity, and invariants
• Whether it can introduce auxiliary functions, auxiliary lines, or equivalent transformations at the right time

These abilities come partly from the coverage quality of mathematical corpora during pretraining and partly from optimization of reasoning strategies during subsequent reinforcement learning.

4. Generalization Across Problem Types

Some models have seen large numbers of competition problems during training and therefore perform well on similarly styled problems; but when the phrasing changes or the background shifts, performance may drop noticeably. Evaluation should focus on whether the model truly understands the underlying mathematical structure rather than merely matching problem styles.

A practical test method is to alter the variable symbols, numerical values, background narrative, or even the language of the same problem and observe whether model performance remains stable. Significant fluctuation suggests a degree of surface-level memorization dependence.

Common Approaches to Model Tiering

Without referring to specific brands, current large models can be roughly divided into the following tiers in terms of mathematical reasoning:

Tier 1: Models That Reliably Handle Open Math Problems

These models stand out in complex reasoning, long exploration, and multi-path attempts. They not only provide answers but also exhibit behavior resembling “strategy search” during reasoning. They are typically used for:

• Advanced mathematics research assistance
• Competition-level problem solving
• Complex modeling and derivation
• Scenarios requiring highly reliable reasoning conclusions

Such models are few in number and tend to consume significant computational resources, making them suitable for on-demand invocation in critical tasks.

Tier 2: Strong Logical Reasoning but Weaker Exploration

These models perform excellently on standard reasoning tasks, with rigorous logic and fairly standardized reasoning paths. However, when facing highly open problems that require repeated trial and error, they may exhibit:

• A single choice of reasoning direction
• Inability to switch strategies effectively mid-course
• Insufficient sensitivity to unconventional constructions

That said, these models often offer better cost-effectiveness in engineering scenarios and are suitable for reasoning tasks with clear constraints.

Tier 3: Usable Basic Math Ability but Unstable on Complex Reasoning

These models can complete common computations, formula transformations, and basic proofs, but are prone to errors in multi-step reasoning, abstract concept manipulation, or boundary-case handling. They are better suited as auxiliary modules in a pipeline rather than as the core reasoning engine.

How to Build Your Own Mathematical Reasoning Evaluation Set

Rather than relying entirely on public leaderboards or scattered opinions on social platforms, it is better to build a reusable internal evaluation set based on your own business scenarios. Consider selecting problems from the following dimensions:

1. Basic Computation and Algebraic Operations

Cover basic symbolic operations, equation solving, inequality handling, and similar tasks to verify whether the model has foundational mathematical ability.

2. Proof-Type Problems

Require the model to provide rigorous logical derivations, not just numerical answers. Set different difficulty levels, from simple inductive proofs to more complex constructive proofs.

3. Open Exploration Problems

These problems give no explicit solution direction and require the model to try on its own. For example:

• Determine whether a proposition holds and explain why
• Find all cases in which an object satisfies a specific property
• Given incomplete conditions, analyze what additional conditions are needed to reach a conclusion

4. Counterfactual and Boundary-Case Tests

Set boundary conditions that are easy to overlook, or pose counterexample construction tasks that run counter to intuition, and observe whether the model can identify and handle them correctly.

5. Cross-Domain Comprehensive Problems

Mix algebra, geometry, combinatorics, probability, and other content to examine the model’s ability to establish connections across different knowledge systems.

During evaluation, record the following metrics:

• Final answer accuracy
• Logical completeness of the reasoning process
• Ability to recover from errors
• Ratio of reasoning steps to necessary steps
• Degree of explicit attention to boundary conditions

Selection Advice: Scenario Determines the Optimal Choice

In real engineering applications, there is no absolute “best” model—only models better suited to the current scenario.

1. High-Value Research Assistance Scenarios

If the goal is to assist researchers in exploring open problems, generating possible proof ideas, or discovering new construction directions, prioritize models that perform strongest on open problems, have flexible reasoning chains, and possess strong self-verification ability. Even if such models respond more slowly and cost more, they can provide irreplaceable value in critical tasks.

2. Large-Scale Automated Reasoning Tasks

If the business needs to handle large volumes of structured reasoning tasks—such as automated grading, standard problem solving, or logic verification—look for the most cost-effective option among mid-tier models. Key metrics include:

• Pass rate on your own evaluation set
• Average response latency
• Reasoning cost and error correction cost
• Support for batch processing and asynchronous invocation

3. Embedded Lightweight Reasoning Needs

Some scenarios require only basic mathematical ability, such as form calculations, simple logic judgments, or data validation. In these cases, there is no need to pursue top-tier reasoning models; lightweight models combined with a rules engine are often sufficient.

A Reference Evaluation Workflow

Below is a simplified model evaluation workflow that can serve as the basis for an internal testing framework:

1. Build a layered evaluation set
   - Basic tier: 100 problems
   - Advanced tier: 80 problems
   - Open tier: 30 problems
2. Run inference for each candidate model
   - Fixed prompt template
   - Record the full reasoning process
   - Set a maximum reasoning step limit
3. Multi-dimensional scoring
   - Answer correctness: 0/1 or partial credit
   - Process soundness: 1-5 points
   - Self-correction behavior: whether errors are found and fixed
4. Aggregate analysis
   - Pass rate by difficulty
   - Compare reasoning chain length distributions
   - Identify common error patterns
5. Form selection conclusions
   - Compute composite scores based on scenario weights
   - Output recommended models and invocation strategies

The advantage of this approach is that conclusions are based on measured data from your own scenarios rather than scattered external opinions, making them more valuable for engineering reference.

Conclusion

Mathematical reasoning ability has become an important dimension for measuring the core intelligence of large models. Performance on open problems in particular can effectively distinguish the reasoning depth and exploratory capability of models at different tiers. Key factors affecting mathematical ability include reasoning chain strategy, self-verification mechanisms, symbolic manipulation ability, and generalization across problem types.

For developers, the more pragmatic approach is not simply chasing the label of a “strongest model,” but building a layered evaluation set based on actual business scenarios and making comprehensive judgments across correctness, stability, cost, latency, and other dimensions. Only in this way can rational and sustainable technology selection decisions be made in an environment of rapid model iteration.