Hierarchical Evaluation Methods for LLM Mathematical Reasoning Capabilities: From Competition Problem-Solving to Open-Ended Exploration

During model selection, determining “which model is better at math” is often one of the most debated questions. The root of this debate usually lies not in the conclusions themselves, but in the lack of standardized evaluation criteria. Some prioritize accuracy on competition problems, others value the readability of the chain-of-thought, while others care about whether the model can provide valuable partial progress on open-ended math problems. These dimensions are not equivalent, and mixing them makes it difficult to reach a consensus.

This article attempts to abstract these discussions into a reproducible evaluation framework, unfolding layer by layer from capability stratification and task design to metric definition and result interpretation, serving as a reference for developers in practical selection and internal evaluation.

1. Why Hierarchical Evaluation is Necessary

Mathematical capability is not a single scalar. A model might perform stably on standard competition problems but fail quickly on open-ended problems requiring long-term exploration. Conversely, some models can provide inspiring ideas on open-ended problems but exhibit unremarkable accuracy in strict, time-limited, step-limited problem-solving scenarios. Measuring with just one set of problems and a single accuracy metric will inevitably lead to distorted conclusions.

A more reasonable approach is to stratify by task nature. A widely accepted stratification concept in the industry is as follows:

• L1 Basic Computation and Formal Derivation: Standard problems like algebraic transformations, calculus, and linear algebra, emphasizing correct results and standardized steps.
• L2 Competition-Level Comprehensive Problem-Solving: Problems requiring multi-step reasoning and strategy selection, emphasizing accuracy and reasoning chain completeness.
• L3 Open-Ended Problem Exploration: Research problems with no known standard answer or multiple valid answers, emphasizing partial progress, idea validity, and verifiable intermediate conclusions.
• L4 Engineering Usability: The stability, latency, cost, and controllability when invoking the model for math-related tasks in real-world development workflows.

After stratification, different task sets and metric systems can be used for each layer. Consequently, conclusions can be mapped to specific scenarios rather than a vague claim of “who is stronger.”

2. Task Set Design Principles

The quality of the task set determines the credibility of the evaluation conclusions. Several principles should be followed during design.

First, the difficulty gradient must be calibrated. There should be a difficulty distribution within the same layer to avoid concentrating entirely on “medium-hard” problems, which reduces discrimination. A mix of public problem sources and internally constructed problems can be used, with an estimated difficulty level annotated for each problem.

Second, the answer format must be automatically verifiable. L1 and L2 layers should尽量 select tasks where the final answer can be structurally compared, such as numerical values, expressions, or finite step checks for proofs. For proof-type problems, a semi-automatic approach of “key lemma hit + logic chain check” can be adopted to reduce manual review costs.

Third, the L3 layer requires an independent review mechanism. Open-ended problems lack standard answers, so the evaluation focus should be on “whether the reasoning process generates verifiable intermediate conclusions” and “whether there are obvious logical leaps or incorrect citations.” It is recommended to have two or more reviewers score independently and provide explanations for any disagreements.

Fourth, control data contamination risks. Public problem sources might have been seen by the model during training. Therefore, a certain proportion of constructed and rewritten problems should be retained, and the problem source composition should be disclosed in the evaluation report.

A simplified task configuration example (pseudo-code, representing structure only):

eval_suite:
  - layer: L1
    size: 200
    source: constructed
    verify: exact_match
  - layer: L2
    size: 120
    source: mixed
    verify: answer_and_step_check
  - layer: L3
    size: 20
    source: open_problems
    verify: dual_review
  - layer: L4
    size: 50
    verify: latency_cost_stability

3. Metric Design and Collection

Different layers require different metrics, and these metrics should be quantifiable.

L1/L2 layers primarily use the pass rate as the main metric, supplemented by process metrics such as “average reasoning steps” and “position of the first error.” Process metrics help explain differences: when two models have similar pass rates, the model with a shorter reasoning chain and errors occurring later is typically more stable.

L3 layer recommends a three-tier scoring system: no valid progress, partial valid progress, and verifiable substantive conclusions. Additionally, “average exploration rounds” and “manual intervention times” should be recorded to reflect the model’s self-correction capability in long-horizon tasks.

L4 layer focuses on engineering metrics: average time per task, invocation cost, output format compliance rate, and stability under excessively long inputs. This part is often overlooked but carries significant weight in actual business operations.

It must be emphasized that metric collection should fix decoding parameters and prompt templates. Temperature, maximum output length, and whether step-by-step reasoning prompts are enabled will significantly affect the results. These configurations must be fully disclosed in the evaluation report; otherwise, the conclusions are not reproducible.

4. Common Pitfalls in Result Interpretation

Pitfall 1: Treating a single leaderboard rank as the capability conclusion. Different task sets have different difficulty distributions. A rank only indicates relative performance on that specific task set. Cross-task-set conclusions require recalibration.

Pitfall 2: Ignoring reasoning chain quality. Two models with the same pass rate may differ greatly in the readability and conciseness of their reasoning chains. For scenarios requiring manual review, verbose and complex reasoning chains increase audit costs, which is a capability difference in itself.

Pitfall 3: Using open-ended problem performance to negate basic capabilities. Performance on open-ended problems is heavily influenced by prompt strategies and exploration budgets, and cannot be directly reverse-engineered to deduce basic problem-solving capabilities. The converse is also true.

Pitfall 4: Ignoring the speed of version iteration. Model capabilities change rapidly. Any evaluation conclusion should be tagged with a timestamp and version information, and the task set should be rerun periodically.

5. Practical Advice for Developers

For teams needing to integrate model capabilities into their projects, the following workflow is recommended:

1. Define the scenario layer: First, determine which layer (L1–L4) the business belongs to. Avoid using competition problem conclusions to guide engineering selection.
2. Build a small private task set: Extract 30–50 problems from real business issues, annotate answers and difficulties, and use them as a long-term regression baseline.
3. Fix evaluation configurations: Standardize prompt templates, decoding parameters, and verification scripts to ensure multiple evaluations are comparable.
4. Use both process and result metrics: Look at both the pass rate and reasoning steps, error positions, and format compliance rates.
5. Conduct periodic re-evaluations: Rerun the baseline after model version updates, recording trends rather than single-point conclusions.

In actual engineering, the evaluation process can also be integrated with general cloud capabilities: use object storage to store task sets and model outputs, use logging services to record configurations and results of each evaluation, and use monitoring and alerting to track changes in regression metrics. These general cloud capabilities can help teams transform evaluation from a one-off script into a maintainable workflow without relying on specific platform interfaces.

6. Conclusion

When discussing “how a model’s math capability is,” the real value lies not in a simple ranking, but in whether the evaluation criteria are clear, whether the task set is reproducible, and whether the metrics cover the target scenario. The significance of a hierarchical evaluation framework is breaking down vague overall judgments into verifiable sub-conclusions: basic problem-solving looks at pass rate and reasoning chain quality, open-ended problems look at partial progress and reviewer consistency, and engineering deployment looks at stability and cost. By executing this method, teams can complete model selection under neutral conditions and develop a sustainably trackable understanding of capability boundaries.