How to Evaluate Long-Reasoning LLMs for Math Proof and Review Capabilities
This article explains how to evaluate long-reasoning large language models through task-oriented benchmarks. It covers layered evaluation tasks, math proof metrics, review metrics, long-context stability, tool-use controls, budget and checkpoint design, engineering platforms, and a pre-launch checklist for model selection.
How to Evaluate Long-Reasoning LLMs for Math Proof and Review Capabilities
1. What Are Long-Reasoning LLMs, and Why Should They Be Evaluated by Task?
Long-reasoning LLMs generally refer to models with capabilities such as long-context processing, multi-step reasoning, tool invocation, and structured output. They are often used for complex tasks such as mathematical proofs, paper reviews, code audits, and project credibility assessment. Unlike ordinary chat question answering, these tasks should not be judged only by whether the output reads smoothly. The evaluation should consider whether conclusions are traceable, whether steps are verifiable, and whether resource costs are controllable.
The key point of task-oriented evaluation is to decompose model capabilities into observable and comparable metrics. At a minimum, these include mathematical proof capability, review capability, long-context stability, tool-use strategy, output readability, and cost efficiency. Only by breaking tasks down can we determine which scenarios a model is suitable for, rather than relying on a single ranking or subjective impression.
2. First, Split the Evaluation Targets into Four Task Types
Before building an evaluation set, it is recommended to stratify tasks because different tasks correspond to different metrics.
| Task type | Focus |
|---|---|
| Mathematical proof | Proposition understanding, proof completeness, logical rigor, edge-case handling |
| Paper review | Core contribution, method reasonableness, experimental credibility, evidence citation |
| Project credibility review | Architecture, code, security, compliance, maintainability, blocking issues |
| Long conversation and multi-turn correction | Goal drift, style drift, conclusion oscillation, insufficient evidence |
After stratification, evaluation is no longer a single overall score. Instead, it produces capability profiles by scenario. For example, one model may produce stable mathematical proofs but become overly focused on secondary details during paper review. Another model may provide robust review conclusions but take longer to complete complex reasoning.
3. How to Evaluate Mathematical Proof Capability
3.1 Set Problems by Difficulty Level
Mathematical proof problems should not be evaluated as one mixed pool. They can be divided into four categories:
- Basic problems: clear concepts and few steps, used to check whether the model fabricates proofs.
- Intermediate problems: require combining multiple lemmas or formulas, testing the ability to organize reasoning chains.
- High-difficulty problems: used to observe whether the model can perform exploratory reasoning and mark points that need verification when uncertain.
- Format-output problems: require structured notes, symbol explanations, step explanations, and conclusion summaries.
3.2 Metrics and Measurement Criteria
Mathematical proof tasks can use the following metrics. To avoid impractical indicators, the measurement method and scoring owner should be specified at the same time.
| Metric | Meaning | Measurement method |
|---|---|---|
| Proposition understanding accuracy | Whether the model correctly restates the conditions and conclusion of the proposition | Human double review, checking whether all conditions are covered |
| Proof completeness | Whether key derivation steps are covered | Rule-based field checking plus expert review |
| Logical rigor | Whether there are jumps, circular reasoning, or invalid assumptions | Experts deduct points based on error types |
| Evidence verifiability | Whether the theorems and lemmas used can be verified | Spot-check cited sources or formula derivations |
| Edge-case handling | Whether special cases and equality conditions are discussed | Check whether specified edge cases are covered |
| Output standardization | Whether structure, symbols, and terminology are consistent | Format validation and human spot checks |
| Resource consumption | Tokens, turns, retrieval calls, time | Statistics from invocation logs |
If the evaluated objects allow manual review, it is recommended to use dual review for logical rigor and record points of disagreement. If automation is pursued, rule-based checks can first verify key fields, and high-risk samples can then undergo manual review.
3.3 Prompt Constraints
Mathematical task prompts should emphasize “no fabrication, no skipped steps, and traceability.” Example:
Please prove the following mathematical proposition:Task requirements:1. Use only verifiable theorems and reasoning rules;2. If a step is uncertain, mark it as pending verification;3. Do not fabricate nonexistent conclusions, references, or lemmas;4. The output should include: proposition restatement, key lemmas, proof steps, edge-case conditions, and a summary;5. If a complete proof cannot be produced, provide the currently completed portion and the reason why it is incomplete.
Such prompts can help reduce the risk of fabricated intermediate steps and excessive terminology in many scenarios, but their effectiveness still needs to be validated on your own evaluation set.
3.4 Example: Comparing Model Output with Expected Output
Take a simple proposition as an example:
Problem: Prove that for any real number x, x^2 + 1 ≥ 2x.
An unqualified output might be:
“This problem can be obtained through classical inequalities and monotonicity of functions. It is a common conclusion, and the process is omitted.”
This output lacks verifiable steps and does not meet review requirements.
The expected output should be close to:
Proposition restatement: For any real number x, it is necessary to prove that x^2 + 1 ≥ 2x.Key derivation:x^2 + 1 - 2x = (x - 1)^2.Since the square of any real number is nonnegative, (x - 1)^2 ≥ 0.Therefore, x^2 + 1 - 2x ≥ 0, which means x^2 + 1 ≥ 2x.Edge case:When x = 1, equality holds.Conclusion:The proposition holds, and equality holds when and only when x = 1.
During evaluation, this can be used to check whether the model provides equivalent transformations, whether it uses nonnegativity, and whether it explains the equality condition, rather than scoring only based on language fluency.
3.5 Evaluation Pseudocode
To unify the budget data structure, the budget can be defined as an object containing used_tokens:
from dataclasses import dataclass@dataclassclass Budget:max_tokens: int = 50000used_tokens: int = 0max_steps: int = 20max_tool_calls: int = 5max_runtime_minutes: int = 30def evaluate_math_model(model_client, problem, budget):history = []trace = []for step in range(budget.max_steps):if budget.used_tokens >= budget.max_tokens:return {"status": "budget_exceeded", "trace": trace}response = model_client.step(task="math_proof",problem=problem,history=history,constraints=["do_not_fabricate","show_all_key_steps","mark_uncertain_points",])trace.append(response)budget.used_tokens += response.tokensif response.is_final:breakhistory.append(response)return grade_proof(problem, trace, budget)
Here, grade_proof should combine rule validation and human review. Rule validation can check whether fields such as proposition restatement, lemmas, proof steps, edge-case conditions, and conclusions are included. Human review then judges whether the logical chain holds.
4. How to Evaluate Paper and Project Reviews
4.1 Overall First, Then Local Details
A common failure in review tasks is that the model spends too much time checking secondary details, such as whether a citation is completely precise or whether a number is exactly consistent, while ignoring whether the overall contribution of the paper is valid, whether the experimental design is sufficient, and whether the project has critical risks.
A high-quality review model should follow an “overall first, then local” output structure:
Overall conclusion: Pass / Conditional pass / FailBlocking issues:1. Issue one2. Issue twoImportant but non-blocking issues:1. Issue one2. Issue twoSuggestions for further verification:1. Verification method one2. Verification method two
Only when details are sufficient to affect the core conclusion should they be listed as critical risks.
4.2 Review Metrics and Measurement Criteria
| Metric | Description | Measurement method |
|---|---|---|
| Key issue recall rate | Whether the model identifies issues that affect the conclusion | Number of blocking issues hit by the model divided by the number of gold-standard blocking issues |
| False positive rate | Whether irrelevant issues are marked as severe issues | Number of non-blocking issues marked as blocking issues divided by the total number of blocking issues identified by the model |
| Review depth | Whether the review covers methods, experiments, data, conclusions, and boundaries | Count of covered dimensions |
| Actionability of suggestions | Whether revision suggestions are specific and operable | Proportion of actionable suggestions judged by humans |
| Overall judgment stability | Whether the conclusion remains consistent during long conversations | Comparison of conclusion labels across multiple rounds |
| Evidence citation quality | Whether verifiable evidence is provided | Spot-check correspondence between citations and the original text |
Paper review example: If a paper claims that a new method improves performance by 3 percentage points on public datasets, but does not explain whether the baseline was tuned, whether data leakage exists, or whether the test set was selectively used, these may constitute blocking issues. In this case, the model should not merely point out that “a sentence is not clear enough,” but should prioritize judging whether the conclusion is credible.
4.3 Stratified Review Prompt
The model can be asked to output the overall conclusion first, then output details:
Please review the following paper.Please output in the following order:1. Overall assessment: whether the conclusion is credible and whether the main contribution is valid;2. Key risks: list only blocking issues that affect the conclusion;3. Secondary issues: issues that do not affect the core conclusion but are worth revising;4. Verification suggestions: experiments, proofs, or data checks that need to be supplemented;5. Final recommendation: pass, conditional pass, or fail.Note: Do not prioritize getting stuck on irrelevant details. Only when details are sufficient to affect the core conclusion should they be listed as key risks.
This type of prompt can help reduce situations of “long process but ineffective conclusion” in many review tasks, but its benefits still need to be validated through real cases.
5. How to Evaluate Long-Context Stability
The stability of long-reasoning models during long conversations is a key challenge in engineering deployment. Some models produce direct conclusions in short conversations, but as the context becomes longer, the following issues may appear:
- Goal drift: Earlier requirements ask for an overall logic review, but later the model shifts to trivial details.
- Style drift: The model shifts from concise explanation to metaphor, analogy, or terminology stacking.
- Insufficient evidence: The model outputs many general judgments without verifiable evidence.
- Conclusion oscillation: The model’s views become inconsistent across multiple rounds of interaction.
- Over-explanation: A simple question is expanded into a long background introduction.
Therefore, long-context tasks should focus on evaluating:
| Metric | Measurement method |
|---|---|
| Conclusion stability | Whether core conclusions remain consistent after multiple rounds of dialogue |
| Goal retention rate | Whether each round of output still satisfies the initial constraints |
| Evidence density | Whether key judgments are supported by evidence |
| Expression redundancy | Whether there is a large amount of uninformative expansion |
| Degree of structuring | Whether the output is easy to read and review |
For review tasks, evidence-based and itemized expression is usually more reliable than literary expression. The model does not need to pursue polished language. It should first ensure that conclusions are clear, evidence is explicit, and reasoning is traceable.
6. How to Manage Retrieval and Tool Use
A common issue with long-reasoning models is over-retrieval. Even when there is no explicit requirement for web access or data retrieval, the model may automatically search, verify, or repeatedly query, leading to longer task duration and higher resource consumption.
The evaluation should clearly specify whether the model is allowed to use tools such as retrieval, calculation, and code execution. If allowed, constraints should be set:
{"allowed_tools": ["retrieval tool", "calculation tool", "code execution tool"],"max_queries": 3,"require_citation": true,"stop_when": ["key conclusions have been verified","the query limit has been reached","the remaining budget is insufficient"]}
Each retrieval should record: the reason for the query, the query content, the returned result, whether it was cited in the final answer, whether it was a repeated query, and whether the query was necessary.
Tool invocation metrics can include:
| Metric | Description | Measurement method |
|---|---|---|
| Necessary retrieval rate | Whether retrieval truly serves the task | Number of necessary queries divided by total queries |
| Repeated query rate | Whether the model repeatedly queries the same information | Number of repeated queries divided by total queries |
| Ineffective retrieval rate | Whether query results were unused | Number of uncited queries divided by total queries |
| Retrieval latency | Whether retrieval significantly increases task time | Ratio of retrieval time to total time |
| Citation accuracy | Whether cited content is consistent with the final answer | Sampling comparison between citations and output |
If the model repeatedly queries the same information, or if query results are irrelevant to the task, efficiency penalties should be applied.
7. How to Design Quotas, Budgets, and Checkpoint Recovery
Complex long tasks often require long-running execution. If a model does not have budget-control capability, it may easily consume too many resources within a limited quota, and may even trigger limits before the task is completed. In engineering practice, quotas should be treated as part of the task state.
7.1 Set Budget Boundaries
Each task should have a clear budget, and the data structure should remain consistent:
budget = Budget(max_tokens=50000,used_tokens=0,max_steps=20,max_tool_calls=5,max_runtime_minutes=30)
When any metric approaches its threshold, the system should proactively reduce the model’s reasoning scale, pause retrieval, or output intermediate results.
7.2 Checkpoints and Resume from Breakpoints
Long tasks should not start from scratch every time. It is recommended to save intermediate states:
state = {"task_id": "proof_case_001","goal": "prove the given proposition and output a structured explanation","completed_steps": ["restating the proposition", "selecting lemmas"],"pending_steps": ["discussing edge cases", "final integration"],"used_tokens": 18000,"used_tool_calls": 2}
If a task is interrupted due to quota, timeout, or system restrictions, the system should support: restoring the context summary, continuing unfinished steps, revalidating key conclusions, avoiding repeated retrieval, and outputting intermediate reports.
7.3 Incorporate Cost into Quality Scoring
Evaluation should not look only at answer quality. It should also consider the cost required to achieve that quality. Assuming all metrics are normalized to a range of 0 to 1, the following example formula can be used:
Composite score =0.35 × task completion+ 0.30 × logical rigor+ 0.15 × output readability+ 0.10 × conclusion stability+ 0.10 × evidence quality- 0.10 × retrieval penalty- 0.10 × token penalty
The retrieval penalty and token penalty also need to be normalized to a range of 0 to 1. If the composite score is below 0, it can be truncated to 0. These weights are only examples. In practice, they should be calibrated according to the task: mathematical proofs can increase the weight of logical rigor, paper review can increase the weight of key issue recall, and cost-sensitive scenarios can increase the weight of budget penalties.
8. Engineering Evaluation Platform and Implementation Checklist
For teams, it is best to build a stable evaluation platform rather than rely on temporary testing by individuals. A reproducible evaluation platform usually includes a test case library, an evaluation executor, a metric calculation module, and a result storage and audit module.
- Test case library: mathematical proof problems, paper review materials, project documents, code repository summaries, multi-turn follow-up scenarios, edge inputs, expected outputs, and scoring criteria.
- Evaluation executor: invoke the model, record inputs and outputs, collect token usage and time consumption, invoke tools, save intermediate traces, and handle timeouts and budget interruptions.
- Metric calculation module: automatic rule scoring, manual review input, multi-round result comparison, stability analysis, cost analysis, and failure-case attribution.
- Result storage and audit: object storage saves raw inputs and outputs, log services record invocation traces, monitoring systems track task success rates and resource consumption, and databases save evaluation scores.
Implementation checklist:
- Have tasks been divided into four categories: mathematical proof, paper review, project review, and long conversation?
- Does each test case have a clear expected result, such as “whether the proof is complete” or “whether blocking issues exist”?
- Do metrics have measurement methods and scoring owners?
- Do prompts require no fabrication, no skipped steps, and marking of uncertain points?
- Do tool invocations have a whitelist, call limits, and stop conditions?
- Are budgets unified into readable and writable objects and do they include used_tokens?
- Do long tasks save breakpoint states and support recovery?
- After model upgrades, can historical test cases be used for regression testing?
9. How to Choose a Long-Reasoning LLM
Different tasks have very different model requirements. A model that performs strongly in mathematical proofs may be too divergent during long-document review. A model that is more robust in review tasks may be slower in complex reasoning. Model selection should not blindly rely on a single leaderboard, but should be based on your own task set.
It is recommended to make judgments from three levels:
- Capability match: Check the task stratification results and confirm whether the model meets requirements on the target tasks.
- Risk control: Check whether the model fabricates evidence, over-retrieves, or shows conclusion oscillation.
- Acceptable cost: Make a comprehensive judgment based on tokens, turns, retrieval calls, and manual review cost.
Output acceptance can be divided into:
- Acceptable: conclusions are basically correct and the format is usable.
- Conditionally acceptable: a small amount of manual correction is required.
- Unacceptable: critical errors, omissions, or misleading content exist.
- High risk: fabricated evidence, logical jumps, or unreliable conclusions exist.
10. Conclusion: Make Decisions by Checklist Before Launch
Evaluating long-reasoning LLMs is not simply about determining “which model is stronger.” The key is to build a reproducible, comparable, and governable engineering evaluation method. Before launch, decisions can be made using the following checklist:
- Split tasks first, and do not use the same score across mixed tasks.
- Build a gold-standard set based on real business needs.
- Metrics must include measurement methods and scoring owners.
- Prompts should constrain the model to avoid fabrication, avoid skipped steps, and remain traceable.
- Tool invocations should have a whitelist, upper limits, and stop conditions.
- Budgets, breakpoints, and cost penalties should be incorporated into unified state management.
- Use platform-based test cases and logs to support regression testing.
Only when model capabilities are evaluated within real business constraints can we determine whether a model is suitable for scientific research, engineering review, and complex reasoning scenarios, and only then can we strike a balance among quality, efficiency, and cost.
