LLM Math Reasoning Evaluation: Solving Problems Is Not the Same as Doing Research

    Math benchmarks for large language models are updated frequently, but high scores can create a misleading impression of research ability. Based on a recent study of open mathematical conjectures, this article argues that solving well-defined problems and conducting mathematical research are fundamentally different. It explains why current benchmarks struggle to measure research capability, including data contamination, in-distribution bias, answer-verifiability bias, and single-point correctness. It then proposes practical improvements: using held-out problems with unknown answers, separating memorization from reasoning, evaluating reasoning process quality, adding difficulty tiers and contamination detection, and building reproducible evaluation infrastructure.

    Introduction: Evaluation Anxiety Behind Rising Scores

    Over the past two years, leaderboards for large language model evaluation have been refreshed almost every one or two months. Rankings across various “math reasoning benchmarks” continue to rise and fall. For engineering teams, these scores often create an illusion of certainty: the higher the score, the closer the model seems to be to “thinking.”

    However, a recent study on open mathematical conjectures offers a more restrained conclusion: no mainstream model has been able to independently solve a still-open mathematical problem.

    The value of this conclusion is not to deny model capability. Instead, it reminds us to reconsider the essence of evaluation: measuring whether a model can solve problems and measuring whether a model can conduct research are fundamentally different tasks. This article examines the methodological limitations of current evaluations and outlines practical directions for improvement.

    1. The Essential Difference Between Solving Problems and Doing Research

    Understanding the limitations of current benchmarks begins with clarifying the difference between these two types of cognitive tasks.

    Solving a problem is essentially like cooking by following a recipe. The problem has clear known conditions, standardized solution steps, and a unique or decidable answer. What the model needs to do is retrieve and combine the correct reasoning chain from the pattern space covered by its training distribution. The steps are already available, and correctness is clear—just as a cook would not randomly add chocolate to a savory stir-fry.

    Research, by contrast, is open-ended creation without a recipe. It has several decisive characteristics:

    • No established path: There is no ready-made solution template. The model must explore, trial and error, and reconstruct the problem itself.
    • Unknown answers: For many open problems, even the person proposing the problem does not know the answer. There is no standard answer against which correctness can be checked.
    • Long-chain reasoning: A research conclusion may require hundreds of logical steps. An error at any single step can cause the entire result to fail.
    • Requires aesthetic judgment and intuition: Deciding which direction is worth pursuing, or which construction may work, often depends on judgment that is difficult to formalize.

    Most existing benchmarks test the first type of capability. They are good at answering questions such as “Given a problem, can the model reproduce the correct steps?” But they can hardly answer questions such as “When facing a problem that humans have not yet solved, can the model produce genuinely valuable new insight?”

    2. Four Reasons Current Benchmarks Struggle to Measure Research Ability

    1. Data Contamination

    Once a math problem set is made public, it may enter the model’s pretraining corpus. When a model has “memorized” the problem rather than reasoned through the answer, a high score only reflects memorization ability, not generalized reasoning ability. This is like a student having seen the exam paper in advance—no matter how high the score is, it does not demonstrate the student’s true level.

    2. The Gap Between In-Distribution and Out-of-Distribution Performance

    Most benchmark problems fall within the range covered by the training distribution. Real mathematical research, however, often occurs outside the distribution: it requires confronting problem structures that were never seen during training. Excellent in-distribution performance cannot be linearly extrapolated to out-of-distribution problems.

    3. Verifiability Bias

    Current benchmarks tend to favor problems whose answers can be automatically judged, such as fill-in-the-blank questions, multiple-choice questions, or verifiable numerical results. This naturally excludes difficult problems with open answers, constructive proofs, or counterexamples. As a result, the evaluation set is systematically biased toward problems that are easy to score rather than problems that are truly difficult.

    4. Single-Point Correctness Versus Process Reliability

    A model that occasionally answers a difficult problem correctly may simply have benefited from sampling luck or an accidental hit across multiple attempts. Research capability requires reproducibility and interpretability of the process, not correctness in a single result. Evaluations that only look at final correctness conceal a critical piece of information: whether the reasoning process itself is robust.

    3. How to Build a More Rigorous Math Reasoning Evaluation System

    To address these issues, engineering teams can improve evaluation design along the following dimensions.

    3.1 Use Held-Out Problem Sets with Unknown Answers

    Open conjectures that human experts have not yet solved can be introduced as stress tests. Even if the model cannot produce an answer, such problems can effectively calibrate the upper bound of capability. The key point is that correctness cannot be judged in the short term for such problems. Therefore, their value lies in observing the model’s exploration process, rather than judging final correctness.

    3.2 Distinguish Memorization-Based Evaluation from Reasoning-Based Evaluation

    At the data level, decontamination should be performed. Newly produced problems created after the model’s training cutoff date should be prioritized. This can reduce inflated scores caused by memorization from the source.

    3.3 Focus on Process Quality, Not Only Results

    An illustrative evaluation process can be designed as follows:

    1. def evaluate_reasoning(model, problem, reference_solution=None):
    2. """
    3. Evaluate mathematical reasoning across multiple dimensions.
    4. - Do not rely only on the final answer.
    5. - Also assess whether the reasoning chain is structurally sound.
    6. """
    7. result = model.solve(problem, return_steps=True)
    8. metrics = {
    9. "final_correct": check_answer(result.answer, reference_solution), # Final answer correctness
    10. "step_validity": score_step_logic(result.steps), # Logical validity of intermediate steps
    11. "consistency": measure_self_consistency(model, problem, n=5), # Consistency across multiple samples
    12. "verification": formal_verify(result.steps), # Pass rate of formal verification
    13. }
    14. return metrics

    Among these metrics, self-consistency across multiple samples can reveal whether the model truly understands the problem or merely got the right answer by chance. Formal verification uses machine-checkable proof systems to distinguish what “looks correct” from what is “actually correct.”

    3.4 Add Difficulty Tiers and Contamination Detection

    The evaluation set can be layered by difficulty, for example: regular problems, competition problems, and open conjectures. This can be combined with contamination detection mechanisms, such as rewriting problem statements or changing numerical values. The goal is to observe how model performance decays as interference increases. A slower decay indicates that the model’s reasoning relies more on genuine capability rather than surface patterns.

    4. Engineering Recommendations for Evaluation

    To make these methods run at scale, a complete set of infrastructure is usually required:

    • Problem and reference answer management: Use object storage and databases to manage evaluation sets, versions, and annotation metadata, ensuring reproducibility.
    • Batch inference and concurrent scheduling: Use container platforms to elastically launch inference tasks and meet the computational demands of large-scale problem sets.
    • Result collection and log tracing: Use logging services to record the complete reasoning process for each inference, making it easier to audit reasoning chain quality afterward.
    • Metric calculation and visualization: Use monitoring alerts and dashboards to present multi-dimensional metrics in real time and quickly identify capability weaknesses.

    For teams that want to build such evaluation pipelines quickly, general cloud capabilities from Baidu AI Cloud—such as object storage, container platforms, logging services, and monitoring alerts—can serve as foundational support. This allows teams to focus on evaluation methodology itself rather than rebuilding every component from scratch.

    5. Conclusion: An Honest Coordinate for Model Capability

    Returning to the original question: why do we need to build new evaluations when so many benchmarks already exist?

    Because high scores can only prove that a model is good at solving problems. They cannot reflect whether the model is capable of research. When facing open conjectures that humans have not yet solved, the collective silence of current models is itself an honest calibration of capability boundaries.

    For developers, this conclusion is not meant to discourage. Instead, it provides a clearer coordinate: do not equate high problem-solving scores with research ability, and do not extrapolate in-distribution performance to out-of-distribution capability. Truly valuable evaluation should distinguish memory from reasoning, results from process, and in-distribution performance from out-of-distribution performance. It should also observe the quality of model exploration on open problems whose answers are unknown.

    Only by building a rigorous, layered, and verifiable evaluation system can we accurately determine the true boundaries of large language models in mathematical reasoning and apply them in the most appropriate settings.