LLM Mathematical Research Evaluation: Boundaries, Formal Verification, and Engineering Deployment
This article explains why high benchmark scores do not necessarily reflect a large language model’s mathematical research capability. It distinguishes routine problem solving from open-ended research, argues that theoretical computer science is a suitable evaluation domain, discusses the difficulty of grading research-level outputs, and examines the role and limits of formal verification. It also recommends a three-tier evaluation dataset design and a reusable engineering system for reproducible, auditable evaluation. The goal is to shift mathematical research evaluation from a single benchmark score to verifiable evidence, failure analysis, and reproducibility.
1. Executive Summary: High Benchmark Scores Do Not Equal Research Capability
For readers involved in model evaluation, AI research, or technical management, the core conclusion is straightforward: the value of evaluating a large language model’s mathematical ability is not whether it can achieve high scores on standard problems, but whether it can produce verifiable mathematical evidence on open-ended or high-difficulty problems.
Problem-solving evaluations are useful for measuring reproduction ability and reasoning fluency. Research evaluations are closer to “starting from zero”: the model may need to prove a conjecture, construct a counterexample, establish an algorithmic lower bound, or clarify an ambiguous definition. If evaluation relies only on public benchmarks, it is easy to confuse “being able to solve exam-style problems” with “being able to do research.”
Benchmarks are useful for quick screening, but they have several natural limitations:
- Many benchmark problems have known standard answers, so models may score well through memorization or pattern matching.
- Answer-based scoring cannot capture research-oriented outputs such as proposing a new lemma, finding a counterexample, or deriving a new upper bound.
- Benchmarks usually do not record why a model failed, nor do they distinguish between “correct idea with minor implementation errors” and “completely invalid reasoning.”
Therefore, evaluating mathematical research ability requires considering problem difficulty, verifiability, and reproducibility together.
2. Problem Solving vs. Research: Where the Boundary Lies
Problem solving is closer to reproducing a verifiable path. The conditions are clear, the objective is well-defined, and a reference solution usually exists. The model is expected to complete reasoning, calculation, or proof steps.
Research is different. The goal may be to prove a conjecture, construct a counterexample, provide an algorithmic lower bound, or resolve an unclear definition. Successful research is not only about whether the final answer is correct, but also about whether the proof chain can be inspected.
A key difference lies in the evaluation standard. A problem-solving task can require the model’s answer to match a reference answer. An open-ended research task usually has no reference answer. Evaluation must instead rely on logical proof, formal verification, peer review, or expert judgment.
In other words, problem-solving evaluation asks: “Can the model solve this particular problem?” Research evaluation asks: “Can the model advance a class of problems?” The former can be highly automated; the latter requires a more complex system of trustworthy judgments.
3. Why Theoretical Computer Science Is Suitable for Research Evaluation
Among many mathematical subfields, theoretical computer science (TCS) is particularly suitable for open-ended evaluation for three reasons.
First, TCS problems can often be expressed with precise symbolic definitions. For example, the halting problem asks whether there exists a universal algorithm that determines whether an arbitrary program will stop. The P vs. NP question concerns the relationship between verification and search in computational complexity. Sorting lower bounds concern how many comparisons are required for comparison-based sorting at minimum. These topics can be formalized and used to construct verifiable propositions.
Second, TCS results rely heavily on proofs or constructions. An algorithmic upper bound requires giving an algorithm and analyzing its complexity. A lower bound requires proving that no faster algorithm exists. Undecidability requires a reduction proof. Whether a conclusion holds cannot depend on verbal description; it depends on a checkable logical chain.
Third, mathematical proofs are generally more suitable for machine verification than most natural-language conclusions. Large language models may generate plausible but difficult-to-verify content in many domains. Mathematics, especially proofs, can be more reliably checked if the proof can be translated into a formal language and submitted to a proof checker. This makes mathematical evaluation closer to “machine-gradable.”
4. The Real Bottleneck: How to Grade Research-Style Outputs
Evaluating whether a model can solve a problem is relatively straightforward. Evaluating whether it can do research is much harder. Two major obstacles stand out.
The first is grading. A model may produce a long proof containing definitions, lemmas, cases, counterexample constructions, and complexity analysis. Reviewing it page by page is costly. Different experts may also apply different standards of rigor. If a model claims to have solved a difficult problem, reviewers must understand the relevant background, otherwise it is difficult to determine whether the claim is valid.
The second is the absence of a ground truth. For many open problems, even humans do not have a definitive answer. Without a reference solution, evaluation must focus on the quality of evidence: Is there a formal proof? Can the counterexample be reproduced? Is the model merely restating known results? Does the argument contain circular reasoning?
A practical grading mechanism should combine machine verification and human review:
| Stage | Practice | Suggested Ratio or Threshold | Dispute Handling |
|---|---|---|---|
| Automated formal verification check | Run proof checkers on outputs that can be converted | Run for all applicable outputs | Archive checker versions and library versions |
| Manual review of non-formalized samples | Experts identify failure reasons | Sample at least 10% | Use two-person review; disagreements go to arbitration |
| Review of high-risk conclusions | Claims of solving open problems or establishing new lower bounds | Full review | Require a proof summary and dependency library notes |
| Result archiving | Save questions, outputs, and logs | Archive every evaluation run | Maintain version numbers and a reproduction checklist |
This approach avoids the impractical requirement that humans review every proof in full, while also preventing the mistake of automatically rejecting any output that a machine checker cannot verify.
5. Formal Verification: Let Machines Check Correctness, but Know the Limits
The basic idea of formal verification is to translate mathematical statements and proofs from natural language into a formal language recognized by a proof system. A proof checker then verifies whether each inference step is valid. Common tools and systems include Lean, Coq, and Isabelle, although their syntaxes, library ecosystems, and proof strategies differ significantly.
The evaluation loop can be simplified as follows:
problem_set = load(open_questions + formalized_problem_statements)for problem in problem_set:model_output = llm.solve(problem)formal_proof = convert(model_output, target=proof_system_language)if formal_proof.is_valid():result = proof_checker.verify(formal_proof)record(problem, result, elapsed_time, reasoning_steps)else:record(problem, result="failed formal verification")
The value of formal verification is that it enables scalable grading. Once a proof passes a proof checker, the logical chain has relatively strong credibility, and evaluation no longer depends on whether the text “sounds right.”
However, formal verification has clear boundaries and should not be treated as a universal answer.
First, there is a semantic alignment cost when translating natural language into formal language. A phrase may correspond to multiple mathematical definitions. If the model misunderstands the statement, the resulting formal proof may verify a different proposition.
Second, differences between proof systems and dependency libraries can affect results. The same mathematical result may be expressed differently in different libraries, with different available theorems and naming conventions. Migration across systems can be costly.
Third, dependency library limits affect the scope of verification. Some conclusions depend on high-level theorems that have not yet been formalized. In such cases, evaluation may only verify local lemmas rather than the full claim.
Fourth, failure to formalize does not necessarily mean the mathematics is wrong. Conversion failure, missing libraries, or syntax incompatibility can all cause a proof to fail verification. This only shows that the current pipeline could not produce verifiable evidence; it does not directly imply that the model lacks research ability.
Fifth, passing formal verification does not necessarily mean a research breakthrough. The model may have verified a known theorem, completed an intermediate lemma, or weakened an open problem to a special case. Evaluation reports must distinguish among “proof succeeded,” “some lemmas succeeded,” and “not successful but the direction has value.”
6. Selecting the Evaluation Dataset: Three Tiers of Questions
To evaluate research ability meaningfully, the dataset should be layered. Three tiers are recommended.
Tier 1: Solved and Already Formalized Classical Problems
These problems are used to calibrate the baseline. Sources can include classic proofs from textbooks, competition problems, formal proof libraries, and theorems from standard references. Problems should have public proofs and clear background so that results can be reviewed.
Difficulty calibration can be based on:
- The time human experts take to complete the task;
- The pass rate of model baselines;
- The average number of reasoning steps.
Tier 2: High-Difficulty Problems That Are Public but Hard to Solve
These problems are used to create separation between models. Sources can include public TCS challenges, verified conclusions from papers, and tasks requiring counterexample construction or complexity analysis.
Scoring criteria should be explicit:
- Did the model provide a verifiable proof?
- Is the counterexample reproducible?
- Did the model reduce the complexity of the problem?
- Is the output merely a restatement of known results?
Tier 3: Still-Open Conjectures
These problems are used to probe the upper boundary of model capability. Sources can include lists of long-standing open problems, survey papers, and conjectures raised in conference discussions.
This tier is not suitable for pursuing large sample sizes. It is better suited as a probe for capability limits. The dataset also needs maintenance: solved problems should be periodically removed, and the latest human progress should be recorded to prevent the benchmark from becoming outdated.
One empirical judgment is that open conjectures remain extremely difficult for current models. As of June 2026, within the public evaluations and cases reviewed for this article, no model was observed to independently prove a long-standing open conjecture through its own effort alone. This judgment only reflects the model versions, problem scope, toolchains, and evaluation criteria at the time of writing, and should not be treated as a permanent conclusion.
7. Engineering Deployment: Building a Reusable Evaluation System
From an engineering perspective, a reusable evaluation system should include at least four modules:
- Compute and model serving;
- Data and storage;
- Formal verification services;
- Logging, monitoring, and alerting.
These capabilities can be combined on general cloud platforms or compliant cloud services without depending on a specific commercial vendor.
To make the evaluation executable, the following checklist is recommended:
- Sample size: Baseline problems should ideally number at least 100. Discriminative problems can range from 50 to 200 depending on resources. Open conjecture problems should be limited to 10–30. Sample sizes may vary by objective, but problem versions must always be recorded.
- Model version: Record model name, version, API snapshot, weight hash, or deployment configuration. Results from different versions should not be directly merged or compared.
- Random seed: Fix parameters such as temperature and top-p. If randomness is allowed, use multiple random seeds and report mean, variance, or confidence intervals.
- Inference timeout: Set a timeout threshold for each problem, such as 30, 60, or 180 minutes, and record the timeout rate.
- Checker version: Record the versions of proof checkers such as Lean, Coq, and Isabelle, as well as dependency library commit IDs.
- Failure classification: At minimum, distinguish among:
- Unable to form a formal proof;
- Syntax error;
- Incomplete proof;
- Missing dependency;
- Timeout;
- Circular reasoning;
- Only verified a special case;
- Model refusal.
- Reproduction fields: Include problem ID, prompt version, model output hash, formalization conversion hash, verification logs, elapsed time, token count, final answer, and human review conclusion.
A recommended pipeline is:
- Freeze the data version;
- Run batch model inference;
- Perform automatic formalization conversion;
- Run proof checker verification;
- Classify failures;
- Conduct human sampling review;
- Generate the evaluation report.
The report should include success rate, failure distribution, average elapsed time, formal verification pass rate, and human review agreement rate, rather than reducing the result to a single score.
8. Common Misconceptions and Conclusion
Several common misconceptions should be avoided in practice.
Misconception 1: A high benchmark score means the model can do research. Benchmark scores mainly reflect problem-solving fluency. They do not directly indicate the ability to advance open problems.
Misconception 2: Passing formal verification means a research breakthrough. It only means that the proposition was verified in the current system. It still remains to determine whether a new problem was solved or whether the output merely restates known results.
Misconception 3: Failure on a single problem defines the model’s ceiling. Failure may come from an overly difficult problem, excessive formalization cost, toolchain limitations, or insufficient prompting.
Misconception 4: A model has no research value if it cannot solve an open problem. Failure outputs may still contain useful lemmas, partial proofs, counterexample clues, or evidence of where the reasoning went wrong.
Overall, evaluation of large language models’ mathematical research ability should shift from a “score competition” to an “evidence competition.” A valuable evaluation should show where a model succeeds, where it fails, why it fails, and whether the results can be reproduced. Formal verification provides a technical foundation for machine-checkable evidence, but it is not the only answer.
For developers and enterprise users, building a trustworthy, reproducible, and scalable evaluation system is more important than chasing a single leaderboard ranking. This is how AI mathematical research capability can be understood within its real boundaries, and how future technical progress can begin from an accurate baseline.
