Benchmarking Frontier LLMs on Mathematical Research: Capabilities, Boundaries, and Evaluation Pipelines
A joint academic-industry evaluation places frontier large language models in theoretical computer science and mathematical research scenarios rather than conventional calculation tasks. The key finding is that no reference model could independently produce a credible proof for an unresolved open conjecture. This article examines the shift from problem-solving benchmarks to research-grade benchmarks, why theoretical computer science is a useful testbed, how to design research-oriented evaluation sets, what frontier model evaluations reveal, five dimensions of capability boundaries, how to build an engineering evaluation pipeline, and practical recommendations for developers and research teams.
A recent study jointly conducted by academic and industry researchers placed several frontier large language models into evaluation scenarios drawn from theoretical computer science and mathematical research. The focus was not on whether a model can solve routine calculation problems, but on whether it can demonstrate reliable research capability on proof-oriented, open-ended, and research-level questions. One of the key conclusions was that, when facing open conjectures that remain unresolved, none of the reference models could independently complete a credible proof. This conclusion is unsurprising but valuable: it clearly exposes the boundary between “a model that can solve problems” and “a model that can conduct research.”
From Problem-Solving Benchmarks to Research-Grade Benchmarks
Most traditional large language model evaluations revolve around multiple-choice questions, fill-in-the-blank items, short-answer questions, or coding problems. Such benchmarks can measure knowledge coverage, reasoning stability, and instruction-following ability, but in essence they remain close to “cooking by following a recipe”: the problem has a clear boundary, the answer has a standard form, and the scoring rule is relatively fixed. A model may obtain a high score if it has seen similar patterns in training data or can perform a limited number of compositional reasoning steps at inference time.
Research-level problems are different. Theoretical computer science focuses on “computation” itself, studying abstract questions such as computability, complexity, algorithm lower bounds, proof complexity, randomness, interaction, and the foundations of cryptography. It is not concerned with how to repair a computer or how to write business code; it is concerned with whether a class of problems can be solved efficiently in principle, whether a certain path is doomed to fail, and whether a proof system is sufficiently strong. Such problems usually do not have ready-made steps, and even the person posing the problem may not know the final answer. Therefore, using conventional accuracy to measure a model’s research capability can easily distort the picture.
A research-grade benchmark should answer at least three questions. First, does the problem set cover a sufficiently broad range of theoretical directions? Second, are the answers primarily verifiable proofs? Third, how should open problems be scored? If only the final conclusion is assessed, a model may appear correct by guessing or by applying common results. If only the form of the proof is assessed, the innovation of the proof idea and the taste involved in problem selection may be ignored. Evaluation design must therefore consider correctness, completeness, verifiability, clarity of assumptions, and the boundaries of conclusions.
Why Theoretical Computer Science Is a Good Testbed for Research Capability
Theoretical computer science sits at the intersection of mathematics and computer science and primarily studies the limits of computation. It is suitable as a touchstone for research capability for three reasons.
First, proofs are verifiable. Compared with general natural-language argumentation, mathematical proofs are easier for machines or experts to check step by step. Definitions, lemmas, reasoning steps, and conclusions in a proof are strongly constrained by logic, making errors relatively easier to locate. Although natural-language proofs still contain ambiguity, formal checking, stepwise review, and cross-review can significantly reduce the cost of grading.
Second, open problems are difficult to solve through memorization. Open conjectures often have no public answer, so a model cannot simply retrieve the final conclusion. It must propose hypotheses, construct counterexamples, establish reductions, analyze complexity, or provide lower-bound proofs. Even if a model produces an apparently complete proof, rigorous review is still required to determine whether it truly advances the problem.
Third, the research process includes “problem formulation” and “negative results.” Theoretical research does not always prove that a conjecture is true. Constructing counterexamples, showing that a certain route is infeasible, and identifying gaps in existing proofs are also valuable. If an evaluation rewards only “solving the problem,” it will overlook negative contributions and problem taste in research.
Design Principles for Research-Oriented Evaluation Sets
A large language model evaluation set aimed at research capability cannot simply concatenate competition problems, textbook exercises, and paper-derived questions. It requires more careful layered design.
1. Domain Coverage and Difficulty Stratification
Problems should cover multiple directions, including algorithms and complexity, proof complexity, computability, randomness and derandomization, foundations of cryptography, quantum computing theory, computational learning theory, and game theory and mechanism design. Difficulty can be divided into course level, paper-understanding level, paper-improvement level, and open-problem level. Different levels correspond to different capabilities: course level tests basic reasoning, paper-understanding level tests technical transfer, paper-improvement level tests proof repair and optimization, and open-problem level tests research judgment.
2. Proof-Oriented Answers and Scoring Rubrics
Answers should not consist only of final conclusions. They should include definitions, assumptions, lemmas, proof steps, and the boundaries of the conclusion. A scoring rubric can be divided into several items: whether the problem is understood accurately, whether the assumptions are complete, whether the proof chain is closed, whether cited results are appropriate, whether counterexamples are valid, and whether conclusions are overextended. For open problems, intermediate results such as “partial progress,” “identification of obstacles,” and “verifiable counterexamples” should be allowed.
3. Combining Machine Checking with Expert Review
For problems that can be formalized, proof checkers can be used to verify key steps. For natural-language proofs, structured parsing can be performed first, followed by blind expert review. Machine checking is responsible for detecting syntax errors, undefined symbols, and obvious logical gaps, while expert review is responsible for judging proof ideas, technical difficulty, and academic value. Only by combining the two can scale and credibility be balanced.
4. Data Contamination and Dynamic Updates
Research-level problems may also be memorized by models. Evaluation sets should include hidden problems, paraphrased problems, problems derived from recent papers, and variants of open problems, and should establish a dynamic update mechanism. For conclusions from public papers, the evaluation should avoid merely testing “restatement of conclusions” and instead test transfer questions such as “does the conclusion still hold after changing the assumptions?” and “which step of the proof depends on a key lemma?”
Insights from the Nine Frontier Model Evaluations
In this type of evaluation, multiple frontier models can provide relatively fluent solutions to course-level and paper-understanding-level problems. Some models can reproduce common proof frameworks and can point out simple gaps. But when it comes to open conjectures and paper-improvement-level problems, model performance is clearly limited. The core problem is not insufficient fluency of language expression, but insufficient reliability of the research chain.
First, models tend to “fill in” proof steps that do not exist. They may skip key reasoning with phrases such as “obviously,” “similarly obtainable,” or “by standard methods,” and these are often exactly the research difficulties. Second, models are insensitive to changes in assumptions. Theoretical research depends heavily on conditions: a conclusion may hold under a randomness assumption, a computational assumption, or a parameter range, and models sometimes ignore the boundary and extrapolate a local conclusion into a global one. Third, models lack stable problem taste. They can generate many directions but have difficulty judging which direction is more likely to produce verifiable progress. Fourth, long proofs are costly to verify. A model can generate dozens of pages of argument, most of which may look reasonable, yet the proof may fail at some intermediate lemma. Fifth, open problems have no standard answers, and models easily make “apparent breakthrough” claims that do not withstand expert review.
Together, these phenomena show that current large models are more like highly efficient research assistants than independent theoretical computer scientists. They can help organize definitions, find similar proofs, generate candidate counterexamples, check symbol consistency, and rewrite proof drafts, but human experts are still needed to propose key lemmas, judge proof routes, and take responsibility for final correctness.
Five Dimensions of Capability Boundaries
From the perspective of engineering and research practice, the boundaries of large models in mathematical research can be summarized along five dimensions.
1. Correctness Boundary
Generating formally correct sentences does not mean generating logically correct proofs. Correctness must be confirmed through stepwise verification and counterexample search.
2. Completeness Boundary
Models tend to provide conclusive summaries and omit proofs of key lemmas. Research-oriented tasks need to enforce output of assumptions, lemma dependencies, and unresolved issues.
3. Verifiability Boundary
Ambiguity in natural-language proofs increases review costs. Models should be encouraged to output structured proofs and, when necessary, convert them into machine-checkable formal fragments.
4. Innovation Boundary
Models can combine existing techniques, but they remain unstable in producing genuinely novel reductions, lower bounds, or counterexample constructions. Innovation evaluation should not only look at textual novelty but at whether verifiable new conclusions are produced.
5. Research Process Boundary
Research is not only answering questions; it also includes topic selection, literature positioning, failure review, and peer review. Models can assist in the process, but they cannot replace the entity responsible for research.
Building an Engineering Evaluation Pipeline
If a team wants to build a similar research-oriented evaluation pipeline, the process can be divided into six stages: problem management, model invocation, proof parsing, machine checking, expert review, and result auditing. The following pseudocode illustrates the structure:
problem = {id, domain, difficulty, statement, assumptions,rubric, known_result, open_flag}answer = model.generate(problem.statement)checks = [parse_structure(answer),check_definition_usage(answer, problem.assumptions),check_proof_steps(answer),search_counterexample(answer),compare_with_known_result(answer, problem.known_result)]machine_score = weighted_sum(checks)if problem.open_flag:human_score = expert_review(answer, problem.rubric, blind=True)final_score = combine(machine_score, human_score)else:final_score = machine_scorelog(problem.id, answer, checks, machine_score, human_score, final_score)
In engineering implementation, an isolated, reproducible, and auditable evaluation environment can be built on general container platforms, object storage, logging services, monitoring, and alerting capabilities provided by Baidu AI Cloud. The key is to retain complete records of problem versions, model outputs, check records, and review comments so that evaluation results are traceable and reproducible. For open problems, blind review by at least two reviewers and a disagreement arbitration mechanism are recommended. For long proofs, structured summaries and key lemma localization should be performed before deep review.
Practical Recommendations for Developers and Research Teams
First, do not equate general benchmark scores directly with research capability. Course problems, competition problems, and research-level problems correspond to different capabilities, and evaluation reports should present them separately. Second, prioritize building proof verification toolchains. Even if the tools can only check symbol consistency, lemma dependencies, and obvious logical gaps, they can significantly reduce review costs. Third, adopt a hybrid model of “model assistance plus expert final review.” Models generate candidate ideas, counterexamples, and proof drafts, while experts judge key steps and final conclusions. Fourth, build a failure case library. In open-problem evaluation, the most valuable information is often not how many questions the model answered correctly, but where and how it failed. Fifth, pay attention to data contamination control. Research-level evaluation sets need hidden problems and dynamic variants to prevent models from obtaining inflated scores through memorization.
Overall, this round of evaluation of frontier large language models’ mathematical and theoretical computer science capabilities reveals a clear boundary: large models can already undertake a large amount of research assistance work, but they still cannot replace human scientists in open conjectures, credible verification of long proofs, and independent research judgment. For developers, the right attitude is neither to exaggerate “AI cracking mathematics” nor to simply deny the value of models. Instead, models should be placed in verifiable, auditable, human-machine collaborative research workflows, making them tools that improve research efficiency rather than the final source of correctness.
