From Competition Math to Research Capability: How Verifiable Benchmarks Test AI Science Skills
Large language models increasingly excel at math competitions, code generation, and complex reasoning. But does solving hard competition problems mean a model can actually do scientific research? This article examines a verifiable evaluation framework that breaks the research process into observable, checkable stages—covering theorem proving, program optimization, combinatorial construction, and result verification—and explains what current results reveal about AI's real research capabilities, where models succeed, and why human researchers remain essential.
In recent years, large language models have made steady progress on mathematical competitions, code generation, and complex reasoning tasks. This raises a natural question: if a model can independently solve high-difficulty competition problems, does that mean it already possesses the ability to conduct scientific research?
The answer is not that simple.
Competition problems typically come with a well-defined problem statement, a clear objective function, and a relatively limited set of solution paths. Research, by contrast, usually starts from an incomplete problem. Researchers must judge what is worth studying, which assumptions are reasonable, whether results are novel, and how to organize scattered observations into verifiable, reusable knowledge. Both activities require reasoning, but research additionally demands problem selection, value judgment, method design, and long-term exploration.
A recent evaluation effort, proposed jointly by AI researchers and theoretical computer science experts, attempts to decompose the research process into multiple observable and verifiable stages, bringing theorem proving, program optimization, mathematical object construction, and result verification into a single evaluation framework. The significance of this framework lies not in simply ranking models by score, but in answering a more critical question: which parts of the research workflow can large models actually complete?
1. Why “Solving Problems” Is Not the Same as “Doing Research”
Traditional math or programming problems usually have a predefined goal. A problem might ask you to prove a specific statement, find a structure satisfying certain constraints, or optimize a program’s runtime. The model only needs to search for solutions within a fixed task space.
Research tasks carry much more uncertainty, typically spanning several layers:
- Problem discovery: identifying unsolved, worthwhile problems from existing knowledge;
- Problem formulation: clarifying the objects, boundary conditions, evaluation criteria, and feasible assumptions;
- Method design: proposing proof strategies, algorithmic approaches, or experimental paths;
- Result verification: confirming that results are logically, programmatically, and empirically sound;
- Value judgment: assessing whether results are general, novel, and practically impactful;
- Knowledge expression: organizing results into a form other researchers can review and reuse.
What large models currently do best is usually localized tasks within the third and fourth stages—especially in settings with clear goals, timely feedback, and well-defined verification mechanisms. By contrast, problem discovery and assessing the value of results still depend heavily on human expertise.
Therefore, when evaluating AI research capability, we should not only check whether the final answer is correct. We should also observe whether the model can produce independently checkable proofs, code, constructed objects, and intermediate conclusions.
2. The Core Idea of a Verifiable Evaluation Framework
The key to such evaluation systems is converting “the model claims it did it” into “an external verification program confirms it did it.”
A reasonably complete evaluation pipeline typically includes four stages.
1. Task Publication
The system presents problems in mathematics, algorithms, or theoretical computer science. Tasks should not specify only the final goal; they must also define input/output formats, allowed tools, resource limits, and verification rules.
For example, for a combinatorial structure construction problem, the task can require the model to output:
- the number of vertices or elements;
- the set of relations or edges;
- the forbidden conditions that must be avoided;
- acceptable size bounds (upper or lower);
- a structured representation that a program can automatically check.
This design prevents models from producing answers that “sound plausible” in natural language but cannot actually be grounded.
2. Model Generation
The model attempts to solve the problem within a constrained time and compute budget. To better mirror real research, the system can allow the model to revise its approach repeatedly, invoke verification programs, check errors, and continue searching.
This stage matters because research rarely produces a final answer in a single shot. Researchers typically go through:
- proposing an initial conjecture;
- discovering counterexamples;
- revising assumptions;
- narrowing the search space;
- re-verifying;
- consolidating into a more robust proof or construction.
If the evaluation allows only one attempt, it cannot reflect the model’s exploratory ability.
3. Automatic Verification
Generated results must go into an independent verifier rather than being judged correct by the model itself. Verifiers can include:
- formal proof checkers;
- program compilation and testing environments;
- constraint satisfaction checkers;
- graph structure analysis programs;
- numerical result re-checking programs;
- automatic comparison modules against known bounds or reference results.
Under this mechanism, the value of a model’s output depends not on how fluent the language is, but on whether the result can be reproduced by external systems.
4. Unified Scoring
Evaluation results should not merely count “number of problems solved.” They should also account for task difficulty, number of attempts, coverage, verification cost, and result quality.
For example, one can separately measure:
- the proportion of theorem proofs completed;
- the number of distinct problems successfully covered;
- the average number of attempts per problem;
- how much a construction improves over the reference value;
- whether complete intermediate lemmas are provided;
- whether results can be stably re-verified by independent programs.
This design helps distinguish “lucky hits” from solution capability backed by stable methodology.
3. What the Results Reveal About Capability
In a comprehensive test suite covering theorem proving, construction problems, and reliability verification, a leading system achieved roughly a 70% theorem completion rate, while another system completed more than twenty tasks across multiple distinct theorems over multiple attempts. Other participating systems had noticeably lower completion rates.
These results show that large models can already handle fairly complex reasoning work on some theoretical tasks. But the data also show that no single model simply dominates across the board.
Some systems rank higher overall yet may fail on certain task types; others score lower overall but succeed repeatedly on a particular structured problem. This indicates that research capability is strongly task-dependent and cannot be summarized by a single aggregate score.
1. Theorem Proving
Theorem proving is one of the easiest research tasks to verify formally. The system outputs proof steps, and an independent checker confirms each step.
Still, theorem proving presents several difficulties:
- models may skip critical lemmas;
- some steps may sound reasonable in natural language but fail in formal logic;
- long proofs are prone to errors in variable scoping, boundary conditions, and quantifier usage;
- models may rely on unproven intermediate conclusions;
- even a correct proof may not carry much theoretical value.
So evaluations should not just ask “did the proof succeed,” but also analyze whether the proof is complete, concise, and generalizable to broader cases.
2. Combinatorial Construction
In an extremal graph problem, a model constructed a graph with 124 vertices and 630 edges containing no triangles or quadrilaterals, exceeding the 630-edge reference value of 629 used in the task.
The significance of this result is that the model did not merely restate a known conclusion—it output a concrete object that a program can check item by item. The verification program can check:
- whether the vertex count is correct;
- whether the edge count reaches the target;
- whether duplicate edges exist;
- whether any forbidden subgraphs appear;
- whether the output format can be read by other programs.
If all these conditions are satisfied, then even if the model cannot provide a complete theoretical explanation, the construction itself may become a starting point for further research. Researchers can analyze the structure’s patterns, attempt to extract a generation method, or prove more general results.
3. Improving Upper and Lower Bounds
In a coding theory task, a model tightened an existing upper bound from 84 to 82. Improvements of this kind are technically valuable because they require finding tighter parameter combinations under complex constraints.
However, a new bound only has scientific significance after a complete proof and independent verification. An unchecked number is at best a conjecture; only an object that a verifier can re-check has a chance of becoming a formal result.
This illustrates an important paradigm for AI-assisted research: the model expands the search space, while humans and verification systems confirm logical validity.
4. Why Partial Success Does Not Equal Full Research Capability
Even if models can complete some proofs and constructions, this does not mean they can autonomously conduct complete research. The main reasons follow.
1. Models Usually Do Not Choose the Problems
Truly valuable research problems are rarely “the hardest problems.” They simultaneously satisfy importance, feasibility, and generalizability. A model can search within a given task, but it may not be able to judge whether a problem deserves months or years of research resources.
2. Models Lack Stable Research Direction Judgment
Research requires a long-term roadmap. For example: can a seemingly isolated construction be generalized to higher dimensions? Can a new proof technique apply to other problems? These judgments require context from the field’s history, existing methods, and future impact—they cannot be measured by the quality of a single answer.
3. Search Budgets Affect Results
A model’s success rate is closely tied to compute resources, number of attempts, and feedback frequency. Allowing multiple generations and repeated verifier calls can significantly raise success probability.
Therefore, experiment reports must clearly state:
- how many attempts were allowed per task;
- whether verification results were visible to the model;
- whether prior answers could be revised;
- how much compute was used;
- whether results could be reproduced in an independent environment.
Otherwise, scores across different systems may not be directly comparable.
4. Correct Results Do Not Mean Adequate Explanations
Research results must not only “obtain an answer” but also explain why the answer holds. A complex structure that passes automated checks may still lack generalizable patterns; a program that passes tests may not state its applicability boundaries.
Future evaluations should therefore track:
- result correctness;
- proof completeness;
- method interpretability;
- result generalizability;
- the ability to analyze failure cases.
5. A Reasonable Division of Labor for AI-Assisted Research
A more realistic path is not to have models fully replace researchers, but to build a collaborative pipeline of “model generation — tool verification — expert judgment.”
What Models Are Good At
- rapidly generating multiple proof ideas;
- searching combinatorial objects and parameter candidates;
- auto-completing programs or formal expressions;
- making local generalizations of existing results;
- discovering counterexamples and edge cases;
- performing preliminary synthesis across large volumes of literature or experimental results.
What Verification Systems Are Good At
- checking formal logic;
- running programs and test cases;
- validating structural constraints;
- re-checking numerical results;
- logging complete trajectories of every attempt;
- guaranteeing reproducibility.
What Human Researchers Are Good At
- selecting problems that truly matter;
- judging research directions and resource allocation;
- interpreting the theoretical significance of results;
- identifying hidden assumptions;
- designing more general research roadmaps;
- taking academic responsibility for final outcomes.
This division of labor reduces the risks posed by model hallucinations and lets researchers focus on problem selection, theoretical abstraction, and result interpretation.
6. Engineering Recommendations for Building Reliable Evaluation Systems
If you want to deploy a similar system in practice, engineering reliability deserves special attention.
First, model outputs should use structured formats rather than relying solely on natural language. For proofs, graph structures, programs, and parameter results, define explicit input specifications.
Second, verifiers must be isolated from the generating model. The model must not modify verification rules or access the verifier’s internal answers. The verification process should log inputs, outputs, logs, error messages, and the runtime environment for later audits.
Third, distinguish “generation failure” from “verification failure.” The former means the model produced no usable result; the latter may indicate malformed output format, resource exhaustion, or invalid logic. Only by classifying failure types can systems be genuinely improved.
Finally, evaluations should publish complete metrics rather than only the top score. For research capability, stability, reproducibility, and failure modes often matter more than a single best performance.
Conclusion
Moving from competition problem-solving to AI-driven research is not simply about making problems harder. It requires decomposing the research process into verifiable, reproducible, and comparable stages.
Current results already show that under clear problem definitions, sufficient budgets, and automatic feedback, large models can complete parts of theorem proving, program optimization, combinatorial construction, and theoretical bound improvement. They are no longer just generating “plausible-sounding” text—they can produce objects that programs and researchers can independently verify.
But this still does not mean models have full research autonomy. Choosing important problems, understanding the value of results, forming long-term research directions, and achieving breakthroughs on major open problems still require human researchers.
A more accurate assessment: large models are becoming efficient explorers and verification assistants in the research workflow, but they remain far from systems that can independently define research goals and take full responsibility for a complete study. What deserves the most attention going forward is not merely whether models can produce answers, but whether they can gradually develop interpretable, reusable, and generalizable research capabilities through rigorous verification, sustained collaboration, and knowledge accumulation.
