AI Research Ability Evaluation: From Problem Solving to Verifiable Outputs
This article explains why AI research ability should not be judged only by final answers in competition-style tests. It describes a benchmark design in which proofs, constructions, optimized programs, and bound improvements are checked by independent verifiers. It also reviews reported results, their limits, and practical guidance for building such evaluations.
High scores on competition problems are often taken as a sign of a model’s reasoning ability. Research, however, does not consist of picking the correct answer from a fixed set of options. Research requires asking testable questions, choosing methods, constructing objects, proving properties, optimizing programs, and surviving peer review. Therefore, assessing whether a model can do research cannot rely only on its final natural-language answer. Proofs, constructions, parameter improvements, and program execution results should be brought into one verifiable system. A public research effort offers a representative approach: turn the outputs of a research process into objects that can be checked independently, and judge success with verifiers rather than subjective impressions. Only then can evaluation results move from “looks plausible” to “can be verified.”
1. Why Verifiable Research Output Evaluation Is Needed
Traditional benchmarks usually use whether the final answer is correct as the only signal. For multiple-choice and short-answer questions, this design is efficient and objective. Research tasks, however, often involve long workflows, many intermediate products, and open solution spaces. A model may produce a fluent proof that omits a key lemma. It may also propose a construction that looks novel but fails the boundary conditions. If an evaluation reads only the final text, reviewers can be swayed by how well something is written, and it becomes hard to tell whether the model made a verifiable research contribution.
The core of verifiable research output evaluation is a closed loop: problem definition, candidate output, independent verification, and statistical aggregation. Candidate outputs can be formal proofs, constructive graph objects, program parameters after optimization, improved upper bounds on codes, or counterexamples. Verifiers perform formal checks, numerical validation, program execution, or property checks, depending on the nature of the task. An output counts as successful only if it passes verification. In this way, evaluation shifts from matching answers to checking objects, which is closer to the principle of reproducible verification in research.
2. Key Design Elements of the Evaluation Framework
First, task families should cover different research actions. Theorem proving tests logical decomposition and lemma selection. Constructive extremal problems test search and property satisfaction. Program optimization tests code modification and performance constraints. Upper-bound improvement tests mathematical derivation and parameter tightening. Reliability verification tests boundary conditions and the ability to find counterexamples. Different task families require different abilities from a model, so a high score on one task does not represent overall research ability.
Second, the verifier is the foundation of the framework. A verifier needs a clear input and output contract: what object the model submits, which properties the verifier checks, and what diagnostic information it returns on failure. For proof tasks, the verifier can check formal steps or key lemmas. For construction tasks, it can verify the number of vertices and edges, or check for forbidden subgraphs. For program optimization, it can run tests and check constraints. An incomplete verifier produces false positives and false negatives, so verifiers need versioning, auditing, and manual spot checks.
Third, budgets and attempt counts must be recorded fairly. Research is not a single generation step. A model may make several attempts within a fixed budget. An evaluation should record the maximum number of attempts, time budget, token budget, and tool-use limits. It should then report success rate, task coverage, best improvement, and average number of attempts. Looking only at single-attempt success rates underestimates search-based ability, while looking only at the best result hides stability problems.
An abstract evaluation pipeline can be written as follows:
def run_benchmark(agent, tasks, budget):records = []for task in tasks:attempts = []for i in range(budget.max_attempts):artifact = agent.produce(task) # Generate a proof, construction, or programok, detail = verifier.check(task, artifact) # Independent verificationattempts.append({"ok": ok,"detail": detail,"attempt": i + 1})if ok:breakrecords.append(summarize(task, attempts, budget))return aggregate(records) # Aggregate success rate, task coverage, improvement margins, etc.
The key to this pipeline is not the code itself but independent verification and transparent budgets. Without a verifier, evaluation easily degrades into text review. Without budget records, comparisons between approaches lack a common basis.
3. Capability Patterns in Public Evaluations
Public evaluation results show that agents differ considerably on research-style tasks. On the composite leaderboard, the leading approach scored about 71.9, and the runner-up scored about 57.4. On theorem-proving tasks, the leading approach completed 14 of 20 theorems without access to intermediate lemmas, a success rate of about 70.0%. The runner-up succeeded 23 times in 60 attempts, covering 9 distinct theorems, for a success rate of about 38.3%. Several other agents did not exceed 11.7%. These figures suggest that some models can complete part of the proof work when goals are clear, budgets are sufficient, and feedback is verifiable. However, stability and coverage still show a clear long tail.
The results are also not a uniform sweep. In one public case, an agent completed an alphabet reliability task related to probabilistically checkable proofs in all three attempts, while the leading overall approach did not solve that problem. This indicates that research ability is not a single scalar. Different tasks depend differently on search strategy, formalization skill, numerical intuition, and sensitivity to boundary conditions. A high composite score does not mean that every subtask is strong. Evaluation reports should keep task-level distributions instead of publishing only one total score.
The value of verifiable outputs is especially clear in construction and improvement tasks. In one public case, an agent constructed a graph with 124 vertices and 630 edges that contains no triangles or quadrilaterals. This exceeds the 629-edge reference construction used for the task. In another case, an agent tightened the upper bound for a coding parameter (22, 10) from 84 to 82. These results matter not because the descriptions are novel, but because the objects, parameters, or proofs can be checked independently. Research evaluation should encourage small but verifiable improvements rather than focusing only on grand claims.
4. Boundaries: What Models Can Do and What They Still Cannot
These cases suggest a capability map. When a problem is clearly defined, the budget is sufficient, and verification feedback is clear, models can complete part of the proof work, program optimization, and construction improvement. They are good at trying, combining, and iterating within a given search space. Still, choosing research directions, judging the value of results, and identifying open problems that really matter still require people. Research topic selection involves more than finding hard problems. It also means judging whether a problem is worth pursuing, whether existing tools are sufficient, and whether the risk of failure is acceptable. These judgments are hard to formalize fully, and a single benchmark score should not replace them.
In addition, the scope of a verifier determines how far its results can be trusted. If a verifier checks only the final conclusion, a model may skip key steps. If task content leaks into training data, scores are inflated. If budgets differ across runs, comparisons are unfair. Engineering teams should therefore version their tasks and verifiers, log all attempts in full, check for data contamination, and bring in human experts to review the verifier’s boundaries and the value of the results.
5. Practical Recommendations for Engineering Teams
For developers building research capability benchmarks, the priority should be domain-specific verifiers, not a large collection of natural-language questions. Each task type should define a verifiable output format, verification rules, and failure diagnostics. Evaluation reports should present the composite score, task-level success rates, coverage, number of attempts, and improvement margins together, so that a single metric does not mislead readers. In enterprise R&D settings, large models can be positioned as proof assistants, construction searchers, and program optimizers. They can work within goals and verifier constraints set by humans, rather than replacing research decisions.
Human oversight should run through topic selection, verifier design, result interpretation, and value judgment. Models expand the search space, while humans define what counts as success and which directions are worth pursuing. Only by combining verifiable outputs, transparent budgets, and expert review can research capability evaluation be both engineering-practical and protected from being misled by high scores on competition problems.
