Evaluating AI Research Capabilities: From Problem Solving to Verifiable Outputs
This article explores the transition from evaluating AI models based on competition scores to assessing their actual research capabilities through verifiable outputs. It outlines the design principles of verifiable benchmarks, the spectrum of research tasks (including theorem proving, combinatorial construction, and program optimization), and the engineering pipeline required for independent verification. By analyzing test results and typical cases, the article highlights the capability boundaries of current AI models and emphasizes that while AI can assist in specific research tasks, problem selection, value judgment, and tackling major open problems still require human intervention.
1. The Problem: Competition Scores Do Not Equal Research Capabilities
Does achieving high scores on math competition problems mean an AI model possesses research capabilities? This is an issue easily obscured by scores. Competition problems usually have clear conditions, standard answers, and short reasoning chains. Research work, however, involves problem selection, method design, intermediate lemma construction, program experiments, counterexample search, bound improvement, result verification, and value judgment. The two differ in task structure, verification methods, and the cost of failure.
Therefore, a more reasonable evaluation method is not to add more multiple-choice or short-answer questions, but to place the key outputs of the research process into the same verifiable system. Models must not only provide “plausible” conclusions but also submit objects, parameters, programs, or proofs that can be independently reviewed. For example, a graph theory construction task requires submitting vertex and edge sets, which are checked by a verifier for the number of edges, girth, and the presence of specific subgraphs. A theorem proving task requires submitting machine-checkable proof scripts or step-by-step lemmas. A program optimization task requires running in an isolated execution environment and verifying correctness constraints. Only by turning “answers” into “verifiable artifacts” can evaluation better approximate the true requirements of research capability.
Recently, academic teams and industry researchers jointly proposed a verifiable evaluation benchmark for theoretical computer science, incorporating research process outputs such as proofs, constructions, and program optimizations into unified evaluation. The value of this work lies not in creating a new leaderboard, but in providing a “capability topography map.” Under clear problems, sufficient budgets, and verifiable feedback, models can complete partial proofs, program optimizations, and construction improvements. However, they still cannot do without human participation in autonomously selecting research directions, judging the value of results, and solving major open problems.
2. Design Principles of Verifiable Benchmarks
The first principle is “verifiable results.” Research outputs cannot be scored solely based on natural language self-consistency; there must be an independent verifier. Verifiers can be divided into several categories: proof checkers inspect reasoning rules and lemma dependencies item by item; graph construction checkers verify combinatorial properties like vertices, edges, girth, and forbidden subgraphs; program checkers run code in isolated environments to verify input/output and complexity constraints; parameter checkers verify numerical bounds, coding parameters, and upper/lower bound relationships. The existence of verifiers ensures that models cannot score high simply by “fluent expression.”
The second principle is “decomposable tasks and controllable budgets.” Research problems are often open-ended and massive. Evaluation needs to slice tasks into manageable sub-problems and provide budgets for the number of attempts, time, and computing resources. Budget constraints are crucial: without them, models can mask methodological flaws through massive enumeration; if too tight, it becomes impossible to observe the model’s search and correction capabilities under limited feedback. A reasonable approach is to record the artifacts, failure reasons, and improvement trajectory of each attempt, rather than just the final success or failure.
The third principle is “multi-dimensional scoring.” We should not only look at the success rate but also the number of theorems covered, construction quality, the magnitude of bound improvement, the ratio of checkable proofs, and whether failure modes are explainable. One model might succeed repeatedly on a few tasks, while another covers more tasks but has a lower success rate per problem; these represent different capabilities. Evaluation should avoid using a single average score to mask structural differences in capabilities.
The fourth principle is “verifiable feedback without answer leakage.” Models can receive pass/fail information from verifiers, or even local counterexamples, across multiple attempts, but cannot directly obtain standard proofs or final constructions. This allows for observing the model’s iterative correction capabilities while maintaining evaluation fairness.
3. Task Spectrum: Proofs, Constructions, Programs, and Parameter Bounds
Evaluation oriented toward research capabilities should not only contain proof problems. A more complete task spectrum includes at least the following types:
Theorem Proving Tasks: Given definitions, known lemmas, and target propositions, models are required to output checkable proofs. The difficulty lies in intermediate lemma selection, dependency management, and boundary condition handling. Evaluation can track whether proofs are completed without providing intermediate lemmas, to distinguish between the ability of “being fed key steps” and “autonomously completing the reasoning chain.”
Combinatorial Construction Tasks: Models are required to construct graphs, matrices, codes, or set systems that meet specific constraints. The verifier directly checks whether the construction is legal and compares target metrics. For example, extremal graph construction under girth constraints needs to satisfy conditions like the number of vertices, edges, and forbidden subgraphs simultaneously. The advantage of such tasks is objective verification and easy design of difficulty gradients.
Program Optimization Tasks: Given correctness constraints and performance goals, models are required to generate or modify programs. The verifier runs tests in an isolated environment and checks resource usage. Such tasks closely resemble the experimental optimization phase in engineering research.
Parameter Bound Improvement Tasks: Models are required to prove the upper or lower bounds of a certain parameter and compare them with published bounds. Such tasks require models to understand definitions, construct counterexamples, or prove inequalities. The verifier checks the proof chain and numerical relationships.
Reliability/Consistency Tasks: For example, alphabet reliability tasks in complexity theory require models to provide reviewable reductions or constructions. Such tasks often test the model’s precise understanding of formal definitions rather than pattern matching.
These tasks collectively form a spectrum: from short reasoning to long proofs, from constructing objects to improving parameter bounds, from program experiments to formal verification. Evaluation results thus resemble a capability profile rather than a single ranking.
4. Evaluation Pipeline: From Task Description to Independent Review
A reproducible evaluation pipeline can be abstracted into the following pseudocode:
Input: Task set T, Model agent A, Budget B, Verifier set VOutput: Attempt records and scores for each taskfor task in T:spec = parse_task(task) # Definitions, targets, constraints, budgethistory = []for attempt in 1..B.max_attempts:artifact = A.generate(spec, history) # Proof/Construction/Program/Parameter boundresult = independent_verify(artifact, spec, V) # Does not rely on model self-evaluationhistory.append(artifact, result)if result.passed:breakscore = evaluate(spec, history) # Success rate, coverage, quality, improvement magnituderecord(task, history, score)
“Independent verification” is the key. Verifiers should be decoupled from the generation model as much as possible to avoid the model acting as both player and referee. For proof tasks, the verifier checks if each step is derived from rules or permitted lemmas; for construction tasks, the verifier directly traverses the object to check constraints; for program tasks, the verifier runs tests in an isolated execution environment and checks resource limits; for parameter bound tasks, the verifier checks numerical relationships and proof dependencies.
In terms of engineering implementation, the evaluation system requires general capabilities like task orchestration, object storage, isolated execution, logging, and monitoring alerts. By leveraging general services provided by Baidu AI Cloud, such as container platforms, object storage, logging services, and monitoring alerts, a scalable evaluation pipeline can be built: task descriptions and artifacts are persisted, execution environments are isolated, failure causes are traceable, and long-running tasks are observable. The key here is not a specific interface, but making “generate-verify-record-reproduce” a closed loop.
5. Result Interpretation: Not a Total Crush, but Capability Structure Differences
In public evaluations, two leading models scored 71.9 and 57.4 respectively, ahead of other agents; the relevant success rates of the remaining six agents did not exceed 11.7%. Without providing intermediate lemmas, the leading model completed 14/20 theorems, with a 70.0% success rate; another model succeeded 23 times in 60 attempts, covering 9 different theorems, with a 38.3% success rate. These numbers indicate that leading models already possess considerable partial capabilities in formal proof tasks, but the success patterns of different models are not the same: some have high success rates on single problems, while others cover more theorems but at a higher attempt cost.
More importantly, leading does not mean a total crush. On the PCP alphabet reliability task, one model completed all three attempts, while another leading model failed to solve it. This reminds us that research capability evaluation cannot just look at total scores, but also task types, attempt budgets, and failure modes. A model might be strong in proof search but weak in specific reductions or constructions; another might have a lower total score but perform outstandingly in certain sub-fields. Leaderboards should be accompanied by task-level details to avoid misinterpretation.
From the perspective of a capability topography map, current models can complete partial proofs, program optimizations, and construction improvements under three conditions: clear problems, sufficient budgets, and verifiable feedback. They can perform local searches, try counterexamples, adjust parameters, and fill in intermediate steps. However, in terms of “autonomously selecting research directions, judging the value of results, proposing new problems, and solving major open problems,” they still rely on human researchers. The value of the evaluation benchmark is precisely to draw this boundary more clearly.
6. Typical Cases: Construction Improvement and Upper Bound Tightening
Two intuitive cases can illustrate the significance of “verifiable artifacts.”
The first is an extremal graph construction task under girth constraints. The goal requires constructing a graph without triangles and quadrilaterals, while maximizing the number of edges. A model constructed a graph with 124 vertices and 630 edges satisfying the forbidden subgraph constraints, exceeding the 629-edge reference value used for the task. This result is discussable not because the model said, “I constructed a better graph,” but because it submitted the vertex and edge sets, allowing the verifier to independently check: whether the number of vertices is 124, whether the number of edges is 630, and whether it truly contains no triangles or quadrilaterals. Construction tasks reduce natural language explanations to a secondary position and put reviewable objects at the center.
The second is a parameter upper bound improvement task. A model proved A(22,10) ≤ 82, tightening the published upper bound of 84 to 82. Such results also require the proof chain to be checkable: whether definitions align, whether inequality derivations are legal, and whether boundary conditions are handled. Even if the final proof still requires manual review, the machine-checkable parts have significantly improved the objectivity of the evaluation.
These two cases also show that models should not provide “seemingly plausible new answers,” but rather objects, parameters, or proofs that can be independently reviewed. The core of research evaluation is not making the model “sound right,” but making the results “pass verification.”
7. Limitations and Boundaries: Where Humans Are Still Indispensable
First, problem selection still requires humans. Evaluation tasks are defined by others, and models only need to search under given targets. In real research, choosing what problem to tackle, judging whether a problem is valuable, and deciding when to abandon a direction are often more critical than solving the problem. Current evaluations cannot cover this open-ended judgment.
Second, judging the value of results still requires humans. Whether an improvement of 1 edge in a construction or a tightening of 2 units in an upper bound holds academic value requires domain knowledge. Verifiers can only check correctness and cannot replace peer review.
Third, major open problems still require humans. Tasks in evaluations usually have clear constraints and verifiable feedback, while major open problems often lack clear paths or even decomposable intermediate goals. Models can help with sub-problems, but leading the research agenda still requires human researchers.
Fourth, verifiers themselves have boundaries. Proof checkers rely on formal libraries and rule sets, construction checkers rely on constraint modeling, and program checkers rely on test coverage. Areas not covered by verifiers may still leave loopholes. Therefore, evaluation reports should simultaneously disclose verifier capabilities and failure cases.
8. Engineering Implementation Suggestions
If a team wants to build a similar research capability evaluation system, it is recommended to start from the following aspects:
- Define verifiers before tasks. Do not write natural language questions first and then think about how to score them; prioritize clarifying what the verifiable artifact is.
- Keep attempt trajectories. Record the generated artifacts, verification results, failure reasons, and budget consumption for each attempt, to facilitate analyzing whether the model “doesn’t know how” or “failed to search it out.”
- Distinguish coverage from success rate. Simultaneously report the number of successful tasks, attempt counts, number of theorems covered, and single-problem success rates to avoid being misled by a single metric.
- Set layered difficulty. Form a difficulty gradient from short proofs to long proofs, from legal constructions to optimized constructions, and from program correctness to performance optimization.
- Introduce manual spot checks. Passing machine verification does not equate to high academic value; manual spot checks can uncover verifier blind spots and model opportunistic behavior.
- Ensure reproducibility. Task descriptions, verifier versions, execution environments, and random seeds should all be recorded to ensure results can be reviewed.
9. Conclusion
There is a gap between “solving competition problems” and “doing research,” bridged by verifiable outputs, failure iteration, value judgment, and problem selection. The verifiable evaluation benchmark for theoretical computer science places proofs, constructions, program optimizations, and parameter bound improvements into the same review system, providing a more fine-grained topography map for observing AI research capabilities. Current models can already complete partial proofs and construction improvements under clear problems, sufficient budgets, and verifiable feedback, even exceeding reference values on individual tasks. However, autonomously selecting directions, judging the value of results, and solving major open problems still require human leadership. For developers and researchers, the key is not chasing a total score, but understanding the capability structure, verification boundaries, and engineering closed loop, thereby positioning AI appropriately.
