Verifiable Evaluation of AI Research Capabilities: From Proof Generation to Closed-Loop Verification

    This article examines how to evaluate AI agents on research tasks through a unified verifiable framework rather than similarity-based or subjective judgment. It outlines task formalization, separation of submissions from verifiers, explicit budget and attempt tracking, and multi-dimensional metrics. It also interprets results from theorem proving, combinatorial construction, and upper-bound improvement tasks, and discusses the boundary between automation inside verifiable closed loops and human judgment in open-ended research.

    In recent years, AI agents have shown growing potential in tasks such as mathematical theorem proving, combinatorial construction, and program optimization. But a key question follows: how can we tell whether a model’s output actually advances a problem, or merely looks plausible? If evaluation remains limited to similarity of natural-language answers or subjective human judgment, it becomes difficult to distinguish language expression ability from verifiable research output. A more reasonable path is to place the key artifacts of the research process—proofs, constructions, parameters, program modifications, and upper-bound improvements—into a single verifiable system, so that scoring is based on objects that can be independently reviewed.

    I. Evaluation Goal: From “Does the Answer Look Right?” to “Can the Submission Be Checked?”

    Traditional question-answering evaluation focuses on whether the final text is correct, while research-oriented tasks often require multi-step reasoning, trial, and correction. A model may produce a formally elegant proof in which an intermediate lemma does not hold. It may also produce a seemingly novel construction that fails to satisfy the constraints. Therefore, a unified evaluation framework should first specify the submission type for each task: a complete proof, a dependency chain of lemmas, a graph structure, coding parameters, or the result of program optimization. The verifier accepts only checkable objects, not narratives that cannot be reviewed.

    This shift brings two direct benefits. First, scoring becomes more stable. As long as the verifier is clearly defined, the same submission should receive consistent conclusions across multiple verifications. Second, capability profiles become more detailed. Beyond final correctness, it is possible to observe how many attempts a model needs, whether it covers different theorems, whether it can complete a proof independently without intermediate lemmas, and whether it can improve beyond a reference value.

    An illustrative pseudocode example is as follows:

    1. task_definition = {problem_statement, submission_type, verifier, budget}
    2. submission = agent.generate(problem_statement, budget)
    3. verification_result = verifier(submission, problem_statement)
    4. score = {
    5. correctness: whether verification_result passes,
    6. verifiability: whether submission can be independently checked,
    7. attempts: attempts required to reach result,
    8. coverage: successful tasks / all tasks,
    9. improvement: gain over reference value
    10. }

    II. Key Design Principles for a Unified Framework

    First, formalize tasks. Each task should have clear inputs, constraints, and decision criteria. For example, a theorem-proving task needs a theorem statement, allowed lemmas, and proof-checking rules; a construction task needs to specify the number of vertices, number of edges, girth, forbidden subgraphs, and other constraints; an upper-bound improvement task needs verifiable parameter relationships and proof objects.

    Second, separate submissions from verifiers. The model is responsible only for generating submissions, while the verifier is responsible for judgment. A verifier can consist of deterministic checking programs, formal proof checkers, combinatorial constraint validators, or human review protocols. The key is that verification logic is independent of the generative model, avoiding “model self-evaluation.”

    Third, explicitly record budgets and attempts. Research exploration allows trial and error, but evaluation must report the cost of trial and error. Looking only at the best answer can overestimate capability, while looking only at single-attempt success rates can underestimate exploratory systems. Therefore, evaluation should report attempts, successes, covered tasks, and success rates together.

    Fourth, use multiple metrics together. A composite score is suitable for horizontal comparison, but it cannot replace fine-grained metrics. Theorem completion rate reflects proving ability, attempt success rate reflects stability, the number of covered theorems reflects generalization range, construction improvement and upper-bound tightening reflect whether a problem is truly advanced, and verifiability determines whether a result can enter a research workflow.

    III. Task Families and Result Interpretation: Capability Distribution Is Not Monotonic

    In a public evaluation, the composite scores on the full leaderboard showed a clear gradient: the top system scored 71.9, the second scored 57.4, and the remaining systems were farther behind. More noteworthy, however, are the fine-grained results. In theorem-proving tasks, the top system completed 14 of 20 theorems without being provided intermediate lemmas, for a success rate of 70.0%; the second system succeeded 23 times in 60 attempts, covering 9 different theorems, for a success rate of 38.3%; the other six tested agents all had success rates no higher than 11.7%. This indicates that, under clear problems, sufficient budgets, and verifiable feedback, some systems are already capable of independently completing part of the proof steps, but stability and coverage still differ significantly.

    Construction tasks better demonstrate the value of “verifiable output.” In an extremal graph construction task with girth 5, one system constructed a graph with 124 vertices and 630 edges that contains no triangles or quadrilaterals, exceeding the task’s reference value of 629 edges. This is not a new answer at the language level, but a graph structure that can be independently reviewed by a constraint validator. Another result tightened an upper bound for a coding parameter from 84 to 82, which is likewise a verifiable mathematical improvement. Such results indicate that a model’s value may not lie in dominating every task, but in construction search and proof improvement on specific tasks.

    At the same time, the evaluation showed localized capability reversals. For example, on the alphabet reliability task for probabilistically checkable proofs (PCP), the second system completed all three attempts, while the top system did not solve that problem. This suggests that a high composite score does not mean strong performance on all tasks; capability distribution may show clear task dependence. Evaluation should avoid treating a single total score as the entire conclusion and should instead output a “capability landscape.”

    IV. Capability Boundaries: Automation Inside Verifiable Closed Loops, Humans for Open Research

    These results together outline a boundary. Models can already complete some proofs, program optimizations, and construction improvements under conditions of clear problems, sufficient budgets, and verifiable feedback. But autonomous selection of research directions, judgment of result value, and formulation of major open problems still require humans. The reason is that a verifier can judge whether a submission satisfies constraints, but it is difficult for a verifier to judge whether a problem is worth studying, whether a proof route has long-term value, or whether an improvement has theoretical significance.

    In addition, verifiable evaluation must guard against reward hacking. A model may target verifier loopholes and generate submissions that pass checks but are meaningless. Therefore, verifiers should use multiple checks, randomized testing, isolated evaluation sets, and retained manual spot checks and audit logs. For developers, implementation should follow the principles of “verification first, task layering, transparent budgets, and human-machine collaboration”: first define decidable verifiers, then design task difficulty; fix attempt budgets and report success rates; record submissions, verification logs, and budget consumption; and assign humans to topic selection, value judgment, and result interpretation.

    Overall, the significance of a unified verifiable evaluation framework is not to create an absolute ranking, but to turn the research output of agents into reviewable objects, thereby identifying capabilities and boundaries more accurately. It shows progress in automated reasoning on closed verifiable tasks, and it also reminds us that truly major research breakthroughs still require humans to pose problems, design verification criteria, and judge value.