Verifiable Research Evaluation for AI: Measuring the Boundaries of Proof and Construction

    This article examines a shift in AI evaluation from judging whether an answer looks correct to verifying whether a research artifact can be independently checked. It describes the core components of a verifiable research evaluation system, including task registration, a unified submission protocol, deterministic checkers, budget and attempt statistics, and audit and reproduction mechanisms. It also interprets anonymized public results, distinguishes success rate from task coverage, and explains task-specific performance differences. The article outlines evaluation dimensions such as correctness, verifiability, autonomy, efficiency, coverage, robustness, and interpretability; provides an engineering pipeline and structured submission example; and discusses limitations and human-AI collaboration. The goal is to show how auditable, reproducible, and anti-gaming evaluation infrastructure can support independent review of AI-generated proofs, constructions, programs, and parameter improvements.

    1. Background: From “Generating Answers” to “Delivering Reviewable Results”

    In research settings, a plausible-looking derivation produced by a model is not the same as an acceptable research output. Traditional question-answering evaluation focuses on whether an answer looks correct. Verifiable research evaluation focuses on whether a result can be independently reviewed. The core idea proposed in related research is to place outputs from research workflows—proofs, constructions, parameter improvements, and program optimizations—into a single verifiable system. The model does not submit an explanatory paragraph; it submits an object that a checker can read, execute, and verify.

    Such objects typically include intermediate lemmas, final proofs, combinatorial constructions, numerical parameters, program code, or optimized solutions. The value of the system is that it turns “trustworthiness” from a subjective judgment into a deterministic check: whether a construction satisfies constraints on the number of vertices, edges, and forbidden subgraphs; whether a proof can be checked step by step under given rules; whether a program passes test cases and resource limits; and whether a parameter improvement can be independently recomputed and reviewed. Evaluation thus shifts from “text similarity” to “reviewable results.”

    2. Basic Components of an Evaluation System

    A verifiable research evaluation system should include at least the following modules.

    1. Task registration and formal specification. Each task needs a clear input, output, constraint set, budget, success criterion, and reference baseline. Tasks should be versioned, with records of the checker version, random seed, and resource limits they depend on. Otherwise, the same task may become incomparable over time as checking rules change.

    2. Unified submission protocol. Submissions should use a structured format that distinguishes proofs, constructions, programs, and parameters. A submission should include fields such as task identifier, artifact type, artifact payload, and budget usage. A unified protocol makes automated review and auditing possible.

    3. Deterministic checkers. Proof checkers, construction verifiers, program testers, and numerical interval verifiers should be as deterministic, repeatable, and auditable as possible. The checkers themselves need sandbox isolation, timeout control, and resource limits so that untrusted code cannot affect the evaluation environment.

    4. Budget and attempt statistics. Research capability is not only about whether a task is solved, but also about cost. The system should record the number of attempts, number of successes, number of tasks covered, whether intermediate lemmas were used, and token or compute budgets. A high success rate may come from solving only easy problems, so coverage must be observed at the same time.

    5. Audit and reproduction mechanisms. Logs, artifact hashes, checker outputs, and reproduction packages should be preserved completely. Leaderboards should include submission limits, hidden task sets, held-out sets, and anomaly review mechanisms to reduce the risk of overfitting and score gaming.

    3. Interpreting Anonymized Results: Not a Simple Overall Advantage

    Public descriptions show that in one type of verifiable evaluation, Model A ranked first with 71.9 points, and Model B ranked second with 57.4 points. Model A completed 14 of 20 theorems without intermediate lemmas, for a success rate of 70.0%. Model B succeeded 23 times in 60 attempts, covering 9 distinct theorems, for a success rate of 38.3%. The other six agents all scored no more than 11.7%.

    These data suggest at least three points. First, under clear problem statements, sufficient budgets, and verifiable feedback, leading models can already complete some proof and construction tasks. Second, success rate and coverage must be viewed separately: 23 successes covering only 9 theorems indicates that attempt strategy and task generalization remain critical. Third, the leaderboard does not show comprehensive dominance. Model B completed the PCP alphabet reliability task in all three attempts, while Model A did not solve that task. Task specificity means that a single overall score cannot replace a fine-grained capability profile.

    More intuitive results come from constructions and upper-bound improvements that can be independently reviewed. In an extremal graph task with girth 5, one model constructed a graph with 124 vertices and 630 edges that contains no triangles or quadrilaterals, exceeding the reference value of 629 edges used by the task. Another model tightened the upper bound of a combinatorial design parameter from 84 to 82. The significance of these results is not that they “look novel,” but that the submitted objects can be independently verified by a checker.

    4. Key Evaluation Dimensions and Metric Design

    Verifiable research evaluation should cover at least the following dimensions:

    • Correctness: whether the artifact satisfies the formal criteria.
    • Verifiability: whether a checker can independently read and reproduce the conclusion.
    • Autonomy: whether the result depends on external intermediate lemmas or hints.
    • Efficiency: the number of attempts and budget required for success.
    • Coverage: the proportion of successfully solved tasks within the target task set.
    • Robustness: performance after changing random seeds or checker versions.
    • Interpretability: whether failure cases can be traced to specific constraints.

    The following illustrative metrics can be used: success rate = successful attempts / total attempts; coverage = set of successfully solved theorems / target theorem set; cost efficiency = number of successful tasks / budget consumed. Leaderboards should display the overall score, coverage, budget, and failure types together to avoid misleading conclusions from a single score.

    5. Engineering Implementation Sketch

    The evaluation pipeline can be abstracted as a closed loop: “task registration—model run—artifact parsing—checker verification—log audit.”

    1. tasks = load_registry(version)
    2. for task in tasks:
    3. submission = run_agent(
    4. prompt=task.prompt,
    5. budget=task.budget,
    6. seed=task.seed
    7. )
    8. artifact = parse_submission(submission)
    9. ok = checker_registry[task.type].verify(
    10. artifact,
    11. spec=task.spec
    12. )
    13. write_log(task.id, artifact, ok, budget_used)

    A structured submission can be designed as follows:

    1. {
    2. "task_id": "task-001",
    3. "artifacts": [
    4. {
    5. "type": "proof|construction|program|parameter",
    6. "payload": "..."
    7. }
    8. ],
    9. "budget_used": {
    10. "attempts": 3,
    11. "time_ms": 120000
    12. }
    13. }

    In engineering terms, logs can be written to general object storage, metrics can be connected to monitoring and alerting, checkers can run in isolated containers, and a hash and version number can be saved for each submission. This supports both leaderboards and post-hoc auditing and reproduction.

    6. Limitations and Human-AI Collaboration

    Verifiable evaluation is still not complete proof of “autonomous AI research.” First, tasks are usually defined by humans with clear problems, constraints, and acceptance criteria. Models are good at searching for proofs, constructions, and optimization solutions within a feedback loop, but autonomously choosing research directions, judging the value of results, and proposing major open problems still require humans. Second, benchmarks may be overfitted, and checkers may be incomplete. Third, budgets, prompt design, and number of attempts can significantly affect results, so cross-task comparisons must control variables.

    A more reasonable framing is that this type of evaluation is like a topographic map of AI research capability. Models can already complete some proofs, program optimizations, and construction improvements under clear problems, sufficient budgets, and verifiable feedback. However, in direction selection, value judgment, and breakthroughs on major open problems, human-AI collaboration remains the mainstream path. For engineering teams, the priority is not to chase a single leaderboard score, but to build evaluation infrastructure that is auditable, reproducible, and resistant to score gaming, so that every “research output” can be independently reviewed.