Designing a Verifiable Evaluation Framework for AI Research Capabilities: Practice and Implementation

    This article describes a verifiable evaluation framework for AI research tasks, including theorem proving, program optimization, and combinatorial construction. It explains the four core elements of the framework: task specification, budget control, evidence packages, and independent verifiers. It also outlines a task taxonomy, proposes statistical measures that go beyond average scores, and describes how to build an evaluation pipeline. Finally, it discusses what such a framework can and cannot show about AI research ability.

    1. Background: From “Looks Capable” to “Can Be Verified”

    Large language models have shown some ability on tasks such as mathematical proofs, program optimization, and combinatorial construction. Traditional evaluations, however, often fall into two traps. The first is judging only whether a final answer looks correct. The second is treating one successful attempt as proof of stable capability. For research-oriented tasks, the valuable output is not a plausible new conclusion. It is an object, such as a set of parameters, a program, or a proof step, that someone can independently check. An evaluation system must therefore separate generation from verification, and tie each result to its supporting evidence.

    The core of the original research addresses this problem directly. It places outputs from research workflows, such as theorem proving, program optimization, and construction improvement, into one verifiable system. Automated verifiers then check whether each output satisfies the task constraints. The value of this approach lies in breaking AI research capability into task units that can be measured, audited, and reproduced. This article covers framework design, task taxonomy, verifier engineering, statistical conventions, and deployment recommendations. It discusses in general terms how to build a verifiable evaluation system for theorem proving and program construction.

    2. Framework Design: A Closed Loop of Task, Budget, Evidence, and Verification

    A verifiable research evaluation framework should include at least four elements:

    1. Task specification: Define the inputs, constraints, permitted tools, objective function, and verification method. A task should not simply say “prove a conclusion.” It should state which proof forms are acceptable, whether intermediate lemmas may be cited, the verifier version, and the timeout limit.
    2. Budget control: Record the number of attempts, the duration of each attempt, the number of tool calls, and the computing resource limits. Research tasks often require multiple attempts. Without budget records, success rates cannot be compared fairly.
    3. Evidence package: Store every object that can be reviewed for each submission. Examples include proof scripts, construction parameters, program source code, before-and-after performance comparisons, verifier output, and run logs.
    4. Independent verifier: The verifier should not rely on the system’s self-reported results. The evaluator should maintain it and run it in an isolated environment. Only submissions that pass verification earn points. Failed submissions have their failure reasons recorded.

    When these four elements form a closed loop, model output changes from a natural-language answer into an executable, inspectable, and replayable research artifact. This is the fundamental difference between verifiable evaluation and ordinary question-answer evaluation.

    3. Task Taxonomy: Theorem Proving, Program Optimization, and Construction Improvement

    Verifiable research tasks should cover more than one question type. They should form a layered taxonomy.

    The first category is theorem proving. These tasks require either a complete proof or a proof script that a proof checker accepts. The evaluation can vary whether intermediate lemmas are provided, which helps distinguish a system’s performance under strong hints from its performance under weak hints.

    The second category is program optimization. The system must improve performance, reduce resource consumption, or fix defects while keeping the program correct. The verifier judges the result using a test suite, constraint checks, and performance thresholds.

    The third category is construction improvement. In combinatorics or coding theory, for example, the system provides a construction that satisfies given constraints. Its size, edge count, or parameter bounds must beat a reference value.

    Take the task of building an extremal graph with girth 5 as an example. The system must output a vertex set and an edge set, prove that the graph contains no triangles or four-cycles, and maximize the number of edges. The verifier can independently check the edge count, the girth constraint, and connectivity. Similarly, a coding parameter upper bound proof task requires a verifiable mathematical argument or a computer-assisted proof. The verifier checks whether the reasoning steps match the numerical bounds. Probabilistically verifiable proof reliability tasks place more weight on protocol parameters and error rate constraints, which makes them useful for testing whether a system understands a formal definition.

    The key to a task taxonomy is a clear object of verification. Each task should answer four questions: what is submitted, how it is verified, how failures are attributed, and how the budget is recorded.

    4. Statistical Conventions: Success Rate, Coverage, and Attempt Count Together

    Verifiable evaluation should not rely on average scores alone. More useful statistics include:

    • Fully automated scoring: A score out of 100, weighted across tasks. The weights should be published.
    • Per-task success rate: The number of successes divided by the number of attempts. This measures stability.
    • Theorem coverage: The number of distinct tasks solved across all attempts. This prevents inflated results from repeatedly solving the same task.
    • Conditional success rate: Results separated by whether intermediate lemmas are provided, whether tool calls are allowed, and whether time limits apply.
    • Failure type distribution: Verification failures, timeouts, format errors, exhausted budgets, and partially correct answers are reported separately.

    At a high level, one evaluation produced the following results. The top-ranked system scored about 71.9 out of 100, and the second-ranked system scored about 57.4. On theorem proving tasks without intermediate lemmas, the top system solved 14 of 20 tasks, a success rate of about 70.0%. The second-ranked system succeeded 23 times in 60 attempts, covering 9 distinct theorems, a success rate of about 38.3%. The other six evaluated systems each had success rates of at most 11.7%.

    These results show that leading systems already have some proof and construction ability, but their performance varies widely across tasks. On a probabilistically verifiable reliability task, for example, the second-ranked system completed the task in all three attempts, while the top system did not solve it. This mixed picture shows why evaluation should keep fine-grained task profiles instead of reporting a single total score.

    5. Engineering Deployment: Key Modules of the Evaluation Pipeline

    The evaluation pipeline can be split into five modules: task injection, sandboxed execution, verifier orchestration, evidence storage, and audit replay. Task injection delivers the task specification and budget. Sandboxed execution isolates the system under evaluation from the verifier. Verifier orchestration calls a proof checker, constraint solver, test runner, or numerical interval verifier, depending on the task type. Evidence storage keeps submissions and logs. Audit replay reproduces results for a given version.

    The following pseudocode illustrates the flow:

    1. function evaluate(task, system, budget):
    2. submission = system.solve(task.prompt, budget)
    3. evidence = package(submission, task.spec, budget.log)
    4. result = verifier.run(task.verifier_id, submission, task.constraints)
    5. store(evidence, result)
    6. return score(result, task.weight, budget.used)

    At the infrastructure level, general-purpose cloud services can reduce engineering effort. Object storage holds evidence packages and verification logs. A container platform isolates untrusted submissions. A log service records the complete audit trail. Monitoring and alerting track timeouts and anomalies. Baidu AI Cloud offers object storage, a container platform, a log service, and monitoring and alerting, which can serve as the underlying support for such a pipeline. The specific choice of services should still depend on isolation, replayability, and cost control.

    Several details also matter in practice. First, freeze the verifier version and record it with each task. Otherwise, historical results cannot be compared. Second, restrict network access and file system permissions for submissions to prevent contamination of the evaluation environment. Third, keep failed samples. The distribution of failure types often reveals the limits of a system more clearly than successful cases do.

    6. Limits of the Results: Verifiable Capability Is Not Autonomous Research

    This kind of framework can answer a specific question: given a clearly defined problem, a sufficient budget, and verifiable feedback, can a system complete partial proofs, program optimizations, and construction improvements? At a high level, some evaluated systems have already exceeded reference constructions or tightened published upper bounds on specific tasks. This suggests that AI has practical potential in constrained research subtasks.

    The framework cannot directly answer whether a system can independently choose research directions, judge the value of results, or solve major open problems. These abilities involve posing questions, aesthetic judgment, long-term planning, and evaluation by the academic community. Human researchers remain essential for them.

    A more accurate description is this: a verifiable evaluation framework is a map of AI