Verifiable AI Research Evaluation: Measuring Models by Evidence, Not Answers
Frontier models keep setting new records on competition math, coding problems, and software engineering benchmarks. But solving problems and doing research are separated by a meaningful gap: the former usually has a clear objective and a unique answer, while the latter has no ready-made answer and demands testable conjectures, iteration through failure, and results that survive peer scrutiny. This article walks through a verification-first evaluation approach drawn from theoretical computer science, where outputs are theorems, improved bounds, and constructed objects. It explains which tasks can be verified, how the verification pipeline keeps results trustworthy, how to separate reproduction from genuine improvement, and how to read the resulting capability landscape, including concrete examples of model output that beat published reference values. It closes with a five-layer architecture and six practical recommendations for building your own verifiable evaluation system.
$$## 1. Introduction: Passing Exams Is Not Doing Research
Large models keep posting new records on competition math, coding challenges, and software engineering tasks. Yet there is still a distance between solving problems and doing research. The first usually has a well-defined goal and a single correct answer. The second typically has no ready-made answer: the researcher must propose testable conjectures, iterate through failure, and produce results that hold up under peer review.
Measuring research ability is itself difficult. Peer review is slow, expensive, and subjective. Traditional academic benchmarks judge correctness against fixed answers, so they cannot test whether a model can produce a new result.
A group of industry researchers and Turing Award laureates proposed a way forward: place research artifacts from theoretical work into a single machine-verifiable evaluation system. Research in theoretical computer science is a good sample for this approach — the goal is not to compute a number but to prove theorems, improve known bounds, or construct extremal objects. Using this system as a lens, three questions come into focus:
- What kinds of tasks can be made verifiable?
- How does the verification pipeline guarantee trustworthy results?
- How far are the scores from genuine research capability?
2. Why Verifiability Is the Core of Research Evaluation
2.1 Once Public Benchmarks Saturate, Harder Tasks Are Needed
Competition math and mainstream coding benchmarks are now heavily represented in training data, and scores are approaching saturation. A model’s solution may rely partly on memorized training corpora, which says little about its ability to reason independently or construct solutions when facing something new.
2.2 The Value of a Research Output Lies in Whether It Holds
In principle, reviewers check whether a proof is rigorous and whether a construction satisfies every constraint. If an output can be certified by a machine, the work shifts from depending on expert judgment to depending on program judgment. Typical checkable artifacts include:
- graphs that satisfy given forbidden-subgraph constraints;
- codebooks with known parameters;
- reductions or combinatorial rules that satisfy their definitions;
- programs that are semantically equivalent and faster;
- scripts that a proof assistant can check.
For these tasks, correctness is machine-decidable and the strength of the result is quantifiable (more edges, a tighter upper bound, a shorter runtime). Because verification does not depend on a reviewer’s subjective opinion, different models can be compared on the same ruler.
2.3 Evaluate Improvement, Not Just Reproduction
If a task only asks “what is this known result?”, a correct answer may simply mean the model recalled the literature. A more discriminating design gives the model a problem instance and the best published result, and asks it to do better. Evaluation shifts from knowledge retrieval to real search, construction, and argumentation.
3. Five Design Principles for Verifiable Research Evaluation
The task design of this system can be distilled into five principles:
- Auditable deliverables. The model does not submit an argument; it submits a machine-readable object or proof, together with the verification method and the key information.
- Separate reproduction from improvement. Being correct is only a passing grade; a metric that exceeds the known best result counts as a real contribution.
- Controlled and comparable budgets. The number of attempts, compute time, and resource allowance are fixed in advance, so “throwing compute at it” cannot mask differences in ability.
- Separate process from outcome. The main leaderboard permits intermediate lemmas and examines the full chain of reasoning; a second setting provides no intermediate lemmas and requires the model to produce a verifiable final result directly within a limited number of attempts.
- Anti-contamination and anti-gaming. Task instances are parameterized and perturbed, held-out instances are kept private, and anomalous patterns are audited.
4. Breaking the Task System Down: Turning Theory into Verifiable Instances
The tasks come mainly from a few areas of theoretical computer science, with one thing in common: the output can be strictly checked by a program.
4.1 Combinatorial Constructions: Extremal Graph Theory
Extremal graph theory studies how large a parameter can be while certain constraints hold. A common evaluation task is graph construction under a girth constraint. Girth is the length of the shortest cycle in a graph; girth at least 5 means the graph contains neither triangles nor quadrilaterals. The model must submit:
- the vertex set and edge set, or a program that deterministically generates the graph;
- the in-memory graph structure when it runs;
- a verification script output as-is.
The checker inspects every pair of vertices for cycles of length 3 or 4, then counts edges. At this scale (something like a hundred-plus vertices and six hundred-plus edges), brute-force checking becomes expensive, and an inefficient implementation times out.
4.2 Coding Theory: Codeword Construction and Upper-Bound Proofs
Given a code length and a minimum Hamming distance, the task is to find the maximum number of codewords in a binary code, written A(n, d). Research on such values generally follows two paths:
- Constructing a lower bound: provide a concrete table of codewords; the checker verifies equal lengths, pairwise distance at least the threshold, and no duplicates.
- Proving an upper bound: use techniques such as linear programming bounds or weighted counting to show that no larger code exists.
The two paths complement each other and map directly onto the two kinds of research output: constructing an object and giving a proof.
4.3 Probabilistically Checkable Proofs: Alphabet Size and Soundness
PCP (probabilistically checkable proof) theory shows that a verifier can read only a constant number of symbols from a proof and still decide the truth of a statement with high probability. Two parameters matter:
- Alphabet size: how large the range of each symbol is; smaller values require shorter queries.
- Soundness: the minimum probability with which the verifier rejects a false statement.
The related tasks typically ask for reductions or combinatorial rules that achieve a smaller alphabet and lower soundness loss while preserving completeness. The checker runs the rule on a set of concrete inputs, checks that the defining properties hold, and reports the actual parameters.
4.4 Algorithm and Program Optimization
Given a problem instance and a reference implementation, the model must submit:
- a program that is semantically equivalent and faster;
- correctness test cases;
- a reproducible timing method.
The platform compiles and runs everything in a controlled sandbox, computes the speedup from the median of multiple runs, and confirms with randomized tests first that the result has not changed. Timing is highly sensitive to hardware and system state, so the environment must be pinned, samples must be taken repeatedly, and noisy runs must be discarded.
4.5 Interactive Proofs
Using an interactive proof assistant, the model writes proofs in a formal language, and the proof kernel checks each inference step by step. This differs from “give the final answer” in a math contest: formal proofs do not allow skipped steps, and any gap anywhere causes the whole thing to fail.
5. The Verification Pipeline: From Submission to Trustworthy Result
A trustworthy evaluation relies on more than a single verification script — it relies on a full engineering pipeline.
5.1 Sandbox Isolation
All code runs in an isolated container: no outbound network access, task data mounted read-only, CPU and memory limits, and a hard timeout. This protects the evaluation platform from malicious or runaway code and keeps runtime conditions identical across models.
5.2 Platform-Side Re-Verification
Verification is performed by the platform, not taken from the model’s self-reported results. For the graph task, the platform ignores the edge count the model reports and rescans the adjacency lists to recompute it. Structural validity checks such as “all edges fall within the vertex ID range” and “no duplicate edges or self-loops” also count toward the result.
5.3 Budget Accounting and Uniform Adaptation
Every attempt records token usage, wall-clock time, and the number of tool calls. Because tool-calling formats differ across models, a uniform adaptation layer wraps them so that every model is measured under the same budget. When the budget runs out, attempts stop, and incomplete attempts do not count toward the success rate.
5.4 Full-Trace Logging
Every attempt’s input, complete output, execution log, and verification result are stored together for later review. For structured objects like graphs and codes, the platform uses JSON as the exchange format. The snippet below illustrates the interface for a task definition and a single verification run:
from dataclasses import dataclass, fieldfrom typing import Callable@dataclassclass VerifyResult:ok: bool # passed all correctness checksmetric: float | None # quantifiable outcome: edge count, code size, speedup...evidence: str # checker output, e.g. number of violations, actual parametersdetail: dict = field(default_factory=dict)@dataclassclass Task:task_id: strprompt: strspec: dict # machine-readable task parameters, e.g. n / forbidden cycle lengthreference: float # reference metric from the literature or adopted by the taskverify: Callable[[dict], VerifyResult]def run_attempt(task: Task, budget: dict, submission: dict) -> dict:"""Platform-side execution: run in sandbox -> re-verify -> record metric and evidence."""result = task.verify(submission) # metrics are always computed by the platform, never self-reportedreturn {"task_id": task.task_id,"solved": result.ok,"metric": result.metric,"improved": result.ok and result.metric > task.reference,"evidence": result.evidence,"budget": budget,}
6. Engineering Against Contamination and Gaming
One problem must be faced directly: once evaluation instances are public, they can be swept into the training data of later models. Some of the metric values also appear in the public literature, so a model may “remember the answer” rather than reason on the spot. The following measures are used in practice:
- Parameter perturbation: randomize instance parameters while preserving the problem structure, so memorized answers cannot be applied directly.
- Held-out instances: run periodic checks on a small private set and compare against public-instance scores to detect memorization.
- Improvement tasks: for these, memory can only help a model reproduce the known result; beating the reference still requires genuine search and construction.
- Process audits: check whether the reasoning cites constraints that are real and verifiable, whether there is speculative behavior aimed at the reference value, and whether the model bypasses the verification logic by submitting the “expected value.”
- Sampling consistency: make multiple independent attempts on the same task and check whether success correlates strongly with the random seed.
7. Scoring and Statistics: Two Metrics That Must Stay Separate
Evaluation results have to answer two questions: how many tasks can the model solve, and how stable is it when it solves them.
7.1 Solve Coverage and Attempt Success Rate
- Solve coverage: the share of distinct tasks solved at least once, reflecting breadth of ability.
- Attempt success rate: the share of all attempts that pass verification, reflecting the reliability of a single output.
The two must be reported separately. Suppose two models have coverage of 70% and 38.3%. At the same success rate, the second needs roughly 1.8x the expected number of calls — a significant difference in engineering cost.
7.2 Main Leaderboard and the No-Intermediate-Lemma Setting
The setting that “allows intermediate lemmas” is close to real research: a large problem is decomposed into lemmas, proved step by step, which makes errors easy to localize. Some models have high error rates partly because the answer is compressed into a very short output, without intermediate reasoning steps to support each step.
The “no intermediate lemmas” setting is closer to black-box search: the model can only guess repeatedly within a limited number of attempts until it produces a result that passes verification. Combining both settings tests:
- structured reasoning ability;
- end-to-end search and construction ability.
7.3 Confidence Intervals at Small Sample Sizes
With 20 problems and 60 attempts, the sample is small, and comparing a gap of a few percentage points directly can mislead. Take “14/20” and “23/60” as an example; the Wilson confidence intervals are:
import mathdef wilson_interval(k: int, n: int, z: float = 1.96) -> tuple[float, float]:"""Confidence interval for a small-sample proportion; k = successes, n = total attempts."""if n == 0:return (0.0, 0.0)p = k / ndenom = 1 + z**2 / ncenter = (p + z**2 / (2 * n)) / denomhalf = z * math.sqrt(p * (1 - p) / n + z**2 / (4 * n**2)) / denomreturn (max(0.0, center - half), min(1.0, center + half))print(wilson_interval(14, 20)) # roughly (0.482, 0.862)print(wilson_interval(23, 60)) # roughly (0.273, 0.508)
The two intervals do not overlap, so the difference is statistically significant. But one is based on 20 observations and the other on 60, so their interval widths differ by nearly a factor of two. Even the leader on the main leaderboard carries more uncertainty in its conclusion. More instances and more repetitions are the only way to make the conclusion more reliable.
8. Reading the Results: A Terrain Map of AI Research Ability
8.1 The Overall Picture
The evaluation shows a clearly stratified picture:
| Rank | Evaluated subject | Main leaderboard score | Attempt success rate | Notes |
|---|---|---|---|---|
| 1 | Model A | 71.9 | 70.0% (14/20) | Completed under the no-intermediate-lemma setting |
| 2 | Model B | 57.4 | 38.3% (23 successes in 60 attempts, covering 9 distinct tasks) | Received extra points for reproducibility of results |
| 3 and below | The remaining six evaluated agents | — | No more than 11.7% | Includes general models and agent systems built specifically for research workflows |
Three things stand out:
- The gap between first and second place is about 14.5 points, while there is a clear break between second place and the bottom six. The ability distribution is not a smooth continuous curve but “concentrated at the top, sparse in the tail.”
- Scores do not map simply onto whether an “agent system” was used. A system that chains retrieval, reasoning, and verification into a pipeline is not necessarily better than a single more capable model. Engineering amplifies model ability; it cannot conjure reasoning ability that is not there.
- At a success rate of roughly 40%, covering 9 distinct tasks means the model already has a “repeatably callable” research output capability. A single lucky success is worthless; producing verifiable results consistently across tasks is the threshold for being usable in engineering.
8.2 Strength Is Not Across the Board
Model A leads on most dimensions, but on the “alphabet soundness” task it failed all three attempts, while Model B completed all three. This shows that:
- a model’s capability profile is uneven and may have clear weaknesses in specific subfields;
- different models emphasize different things in “breadth of exploration” versus “depth at a single point”;
- an evaluation should not produce only a ranking, but also a per-dimension capability profile, so users can pick models per task type.
8.3 What Really Counts Is a Verifiable Improvement
The results that carry real weight are the ones that exceeded the reference values.
Extremal graph construction. On a graph construction task with a given girth constraint, an evaluated model produced a graph with 124 vertices and 630 edges containing neither triangles nor quadrilaterals, beating the task’s reference value of 629 edges.
- Construction process: first derive the theoretical upper bound from the constraints to estimate a feasible parameter range; then generate candidate graphs via computer-assisted search and improve them with edge-swap local optimization; finally check the cycle structure pair by pair to confirm that no newly added edge creates a cycle of length 3 or 4.
- Sense of scale: 630 edges spread over 124 vertices means each vertex is adjacent to about 10.2 others on average. Adding one more edge requires satisfying constraints across roughly 7,000 vertex pairs, and even a single triangle or quadrilateral would make the result invalid.
- Basic self-consistency: classic theory gives a simple constraint for such graphs — if the minimum degree is at least d, then the number of vertices n is at least d² + 1. Here n = 124, so the minimum degree can be at most 11; the average degree is about 10.16, which suggests the construction is already close to the theoretical limit at this vertex count.
Improved coding bound. On the question of how many codewords a binary code of length 22 with minimum Hamming distance 10 can have, an evaluated model gave a proof of no more than 82, tightening the published upper bound of 84 to 82.
- Background: values like this usually exist only as an interval between lower and upper bounds. The lower bound comes from a construction; the upper bound comes from arguments such as linear programming bounds or weighted counting. Tightening an upper bound usually means introducing finer analytical techniques.
- Verifiable proof: this kind of proof does not require large-scale enumeration but can be re-checked step by step with symbolic computation, which fits the “independently verifiable” requirement exactly.
Constructions and proofs correspond to two kinds of research output. What the models produced was not a “seemingly reasonable new answer,” but an object that can be independently re-checked or an argument that can be audited. That is the most instructive part of this evaluation: measuring research ability ultimately comes down to evidence, not narrative.
9. Engineering Takeaways: Building Your Own Verifiable Evaluation System
9.1 A Five-Layer Architecture
As illustrated above (described in text), a reusable evaluation system can be split into five layers.
- Task layer: each task consists of a machine-readable specification, a prompt, a reference metric, and a verifier, with metadata registered uniformly.
- Execution layer: container platforms provide isolated sandbox runtimes that scale elastically on demand; for lightweight, high-frequency work such as verification, event-driven serverless functions can handle the load.
- Verification layer: the platform re-runs the checks, emits structured results, and covers three parts: correctness checking, metric computation, and evidence retention.
- Scheduling layer: a message queue decouples “submit — verify — score,” supports retries on failure and peak shaving; long-running tasks can be split into parallel subtasks.
- Data layer: task versions, model versions, budget configurations, raw artifacts, and scoring results are archived together. Object storage can hold model outputs and logs with versioning enabled, so any evaluation run is traceable and replayable.
On Baidu AI Cloud, these generic capabilities (container platforms, message queues, object storage, logging services, and monitoring/alerting) can be combined as a reference architecture for a self-built evaluation system.
9.2 Six Practical Recommendations
- Start from verifiability when choosing tasks. Prefer problems with objective metrics and affordable checking cost; verification time should be far shorter than generation time, or the evaluation will be slowed down by verification itself.
- Maintain verifiers like unit tests. The correctness of the verifier directly determines how trustworthy the conclusions are, so ship positive and negative test cases with it and keep running regression tests.
- Score process and outcome separately. Look at the final deliverable, but also at whether intermediate steps hold up.
- Always attach confidence intervals at small sample sizes. Report coverage and success rate together with a measure of uncertainty, and avoid drawing conclusions from a single number.
- Keep the task pool fresh. Introduce new instances regularly and retire saturated tasks, so difficulty rises in step with technical progress.
- Design tasks by capability tier. Distinguish levels such as “knowledge retrieval,” “local improvement,” “end-to-end construction,” and “open-ended exploration.” The higher the tier, the harder machine verification becomes, and the more you need to combine expert review with formal verification.
10. Limitations and Outlook
This evaluation has boundaries, and they need to be understood clearly:
- Gaming the reward. A system that optimizes only for the verified metric may produce results that “pass the check but mean nothing,” which calls for sampled human expert review as a supplement.
- The tension between openness and contamination. Once instances are public, contamination risk exists; but keeping them entirely private damages reproducibility. A trade-off is unavoidable.
- Budget dependence. Scores measured under a controlled budget do not represent capability without budget limits, and cross-model comparisons must align the budget definition.
Back to the original question: models can already complete some proofs, program optimizations, and construction improvements when the problem is well defined, the budget is sufficient, and feedback is verifiable. But choosing research directions autonomously, judging the value and significance of results, and cracking major open problems still depend on human judgment and leadership.
For developers and enterprises, the more realistic path is to treat the model as a tireless research assistant:
- keep measuring its current capability boundary with verifiable tasks;
- wire verifiable outputs into automated pipelines;
- concentrate human effort on posing questions and judging value.
As the evaluation system matures, this “terrain map of research ability” will keep getting sharper, helping us use AI in the right places.
