Verifiable Evaluation of AI Research Capabilities: Building a Trustworthy Benchmark with TCSExam
Evaluating whether an AI model can "do research" is far harder than scoring exam answers, because research outputs are proofs, programs, constructions, or parameters rather than options to match. This article walks through how a joint study—led by the founder of a Chinese large-model team together with Turing Award laureate Andrew Yao and Tsinghua University—built TCSExam, a verifiable evaluation system that folds theorem proving, program optimization, and combinatorial and coding constructions into a single framework where results are judged by independent verifiers rather than by how plausible they look. It covers task decomposition, verifier design, the evaluation workflow, budget accounting, metric interpretation, and cloud engineering implementation, helping R&D teams understand model capability boundaries and turn evaluation conclusions into reproducible engineering practice.
1. Background: Why Evaluating Research Ability Is Harder Than Scoring Answers
Over the past few years, most large-model evaluations have revolved around one question: “Is there a ground-truth answer?” Multiple-choice questions are scored by comparing options, coding problems by running unit tests, and math problems by checking the final value. These methods work well for question-answering tasks. But the output of research is rarely an option—it is a proof, a program, a concrete construction, or a set of parameters. Such outputs cannot be judged by “answer matching”; they can only be judged by independent review.
For a long time, people assessed whether a model could “do research” in one of two ways:
- Subjective expert review: Senior researchers read the model’s written description and judge whether the reasoning is sound and the conclusion credible. This approach is costly, inconsistent, and prone to mistaking “sounds right” for “did it right.”
- Code and math benchmarks: These constrain the model to some degree, but the task granularity differs substantially from real research. A model may pass tests through rote pattern-matching, yet researchers still cannot tell from that whether it can handle genuine research problems.
Generative models also carry systemic risks in their reasoning: a wrong definition, a missed case, or an unstated implicit assumption can make an argument that looks rigorous fall apart in practice. A single unchecked calculation error is enough to invalidate an entire reasoning chain. So in research settings, “the output looks plausible” is worth far less than “the result can be independently verified.”
A study jointly proposed by the founder of a Chinese large-model team and Turing Award laureate Andrew Yao of Tsinghua University is a direct response to this problem. The team designed TCSExam, an evaluation system aimed at theoretical computer science that folds previously scattered research outputs into a single verifiable system, so that different models are tested under unified rules. The evaluation also sends a clear signal: a model’s output is no longer “a seemingly reasonable new answer,” but an object, parameter, or proof that others can independently re-check.
2. Core Paradigm: Replace “Looks Reasonable” with “Can Be Verified”
The starting point of verifiable evaluation is to break research activity into subtasks that a program can automatically judge. These subtasks share three common characteristics:
- Deterministic result: Under given rules, an output is either valid or invalid, with no gray zone.
- Programmatically checkable: The judgment can be written as an independent program that runs repeatedly in an isolated environment and reaches the same conclusion.
- Quantitatively scorable: Once verification passes, output quality can be measured with comparable quantities such as edge count, number of theorems, or numerical bounds.
In industry research on training models with verifiable feedback, “verifiable rewards” are a common core concept: the feedback signal does not come from human scoring but from an automatically executable checking program. Evaluation follows the same logic, but demands stricter independence and reproducibility than training.
Take the “girth-5 graph construction” task in extremal graph theory as an example. The model must output a graph satisfying these conditions:
- It contains no cycles of length 3 or 4 (that is, no triangles and no quadrilaterals);
- Subject to those constraints, the number of edges is as large as possible.
Around this set of constraints, the verifier’s responsibilities are clear:
- Parse the edge set given by the model;
- Check each edge for whether vertex IDs are valid and whether duplicate edges exist;
- Check that all triangles and quadrilaterals are fully avoided;
- Count the edges and compare against the reference value given by the task.
No matter how elegant the model’s derivation looks, a single triangle or quadrilateral on the graph means the attempt scores nothing; conversely, even if the reasoning steps are described briefly, as long as the graph structure itself satisfies the constraints, it remains a valid result. This differs fundamentally from “scoring based on the written description” and embodies the first principle of verifiable evaluation: only outputs that withstand independent re-checking are given credit.
3. Task System: Breaking the Research Workflow into Decidable Units
Verifiable evaluation does not require covering every stage of research activity; it extracts the parts that “a machine can judge.” Using the theoretical computer science areas covered by the paper, four typical task categories emerge.
| Task category | Typical input | What the model must produce | Verification method |
|---|---|---|---|
| Theorem proving | Proposition to prove, existing lemmas and tools | A checkable formal proof | Kernel check by an interactive theorem prover |
| Program optimization | Baseline program, optimization goal and constraints | A modified program | Dual check of correctness and performance |
| Construction and search | Constraints, optimization goal | A concrete object (e.g., a graph, a table) | Line-by-line constraint check with quantitative scoring |
| Numerical bound improvement | Combinatorial structure or coding parameters | New upper/lower bounds plus corresponding constructions | Comparison against published reference values |
These four categories cover the most representative—and most error-prone—parts of research output:
- Theorem proving corresponds to the interactive theorem provers most widely used in mathematics and theoretical research today. The proof is checked step by step by the prover kernel, and any step that does not hold causes the whole to fail.
- Program optimization corresponds to the algorithm implementation and performance tuning common in experimental science. It requires the model to guarantee both functional equivalence and performance improvement, with both automatically verifiable.
- Construction and search corresponds to the research paradigm in combinatorics, coding theory, and similar fields where “the construction comes first, the proof later.” Output quality can be measured with concrete numbers.
- Numerical bound improvement corresponds to the large body of literature work on “tightening a known upper or lower bound,” whose contribution can be judged directly by exact parameters.
When designing tasks, a few engineering principles must also be upheld:
- Unique judgment process: Given the same input, the verifier should reach the same conclusion no matter how many times it runs.
- Traceable reference values: Baseline numbers cited in a task should clearly state their source and year, so the model does not rely on outdated information.
- Distinguish full tasks from hardcore tasks: Basic problems differentiate models’ fundamentals; hard problems probe capability ceilings. Complete tasks like “folding multiple research outputs into one system” are better suited as a hardcore subset for observing how models perform on long-chain work.
4. Evaluation Workflow: A Complete Loop from Task Creation to Review
A reproducible evaluation workflow typically has six stages.
1. Task construction
Researchers abstract a mathematical or program-level description of the problem from published work, define the input, constraints, and acceptance criteria, and fix a known-feasible reference solution.
2. Verifier development
Write an independent verification program for each task category. For a graph-construction task, the core judgment logic can be written as:
def verify_graph(n_vertices, edges, max_triangle=0, max_quadrilateral=0):"""Verify whether a graph object satisfies local constraints and return whether it is valid.n_vertices: number of verticesedges: edge set, each edge is a tuple of two vertex IDsmax_triangle: maximum allowed number of triangles; 0 means no triangles allowedmax_quadrilateral: maximum allowed number of quadrilaterals; 0 means no quadrilaterals allowed"""adj = {v: set() for v in range(n_vertices)}# 1. Check edge-set validity: out-of-range vertex IDs, self-loops, duplicate edgesfor u, v in edges:if u not in adj or v not in adj or u == v:return Falseif v in adj[u]:return Falseadj[u].add(v)adj[v].add(u)# 2. Triangle check: for each edge (u, v), a shared neighbor w forms a triangletriangles = 0for u, v in edges:triangles += len(adj[u] & adj[v])# 3. Quadrilateral check: for each pair of vertices (u, v), a shared-neighbor count c# forms C(c, 2) quadrilateralsquadrilaterals = 0for i in range(n_vertices):for j in range(i + 1, n_vertices):common = len(adj[i] & adj[j])quadrilaterals += common * (common - 1) // 2# 4. Return whether all constraints are satisfiedreturn triangles <= max_triangle and quadrilaterals <= max_quadrilateral
This code deliberately adopts a “check only, do not infer” style. The verifier should not fill in reasoning on the model’s behalf; it should only answer “does the result hold.” This boundary brings two benefits:
- The judgment logic is clear and easy to review;
- The room for complexity optimization is explicit: the number of quadrilaterals in a graph can be computed via shared-neighbor counting, without enumerating every four-vertex combination.
3. Model execution
Organize the task description and constraints into a unified prompt, and let the model respond within a defined compute budget. When needed, the model may be allowed to iterate repeatedly with tools such as code execution.
4. Result recording
Save the raw output, reasoning summary, and resources used for every attempt, so each can be traced later. This is the key step for preventing an “evaluation black box.”
5. Verification and judgment
Send each output to an isolated environment to run the verifier, recording success/failure and the specific reason. The failure reason is itself important diagnostic information: was it a construction error, a calculation error, or a failure to produce a complete object at all?
6. Metric aggregation
Aggregate the per-case results into comparable metrics, and archive them together with the experiment configuration.
The overall workflow can be described by the following pseudocode:
def run_evaluation(task, config):results = []# Run multiple attempts under the configured budget; single or multiple is decided by configfor attempt in range(1, config.max_attempts + 1):# Record prompt version, temperature, and other parameters to ensure reproducibilityprompt = build_prompt(task, version=config.prompt_version)output = model.generate(prompt, temperature=config.temperature)# Run the verifier in an isolated environment to keep untrusted code from affecting the hostverdict = sandbox.run(task.verifier, output, timeout=task.timeout)# Leave a trace for every attempt, including failuresresults.append({"attempt": attempt,"valid": verdict.valid,"reason": verdict.reason, # failure reason or success summary"output": output,})valid_cases = [r for r in results if r["valid"]]# Aggregate: success count, number of distinct tasks covered, best scoresummary = {"success_count": len(valid_cases),"best_score": max([r["score"] for r in valid_cases], default=0),"attempts": config.max_attempts,}return summary
5. Budget Accounting: Why “70%” and “38.3%” Are Not Directly Comparable
When reading evaluation results, the most easily overlooked factor is the compute budget. In TCSExam’s full leaderboard, two participating systems differ markedly:
- One system scored 71.9, ranking first; another scored 57.4, ranking second.
- Without intermediate lemmas provided, the first-place system completed 14 of 20 theorems, a success rate of 70.0%.
- The second-place system succeeded 23 times in 60 attempts, covering 9 distinct theorems, a success rate of 38.3%.
On the surface, 70.0% and 38.3% look directly comparable. But once reduced to the budget question, they actually measure different things:
| Metric | How it is computed | What it means |
|---|---|---|
| Single-attempt success rate | successes ÷ total attempts | Probability of getting it right in one attempt |
| Task coverage | successful tasks ÷ total tasks | How many distinct problems can ultimately be solved |
| Number of attempts | total independent model calls | Compute budget invested |
The first-place system’s 70.0% is closer to a single-attempt success rate. The second-place system clearly would not have reached 23 successes without 60 attempts; covering 9 distinct theorems under a 60-attempt budget shows that repeated exploration can significantly expand the range of problems it can solve. So this kind of comparison should be split into two questions:
- Efficiency question: Under a fixed budget, who can complete tasks with fewer attempts?
- Breadth question: With a sufficient budget, who can cover more types of problems?
If an evaluation report gives only a single percentage, it hides this layer of difference. A rigorous approach is to:
- Disclose the number of attempts allowed per task;
- Report both single-attempt and multi-attempt success rates (the latter resembles the pass@k idea commonly used in code evaluation);
- State whether tool use is allowed and whether intermediate lemmas may be referenced when comparing.
The “no intermediate lemmas” setting in the paper is also worth attention. In real research, complex theorems often depend on existing lemmas, and providing a key lemma effectively lowers the difficulty. Setting a unified boundary on difficulty is a precondition for letting different systems “compete on equal footing.”
6. Interpreting Results: What Models Can Already Do and What They Cannot
TCSExam’s value lies not only in its rankings but in revealing the true distribution of models’ research capabilities.
On proof tasks, leading systems already show strong stability. For a task on PCP alphabet reliability (the soundness parameter in Prover-Delayer games), one system completed all three attempts, while the top-ranked system on the leaderboard actually failed to solve it. This shows that a top-ranked model does not dominate every subfield, and success or failure on fine-grained tasks exposes more specific weaknesses.
On construction tasks, models can already produce “independently verifiable improvements.” In the girth-5 extremal graph construction task, a model produced a graph with 124 vertices and 630 edges:
- The graph contains no triangles and no quadrilaterals;
- Its edge count exceeds the task’s 629-edge reference value.
Unlike algorithm competitions, answers to such research problems are not a single path; any construction satisfying the constraints counts as a “new result.” One extra edge means a genuine piece of research progress.
On numerical bound improvement, models can likewise offer useful contributions. One model produced the result A(22,10) ≤ 82, tightening the upper bound of 84 from the published literature to 82. Here, A(n, d) denotes the maximum number of codewords in a binary code of length n with minimum Hamming distance d. In coding-theory tables, such numbers often update only once in decades, and whether the published upper bound can be pushed one step further directly reflects whether a model has the ability to “continue from existing work.”
The common thread in these results is that the output is not a passage of narrative but an object or proof that a third party can directly check.
At the same time, the paper draws a clear capability boundary: models can already complete some proofs, program optimizations, and construction improvements under conditions where “the problem is well-defined, the budget is sufficient, and feedback is verifiable”; but autonomously choosing research directions, judging the value of results, and completing major open problems still require humans. This suggests researchers should read evaluation conclusions as a “capability terrain map” rather than as “across-the-board superiority.”
7. Engineering Implementation: Building a Reproducible Evaluation Pipeline
From an engineering perspective, the evaluation system above is essentially a data pipeline of “task dispatch → model inference → verification judgment → metric aggregation.” The following recommendations focus on design principles; tool selection should follow the team’s existing stack.
1. Decouple tasks from verifiers
Each task contains at least five types of information: task description, input data, verifier, reference value, and budget configuration. This lets the same scheduling and aggregation logic be reused across different task types. A simplified task description can be written as:
task_id: extremal-graph-girth-5category: constructiondescription: Construct a graph with no triangles and no quadrilaterals, maximizing the number of edgesvalidator: verifiers/graph_girth.pyreference: 629 # edge-count reference value adopted by the taskbudget:max_attempts: 3timeout_seconds: 1800allow_tools: true
2. Run verifiers in an isolated sandbox
Model-generated code or data structures are untrusted input. Verifiers should run in isolated containers with resource limits, including CPU, memory, runtime, and network access. This both prevents accidental damage to the evaluation environment and avoids malicious output affecting results.
3. Leave a trace across the full pipeline
Every evaluation should save the prompt version, model parameters, raw output, verifier return value, and timestamps. When metrics fluctuate abnormally, this information helps quickly locate whether the problem lies in the prompt, the model, or the verifier.
4. Unify foundational capabilities
The evaluation pipeline repeatedly relies on several kinds of foundational capabilities:
- Elastic compute: to quickly spin up large numbers of inference jobs during concentrated evaluation periods;
- Object storage: to hold raw outputs and verification logs;
- Message queues: for task dispatch and smoothing traffic peaks;
- Logging, monitoring, and alerting: to track task backlog and failure rates.
For example, on Baidu AI Cloud, one can combine containers, object storage, message queues, logging, and monitoring products, without building infrastructure from scratch.
5. Manually spot-check judgment results
Automated verifiers are the foundation of the evaluation system, but they themselves may have flaws. It is advisable to periodically sample and manually review verdicts—especially cases where the verifier passes but the result seems “too clever.” The correctness of the verifier determines the trustworthiness of the evaluation.
8. Summary and Action Recommendations
The significance of work like TCSExam lies not in ranking a particular model competition but in establishing a methodology for evaluating research capability:
- Replace “checking plausibility” with “verifying correctness”: The value of research output ultimately depends on whether it can be independently re-checked.
- Measure efficiency and breadth with a unified budget: Success rate and coverage address different questions and should not be conflated.
- Use fine-grained tasks to reveal capability distribution: Different models win and lose on different subfields; a ranking is not the full picture of capability.
- Turn evaluation into reproducible engineering assets: Only by archiving tasks, verifiers, logs, and metrics together can they be continuously reused.
For teams that want to assess their own AI research capabilities, here are a few steps to start with:
- Begin with a capability inventory: Map the research stages your team most often invests in, and pick out the parts that can be judged automatically.
- Build a minimal verifier first: Start with a concrete task, such as graph construction or program optimization, and first ensure the judgment logic is correct and results are reproducible.
- Unify budget accounting: Define the number of attempts allowed per task, whether tool calls are permitted, and whether intermediate results are provided, and state these in the report.
- Finally, establish regression evaluation: Make evaluation a regularly executed process that tracks capability changes from model upgrades, so that introducing “AI-assisted research” is backed by evidence.
