Evaluating AI Research Capability: Building and Deploying Verifiable Systems

    A practical guide to measuring AI research capability through automatically verifiable systems. It explains how to bring theorem proving, program optimization, and structural construction tasks under one unified verification pipeline, why prompt strength and evaluation budget matter for fair comparison, and how to read a total score alongside local advantages. It also walks through how enterprises can build their own verifiable evaluation pipelines — writing independent verifiers, running them in isolated sandboxes, recording budgets, and producing layered capability reports — while covering common verifier mistakes and key limitations such as data contamination and generalization boundaries.

    1. Introduction: From “Answering Questions” to “Being Verifiable”

    When large language models first became widespread, communities often used knowledge Q&A, code generation, and math problems to measure capability. This kind of evaluation is cheap and scores quickly, but as model capabilities improve, its ability to tell models apart declines. More importantly, many problems only test whether a model can reproduce a known solution, whereas the core of research work is often a new proof, a better program, or an improved construction.

    So the evaluation perspective is shifting from “can the model answer questions” to “can the model produce research results that can be independently reviewed.” Automatically verifiable evaluation systems place research tasks such as proofs, program optimization, and structural constructions into a machine-judgment framework, giving AI research capability a more objective and horizontally comparable measure.

    The core of this kind of evaluation is not whether the model “sounds convincing,” but whether it can submit objects that can be checked by a program, a solver, or a formal kernel. Scoring is based on verification results, not on the model’s self-report. For enterprises, the same method can be applied to R&D process evaluation: define the task first, verify it automatically, and finally produce an iterative capability report.

    2. Three Types of Research Outputs and a Unified Verification Framework

    In theoretical computer science, algorithm optimization, and combinatorial construction, three types of outputs are common: theorem proofs, program and algorithm optimization, and structural constructions. What they share is that correctness can be determined by a machine within finite time.

    Output type Typical example Verification method Evaluation focus
    Theorem proof Proving combinatorial or computational complexity propositions Formal proof checking or manual review Whether it relies on extra hints; whether the proof holds step by step
    Program and algorithm optimization Scheduling, compilation, solver algorithm improvements Correctness tests + performance benchmarks Whether results are stably correct; whether performance gains are reproducible
    Structural construction Extremal graphs, error-correcting codes, combinatorial designs Checking constraints, edge counts, and parameters one by one Whether it beats the reference value; whether it scales

    In the past, these three types of tasks were often scattered across different communities, with different scoring standards: proof evaluation focused on formalization, algorithm evaluation focused on runtime, and construction evaluation focused on whether a record was broken. The key to a unified verification framework, then, is to bring them into the same pipeline:

    1. Task standardization: Each task has a clear input, constraints, and judgment criteria, not relying on the evaluator’s subjective understanding.
    2. Output structuring: Proofs are submitted as formal scripts, programs as source code, and constructions as structured data such as adjacency lists.
    3. Verification automation: An independent verifier performs the judgment, producing results that are reproducible and auditable.
    4. Explicit budgeting: The number of attempts, upper limits on computing resources, and whether external tools are allowed are all specified.
    5. Structured reporting: Not just a total score, but also per-category scores, number of attempts, time spent, and failure types.

    This turns evaluation from a one-time exam into an iterative process with feedback, closer to the trial-and-error process in real research.

    3. Prompt Strength and Evaluation Budget: Two Key Variables for Fairness

    3.1 Prompt Strength

    In theorem-proving tasks, whether intermediate lemmas are provided significantly affects difficulty.

    • With intermediate lemmas: The model only needs to fill gaps in the proof route, closer to solving a problem with hints.
    • Without intermediate lemmas: The model must break down the problem and choose a proof path itself, closer to independent research.

    The same model may perform very differently under the two settings. Therefore, evaluation results must state the prompt conditions; otherwise the numbers are not comparable.

    3.2 Evaluation Budget

    The budget mainly includes the number of attempts and computing resources.

    • Single submission: Tests first-attempt success rate, i.e., the model’s ability to get it right in one go.
    • Multiple attempts: Tests the ability to approach a correct solution under feedback, close to the pass@k idea.

    For example, one system succeeds 14 out of 20 times, a single-attempt success rate of 70%; another succeeds 23 out of 60 times, a success rate of 38.3%. The two differ in absolute success count and in single-attempt efficiency, and they answer different questions. Only by putting the number of attempts, success rate, and task coverage together is the conclusion complete.

    4. Illustrative Leaderboard Review: Total Score vs. Local Advantage

    To avoid misquotation, the leaderboard information below is treated as illustrative cases, used to explain the relationship between “total score” and “local advantage.” For formal evaluation, the evaluation name, link, release time, sample size, number of attempts, prompt strength, runtime environment, and scoring formula should be added.

    System Sample total score Proof task scale Attempts Intermediate lemmas Success rate Observation
    System A 71.9 20 problems Single or unified budget No 70.0% Overall lead
    System B 57.4 Covers 9 theorems 60 No 38.3% Wider coverage with more attempts
    Other systems Below 11.7% Needs unified description Needs unified description Needs unified description Needs unified description Not directly comparable

    This example shows that a system with a high total score is not necessarily strong on every subtask. For instance, on a hard problem about PCP alphabet soundness, System B completed all three attempts while System A did not solve it. PCP usually refers to settings related to probabilistically checkable proofs; the PCP theorem connects probabilistically checkable proofs with NP problems, and the alphabet size affects proof length and verification cost. So alphabet soundness involves both proof construction and verification efficiency.

    Now consider two construction and proof tasks. One system searched for a triangle-free and quadrilateral-free graph on 124 vertices with as many edges as possible, finding a 630-edge construction, exceeding the 629-edge reference value. Another system tightened a certain binary code parameter upper bound from 84 to 82. Here A(n, d) denotes the maximum number of codewords in a binary code of length n with minimum Hamming distance d. The former tests combinatorial search and structural construction; the latter tests quantitative analysis and proof ability.

    Three conclusions follow:

    • The total score reflects overall capability, while local results reflect specialized capability; the two cannot replace each other.
    • Selection should combine specific tasks, not just the overall ranking.
    • When comparing, the number of attempts, prompt settings, budget, and runtime environment must be checked for consistency.

    5. Capability Landscape: Where Models Are Strong and Where They Are Weak

    Summarizing the results above is closer to a “capability landscape” than a simple ranking.

    Capabilities models already have include:

    • Completing some theorem proofs when the problem is well-defined and feedback is automatically obtainable;
    • Making verifiable optimizations to existing algorithms or programs;
    • Improving existing constructions, producing new objects that beat reference values and can be checked item by item.

    But three kinds of high ground remain hard to surmount:

    1. Problem selection: Tasks are designed by humans; models usually are not responsible for judging which problems are worth researching.
    2. Value judgment: Verifiers can judge right from wrong, but not what is important.
    3. Open problems: Major open problems lack intermediate feedback and have long verification cycles; relying solely on repeated attempts is hard to break through.

    A more reasonable positioning, therefore, is that the model is an efficient executor in the research process, not an autonomous researcher. In stages with clear goals and timely feedback, it can lower the cost of trial and error; in direction selection and value judgment, human researchers remain indispensable.

    6. How Enterprises Build Verifiable Evaluation Pipelines

    This method applies not only to academic research but also to software development scenarios such as compilation optimization, query optimization, task scheduling, and test generation. Enterprises can follow these steps.

    6.1 Define the Target Output and Acceptance Criteria

    Prioritize high-frequency, high-value scenarios with clear acceptance, such as SQL query optimization, core algorithm acceleration, scheduling strategy generation, and test case generation. Each scenario should define:

    • Inputs and constraints;
    • The test cases, proof checkers, or solvers used for correctness judgment;
    • Scoring metrics, such as pass rate, performance gain magnitude, and resource consumption.

    6.2 How to Tell Whether a Task Suits Verifiable Evaluation

    A simple criterion is: can the output be turned into a structured object and automatically judged by a program? Verifiable tasks usually meet these conditions:

    • Outputs can be structured, such as source code, adjacency lists, test cases, proof scripts;
    • Correctness can be automatically checked, without relying on subjective human evaluation;
    • Resource boundaries are clear, such as CPU, memory, and time;
    • Scoring is quantifiable, such as success/failure, edge count, time, and coverage.

    6.3 Writing an Independent Verifier

    The verifier must be independent of the model and accept only structured output. The example below uses an “extremal graph with girth 5.” The task goal is to construct a graph with no triangles or quadrilaterals, using edge count as the scoring basis. Small-scale samples are used to verify the verifier’s logic and do not represent reaching the large-scale task baseline.

    1. def verify_girth5(n: int, edges: list, baseline: int = 630) -> dict:
    2. """A graph with girth 5: no triangles (C3) or quadrilaterals (C4).
    3. Returns whether the structure is valid; if valid, returns the edge
    4. count and whether it beats the baseline.
    5. For large-scale scenarios, acceleration can be combined with bit
    6. operations, SAT/SMT solvers, or graph libraries.
    7. """
    8. # 1) Structural validity
    9. if not isinstance(n, int) or n <= 0:
    10. return {"ok": False, "reason": "invalid number of vertices"}
    11. if not isinstance(edges, list):
    12. return {"ok": False, "reason": "invalid edge set format"}
    13. adj = [set() for _ in range(n)]
    14. seen = set()
    15. for e in edges:
    16. if not isinstance(e, (list, tuple)) or len(e) != 2:
    17. return {"ok": False, "reason": "invalid edge format"}
    18. u, v = e
    19. if not (isinstance(u, int) and isinstance(v, int)):
    20. return {"ok": False, "reason": "vertex indices must be integers"}
    21. if not (0 <= u < n and 0 <= v < n) or u == v:
    22. return {"ok": False, "reason": "vertex index out of range or self-loop present"}
    23. key = (min(u, v), max(u, v))
    24. if key in seen:
    25. return {"ok": False, "reason": "duplicate edge present"}
    26. seen.add(key)
    27. adj[u].add(v)
    28. adj[v].add(u)
    29. # 2) Constraint 1: no triangles
    30. # If an edge (u, v) has a common neighbor w, then u-v-w-u forms a triangle
    31. for u, v in seen:
    32. if adj[u] & adj[v]:
    33. return {"ok": False, "reason": "triangle present"}
    34. # 3) Constraint 2: no quadrilaterals
    35. # If two neighbors b, c of a share a common neighbor d other than a,
    36. # then a-b-d-c-a forms a quadrilateral
    37. for a in range(n):
    38. neighbors = list(adj[a])
    39. for i in range(len(neighbors)):
    40. for j in range(i + 1, len(neighbors)):
    41. b, c = neighbors[i], neighbors[j]
    42. # Safety check: if b and c are directly connected, there is a triangle
    43. if b in adj[c]:
    44. return {"ok": False, "reason": "triangle present"}
    45. common = adj[b] & adj[c]
    46. if a in common:
    47. common.remove(a)
    48. if common:
    49. return {"ok": False, "reason": "quadrilateral present"}
    50. # 4) After passing verification, use edge count as the scoring basis
    51. return {
    52. "ok": True,
    53. "edges": len(seen),
    54. "beats_baseline": len(seen) >= baseline
    55. }

    Failure example: a four-point cycle is rejected.

    1. {"n": 4, "edges": [[0, 1], [1, 2], [2, 3], [3, 0]]}

    The corresponding result is similar to:

    1. {"ok": false, "reason": "quadrilateral present"}

    Passing example: a pentagon has no triangles or quadrilaterals and passes structural verification. If the task baseline is 630 edges, this sample only proves the structure is valid and does not represent reaching the scoring baseline.

    1. {"n": 5, "edges": [[0, 1], [1, 2], [2, 3], [3, 4], [4, 0]]}

    The corresponding result is similar to:

    1. {"ok": true, "edges": 5, "beats_baseline": false}

    6.4 Common Verifier Errors

    Verifiers themselves can make mistakes. Before going live, focus on checking:

    • Mistaking adjacent nodes for a quadrilateral;
    • Failing to handle duplicate edges, self-loops, or out-of-range indices;
    • Checking only some constraints, e.g., checking for triangles but not quadrilaterals;
    • Ignoring timeouts, memory overflows, or abnormal input;
    • Treating the model’s self-report as the basis for verification.

    Before going live, add correct samples, error samples, and mutation samples. For example, manually insert duplicate edges or triangles and confirm the verifier can reject them.

    6.5 Executing in an Isolated Sandbox

    Evaluation often involves large amounts of code execution, so use a container platform, assign an independent runtime environment for each verification, and set CPU, memory, and execution time limits. Network access should be off by default, to prevent the program under test from making outbound requests.

    6.6 Setting a Budget and Recording the Whole Process

    Distinguish first-attempt success rate from limited-attempt success rate. All inputs, outputs, verification logs, and error messages should be fully retained for reproducibility and accountability. When task volume is large, use a message queue to queue and distribute tasks, avoiding resource contention that would affect result stability.

    6.7 Carrying the Pipeline with General Cloud Capabilities

    When implementing, enterprises can rely on general products from cloud platforms such as Baidu AI Cloud to carry the load:

    • Container platforms run sandbox tasks;
    • Object storage saves evaluation datasets, model outputs, and verification logs;
    • Log services and monitoring alerts track failure rates, timeout rates, and resource consumption;
    • Message queues smooth out peaks and fill valleys.

    At the same time, security and compliance cannot be omitted: private problems should be encrypted at rest and data-desensitized; log access should follow the principle of least privilege; keys should be centrally managed and rotated regularly; and evaluation data, model outputs, and verification logs should be authorized by role in tiers to avoid leakage.

    6.8 Moving from Total Score to Layered Reporting

    The final report should include four parts:

    • First-attempt success rate and limited-attempt success rate;
    • Per-category scores by task type;
    • Average time, computing power consumption, and the distribution of failure reasons;
    • Typical success and failure cases.

    This forms a capability landscape that can guide selection and iteration, avoiding being misled by a single score.

    7. Risks and Limitations

    Verifiable evaluation is not omnipotent; in practice, be wary of:

    • Data contamination: Public problems may enter the training corpus, inflating scores; private problems should be retained and the evaluation set periodically changed.
    • Verifier defects: A loophole in a verifier affects the credibility of all objects under test; version control is needed, with changes recorded.
    • Cost and energy consumption: Long-chain reasoning and large-budget search significantly increase computing consumption; budgets should be set in tiers.
    • Generalization boundaries: A high score on limited tasks does not equal general research capability; it should not be inferred that a model can independently complete end-to-end research.

    8. Conclusion

    The value of verifiable evaluation is turning the vague “AI research capability” into definable, executable, and reviewable metrics. Models can already complete some proofs, program optimizations, and construction improvements under conditions of clear goals, sufficient budget, and automatically obtainable feedback. But research direction selection, value judgment of results, and cracking major open problems still cannot do without humans.

    For developers, rather than asking “which model is the strongest,” it is better to first answer three questions: Can my task be rigorously verified? Is my budget explicitly set? Is my comparison standard consistent? Only by doing these three things solidly, and then using general cloud capabilities to complete the engineering support, can human-machine collaboration steadily deliver value in real R&D processes.