Verifiable AI Research Systems: From Theorem Proving to New Structure Discovery

    This article examines a unified framework for verifiable AI research, covering task specification, candidate generation and search, isolated execution, automated verification, scoring and ranking, and result archiving. It reviews reported benchmark outcomes, including theorem-proving and combinatorial-construction results, explains why independent verification matters more than fluent explanation, and offers engineering guidance for building systems that produce reproducible, checkable research artifacts.

    In recent years, artificial intelligence has improved rapidly in code generation, mathematical question answering, and scientific computing. Yet a clear gap remains between “being able to give an answer” and “being able to complete verifiable research work.” The former focuses more on whether the expression is fluent and whether the conclusion is close to a standard answer; the latter requires outputs to have rigorous logical grounding and to be reviewable by independent programs, formal systems, or other researchers.

    A study jointly proposed by industry researchers and university scholars attempts to place problem definition, model search, program execution, formal verification, result archiving, and human review into a single experimental workflow. The system does not treat a single model response as a research result. Instead, it requires the model to submit inspectable objects, parameters, code, or proofs, and uses automated verifiers to determine whether they truly satisfy the task requirements. This approach provides a stricter reference for measuring AI research capability than traditional question-answering tests.

    1. From “Answering” to “Proving”

    Traditional language model evaluation usually treats the correctness of the final answer as the core metric. For example, in a math problem, a model may be judged successful as long as its output matches the standard result. Research problems, however, often do not have a unique textual answer. This is especially true in theorem proving, extremal construction, coding design, and program optimization, where what matters is not whether the conclusion looks reasonable, but whether it can be reproduced and verified.

    A qualifying research-oriented output should meet at least the following conditions:

    1. Clear object: The model must provide a specific graph, set, code, program, or proof step, rather than merely describing that “some structure exists.”
    2. Satisfied constraints: The output object must satisfy all conditions in the problem, not just some of them.
    3. Independent verification: The verification process should not rely entirely on the model’s own explanation. It should be completed by an independent checker, computational program, or formal rule set.
    4. Reproducible result: Other researchers should be able to rerun the experiment according to the record and obtain the same or explainable results.
    5. Comparable improvement: If the model claims a better result, it must be rigorously compared with existing baselines, public upper bounds, or reference constructions.

    Therefore, the core of a research-oriented AI system is not only language generation capability, but the closed-loop capability of “generation—execution—verification—feedback.”

    2. Basic Components of a Unified Evaluation Framework

    Such systems can usually be divided into six functional modules.

    1. Task Description Module

    The task description needs to convert a natural language problem into a machine-executable objective. For example, a combinatorial mathematics task may require constructing a graph with a specified number of vertices and edges while forbidding certain local structures. Simply handing the problem to the model and asking it to “give the answer” is insufficient. It is also necessary to specify:

    • Input scale and resource limits;
    • Allowed representation formats;
    • Hard constraints that must be satisfied;
    • Scoring criteria for results;
    • Rules for determining success and failure;
    • Whether external computational programs are allowed.

    The clearer the task definition, the more the evaluation results can reflect the model’s own capability rather than prompt ambiguity or differences in human judgment.

    2. Generation and Search Module

    At this stage, the model proposes candidate proofs, candidate programs, or candidate mathematical structures. For complex problems, a single generation is usually unlikely to succeed, so the system needs to support multi-round search, including:

    • Having the model propose multiple candidate solutions;
    • Locating failure causes based on verification errors;
    • Making local modifications to existing solutions;
    • Switching between different representation formats;
    • Recording the input, output, and resource consumption of each attempt.

    This means the model is no longer just a one-shot answer generator, but a policy component within a search process.

    3. Execution Module

    If the task involves programs, graph structures, or coding objects, the model output needs to be converted into an executable representation. For example, the model may output an edge set, adjacency matrix, constraint list, or pseudocode, and the execution module handles parsing, format checking, and actual execution.

    The execution module must provide isolation to prevent erroneous code from affecting the host environment. Common practices include limiting runtime, memory, file access permissions, and network access permissions, and classifying and recording abnormal exits, infinite loops, and format errors.

    4. Automated Verification Module

    The verifier is the key to the entire system. It is not responsible for “guessing whether the model is correct,” but for making deterministic judgments according to predefined rules.

    For a graph-structure construction task, for example, the verifier can check:

    • Whether the number of vertices meets the requirement;
    • Whether edges are duplicated;
    • Whether both endpoints of each edge are valid;
    • Whether the number of edges reaches the target;
    • Whether forbidden triangles, quadrilaterals, or other subgraphs exist;
    • Whether the properties claimed by the model can be recomputed by the program.

    For theorem-proving tasks, the verifier can check whether the proof term passes formal rules, whether each derivation step uses allowed premises, and whether the final proposition matches the goal.

    5. Scoring and Ranking Module

    Simply counting “success” and “failure” cannot fully reflect research capability. A more reasonable metric system should consider:

    • Success rate;
    • Number of distinct tasks covered;
    • Number of attempts per task;
    • Quality of the proof or construction;
    • Magnitude of improvement relative to existing baselines;
    • Runtime and computational resources;
    • Whether the output can be independently reviewed.

    For example, one model may achieve extremely high-quality results on a small number of tasks, while another model may obtain moderate improvements on more tasks. Their research value cannot be judged by a single accuracy metric alone.

    6. Result Archiving Module

    Every experiment should preserve a complete trajectory, including the task version, prompt content, model output, verification logs, resource consumption, and final status. This has three benefits.

    First, it can distinguish incidental success from stable capability. Second, it can identify whether system failure occurs in generation, parsing, or verification. Third, it allows subsequent researchers to reproduce the experiment and improve search strategies.

    3. What Evaluation Results Reveal About Capability Differences

    In a set of tests covering theorem proving, coding reliability, and combinatorial construction, the highest-scoring system reached 71.9, while another system scored 57.4. The former completed 14 of 20 theorems without relying on intermediate lemmas, for a completion rate of 70.0%. The latter succeeded 23 times in 60 attempts, covering 9 distinct theorems, with an overall success rate of 38.3%. The remaining systems scored no higher than 11.7%.

    These data show that advanced models can already independently complete some research-like tasks with the help of clear objectives and automated verification feedback. But the data also show that model capabilities are not uniformly balanced.

    For example, in a coding alphabet reliability task, the second system completed the target three consecutive times, while the highest total-score system failed to solve the problem. This result is significant: an overall leaderboard ranking reflects only aggregate performance and does not show that a model dominates every problem type. Research tasks usually have highly structured domain differences. A model may perform strongly on one type of problem, such as proof reasoning, program search, graph construction, or coding design, while failing clearly on another.

    Therefore, when evaluating models, it is necessary to report overall scores, per-task performance, successful examples, and failure types at the same time, rather than publishing only a ranking.

    4. Observing Model Outputs in Mathematical Construction

    The most convincing results in this type of research are often not language answers, but new verifiable objects that the model actually constructs.

    1. Extremal Graph Construction

    In an extremal graph problem, the model produced a graph with 124 vertices and 630 edges that contains no triangles and no quadrilaterals. The reference value previously used for the task was 629 edges, so this construction moved the known reference result forward by one step.

    The value of such a result is that the model output is not an unchecked description. It can be encoded as an edge set and then verified item by item by an independent program. The verifier can traverse all possible local vertex combinations, check whether forbidden subgraphs exist, and recount the vertices and edges.

    However, a successful construction does not mean the theoretical problem has been fully solved. A new finite object may only improve a lower bound, and it may also involve representation errors, verification omissions, or duplication of existing results. Therefore, human researchers still need to confirm its novelty, analyze the construction pattern, and attempt to generalize the finite-scale result into a general theory.

    2. Improving a Coding Parameter Upper Bound

    In another coding theory task, the model proved that a certain parameter satisfies an upper bound of no more than 82, whereas the previous public reference upper bound was 84. On the surface, this is only a small numerical improvement, but in combinatorics and coding theory, a tighter upper bound may affect subsequent constructions, complexity analysis, and theoretical boundary judgments.

    Such proofs require particular attention to logical completeness. The model may give a correct conclusion while omitting key conditions, or it may use an unproven assumption at some step. Only formal checking or step-by-step review by domain researchers can determine whether the result truly holds.

    5. Why Automated Verification Matters More Than Language Explanation

    Language models can easily generate text that is “formally correct but substantively wrong.” For example, a proof may contain many technical terms and a seemingly reasonable derivation order, but one key inequality may not hold. A program may run but fail to satisfy all constraints required by the task.

    Automated verification can turn “trust judgment” into “rule checking.” This does not mean the verifier can solve all problems, but it can effectively filter out the most common errors:

    • The format is legal, but the object does not satisfy the constraints;
    • Proof steps lack premises;
    • Statistical results are inconsistent with the actual object;
    • The program passes only some samples;
    • The improvement claimed by the model does not exceed the real baseline.

    In engineering practice, the verifier should be as small, clear, and independent as possible, and should avoid sharing the same set of erroneous assumptions as the model. The verifier itself also needs testing; otherwise, the system may merely move errors from the model to the evaluation program.

    6. Engineering Recommendations for Moving from a Research Prototype to a Usable System

    If you want to build a similar research agent, it is worth focusing on the following aspects.

    1. Define Verifiable Objectives First

    Do not start with “letting the model conduct research autonomously.” Instead, first choose problems for which success or failure can be clearly judged. For example, program performance optimization, finite structure search, formal theorem proving, and parameter improvement are all suitable early tasks.

    2. Split Large Problems into Multiple Stages

    The workflow can be divided into problem understanding, candidate generation, execution verification, error analysis, and result summarization. Each stage uses different checking rules, reducing the pressure on the model to complete all work in one pass.

    3. Preserve Failure Trajectories

    Failure samples are often more valuable than success samples. The system should record failure causes, such as parsing failure, resource limit exceeded, local constraint violation, missing proof step, or suboptimal result. Only by building a failure taxonomy can prompts, search strategies, and verifiers be improved in a targeted way.

    4. Use Multiple Independent Attempts

    Research tasks involve randomness. A single success may come from incidental search, while multiple independent runs can help determine whether the model has truly mastered a strategy. When reporting results, the number of attempts, number of successes, and number of tasks covered should be published together.

    5. Distinguish Discovery from Explanation

    The model may first find a valid construction, and then human researchers explain its pattern. The former is computational discovery, and the latter is theoretical understanding. Both are important, but they cannot be conflated. Only when a construction can be summarized, proved, and generalized can it potentially form a more complete scientific result.

    7. Current Capability Boundaries and Future Directions

    At present, AI systems can already complete some proof, program-search, and mathematical-construction tasks under clear problems, sufficient computational budgets, and reliable feedback mechanisms. But there is still a clear gap before they can truly conduct major research independently.

    First, models are usually not good at independently choosing long-term valuable research directions. They can search on a given objective, but they may not know which problem is more important or more worth investing in. Second, models’ judgment of result value is still limited. Being able to discover a new construction does not mean the model understands why that construction matters. Third, when facing open problems, models tend to get stuck in local improvements and lack stable long-term planning and theoretical abstraction capabilities.

    Future research agents may need three types of capabilities:

    • Problem discovery capability: Identifying worthwhile research gaps from literature, data, and existing results;
    • Verifiable search capability: Continuously proposing, testing, and revising hypotheses within formal tools and computational environments;
    • Theoretical abstraction capability: Summarizing patterns from finite experiments and forming generalizable theorems or methods.

    Overall, the key to AI participation in research is not to let models replace researchers, but to build a workflow that constrains errors, preserves evidence, supports reproducibility, and promotes collaboration. The model expands the search space, the verifier filters reliable results, and researchers judge problem value, explain discoveries, and complete theoretical advancement. Only when these three form a closed loop can AI gradually develop from an “assistant answering tool” into genuinely useful research infrastructure.