AI Research Agents: Verifiable Reasoning and Discovery

    This article examines a unified evaluation framework for AI research agents that shifts assessment from plausible text generation to independently verifiable artifacts. It summarizes results on theorem proving and combinatorial construction, explains why leading systems can have task-dependent strengths, outlines engineering modules for task specification, search, verification, evidence logging, and human collaboration, and clarifies the current boundaries of AI-driven research.

    In recent years, large models have continued to improve in code generation, knowledge question answering, and mathematical reasoning. Yet a clear gap remains between “being able to give an answer” and “being able to conduct research.” The core of research is not merely producing plausible text. It involves formulating testable questions, constructing rigorous solutions, verifying whether results hold, and judging whether the outcomes have real research value.

    A recent effort by an AI research team and university researchers in theoretical computer science proposed a unified evaluation framework for the research process. The framework does not primarily judge model outputs by length, language fluency, or surface-level reasoning steps. Instead, it requires systems to submit objects that can be independently reviewed, such as formal proofs, graph structures satisfying constraints, coding schemes, program implementations, or validated parameter improvements.

    From Generating Answers to Submitting Verifiable Results

    Traditional large-model evaluations usually focus on whether the final answer is correct. Research tasks, however, are often more complex: a problem may not have a unique answer, part of an answer may need to be checked by a program, and a proof process may involve many intermediate steps. As a result, human reading or string matching alone makes it difficult to determine whether a model has truly completed a task.

    A unified evaluation framework typically includes the following stages:

    1. Problem normalization: Convert a natural-language research problem into clear inputs, constraints, and objective functions.
    2. Candidate generation: Have the model propose proofs, algorithms, programs, parameters, or combinatorial structures.
    3. Automated verification: Use theorem checkers, constraint solvers, graph search programs, or independent test scripts to check results.
    4. Deduplication and archiving: Avoid repeatedly counting the same idea and preserve complete experimental traces.
    5. Human review: Have researchers judge whether results have novelty, explanatory value, and practical research value.

    The key to this design is that the model no longer only needs to “convince people.” It must submit a result that an external system can check. For mathematical proofs, the proof object must satisfy logical rules. For program optimization, the improved program must run correctly on a fixed test set. For combinatorial constructions, the generated structure must satisfy each problem constraint.

    What the Evaluation Results Show

    In a combined task suite, the top-ranked systems showed a clear advantage. The first-place system scored about 71.9, the second-place system about 57.4, and the other participating agents scored noticeably lower. These scores should not be equated with general intelligence. They reflect comprehensive performance under a specific task set, compute budget, and verification process.

    In theorem-proving tasks, the first-place system completed 14 of 20 theorems without relying on human-provided intermediate lemmas, reaching a success rate of 70%. Another system succeeded 23 times in 60 attempts, covering 9 different theorems, with a single-attempt success rate of about 38.3%. The remaining systems had success rates no higher than 11.7%.

    These data point to two important trends.

    First, increasing inference time and the number of attempts can indeed improve the probability of completing research tasks. Complex proofs are often not generated successfully in one pass. A model needs to revise assumptions, fill in lemmas, and adjust search paths.

    Second, simply increasing the number of calls cannot replace system design. If each attempt lacks reliable feedback, the model may repeatedly generate similar errors. Only by combining verifiers, error information, candidate management, and search strategies can compute budget be converted into effective output.

    Leading Systems Are Not Universally Dominant

    A high overall ranking does not mean that a system outperforms others on every task. For example, in a constraint task related to coding reliability, the second-place system completed the objective three consecutive times, while the system with the higher total score did not solve that problem.

    This phenomenon shows that the capabilities of research agents are strongly task-dependent. A system may be good at long-chain proofs but poor at handling discrete constraints. Another system may have advantages in program search or structural construction but perform inconsistently on theorem proving that requires extensive abstract concept transformation.

    Therefore, when evaluating research agents, it is not enough to publish only a total score. Reports should also include:

    • Scores by task type;
    • Number of attempts per problem;
    • First-attempt success rate and cumulative success rate;
    • Whether human-provided lemmas or hints were used;
    • Whether the verifier was fully independent;
    • Whether results were rediscovered;
    • Complexity and interpretability of generated results.

    Only by breaking out these dimensions can one judge whether a system completes tasks through stable capabilities or through many random attempts that yield a small number of successes.

    Two Representative Construction Results

    The more intuitive results in the study came from combinatorial mathematics construction tasks.

    In an extremal graph problem with girth 5, the agent generated a graph with 124 vertices and 630 edges that satisfies the constraints of containing no triangles and no quadrilaterals. The reference value used in the earlier task was 629 edges, so this construction advanced the known reference result by one step.

    The difficulty of such tasks is that the model cannot simply output a description of a graph. It must provide the complete vertex set, edge set, and constraint-verification results. The verification program needs to check whether any three vertices form a triangle and whether any four vertices form a quadrilateral, while also confirming that the number of edges, number of vertices, and structural description are consistent with one another. If even one violating subgraph exists, the entire result cannot hold.

    Another result involves a parameter upper bound in coding theory. The system gave a proof that a certain parameter is no more than 82, further tightening the existing upper bound of 84. The value of improving an upper bound is not only that the number becomes smaller. It may also narrow the search space of a theoretical problem and provide a basis for subsequent exact evaluation or for constructing better schemes.

    These two types of results represent two capabilities of research agents: one is searching for and generating discrete objects that satisfy complex constraints, and the other is compressing the range of theoretical parameters through combinatorial arguments. The former leans toward computational construction, while the latter is closer to traditional mathematical proof. Both require results that can be independently verified.

    Engineering a Unified Pipeline

    To turn such methods into a reusable research system, the following modules are worth building.

    1. Task Description Layer

    Task descriptions should clearly specify input format, output format, constraints, and evaluation criteria. Vague natural-language goals cause models to spend substantial time interpreting ambiguity and also increase the difficulty of designing verifiers.

    2. Generation and Search Layer

    The system can adopt strategies such as parallel multi-candidate generation, staged reasoning, error-driven revision, and result reranking. For proof tasks, it can first generate a proof plan and then fill in lemmas. For construction tasks, it can first form an initial structure and then use local search to reduce violations.

    3. Verification Layer

    Verifiers should be as independent as possible from the generative model, avoiding situations where the model “judges itself to be correct.” For program tasks, sandbox environments and fixed test sets should be used. For mathematical tasks, formal checks or independent computational scripts should be introduced. For combinatorial constructions, automated checkers should be designed to cover all constraints.

    4. Evidence Preservation Layer

    The system should record input versions, model outputs, verification logs, failure reasons, random seeds, computation time, and final results. Complete traces help reproduce experiments and allow researchers to analyze exactly where a model made an error.

    5. Human-AI Collaboration Layer

    Researchers should not be responsible only for final “acceptance.” They can also provide guidance in task decomposition, key definitions, search-space pruning, and result interpretation. Reasonable human-AI collaboration can significantly reduce ineffective search while preserving the model’s advantages in large-scale enumeration and solution combination.

    Current Boundaries

    Although the experimental results are encouraging, they do not mean that agents can already conduct complete research independently. Current systems still typically depend on three preconditions: the problem has already been clearly defined, the evaluation criteria can be automated, and the compute budget is sufficient.

    Real-world research often starts from ambiguous phenomena. Researchers need to judge which problems are worth studying, which phenomena may be universal, and which direction has long-term value even if only partial results are obtained. These judgments cannot be easily converted into an immediate verifier.

    In addition, the fact that a model finds a correct construction does not mean it understands the underlying pattern. Proving a special case is not the same as discovering a generalizable theory. Research results also require explanations of mechanisms, connections, directions for generalization, and long-term review through peer evaluation.

    A more accurate conclusion is therefore this: AI can already serve as a high-intensity search and verification assistant in the research process. Under conditions of clearly defined problems, automated feedback, and sufficient resources, it can complete some proofs, program improvements, and structural constructions. However, human researchers remain irreplaceable when it comes to choosing important problems, assessing theoretical value, forming original research routes, and solving major open problems.

    In the future, the development of research agents may focus not only on generating more answers, but also on building more reliable verification environments, more efficient search strategies, more transparent evidence chains, and better human-AI collaboration mechanisms. Only when “proposing ideas, executing experiments, verifying results, and explaining significance” can form a closed loop will AI have the potential to evolve from an answering tool into real research infrastructure.