The AI Research Capability Map: What Verifiable Evaluation Reveals About the Limits of AI Agents

    This article examines how verifiable evaluation changes the way we assess AI research agents. It defines what makes a research task verifiable, breaks down the main task categories (constraint-satisfaction construction, parameter improvement, program optimization, proof assistance, and counterexample generation), and explains how to design an evaluation system around clear task definitions, attempt budgets, verifiers, and multi-metric reporting. It then reviews current agent performance, including where strong reasoning plus sufficient budget delivers measurable gains, where long-chain reasoning still falls short, and how capabilities vary unevenly across systems. Finally, it covers the engineering of a research-agent pipeline, the boundaries that still require human leadership, and practical recommendations for developers and enterprises that want to adopt verifiable research tasks in production.

    Introduction: From “Looks Plausible” to “Can Be Verified”

    In research-assistance scenarios, traditional generative models can quickly produce explanations, derivations, pseudocode, candidate constructions, or experimental ideas. The problem is that these outputs cannot always be independently verified. An answer that reads as logically coherent is not necessarily sound at the level of mathematical constraints, boundary conditions, program execution, or formal proofs. As agent technology matures, the focus of evaluation is shifting from “does the answer look right” to “can the result be verified.”

    A verifiable research task is one with clear inputs, outputs, constraints, and a verification method. Examples include constructing a mathematical object with a specific property, giving an upper or lower bound for a parameter, generating a runnable optimization program, providing the proof steps for a proposition, or supplying a concrete scheme for a combinatorial design. The defining characteristic of such tasks is that an agent must not only produce a conclusion but also produce an object, a parameter, or a proof that can be independently checked.

    The value of this evaluation approach is that it moves model output from “subjectively credible” to “objectively testable.” For developers and technical teams, this means it becomes feasible to design a more engineering-oriented automated research-assistance system: the model generates candidate solutions, a verifier filters them and returns feedback, and human experts make the final decision.

    1. What Makes a Research Task Verifiable

    To build a sound evaluation system for research agents, the first step is to define what “verifiable” means. Not every research problem is a good fit for an automated agent, especially open-ended, long-horizon problems that depend on domain intuition. By contrast, the following task types work better as evaluation targets and engineering entry points:

    1. Constraint-satisfaction construction tasks. Given a set of constraints, the agent must output an object that satisfies them. For example, in combinatorics, constructing a graph with a specific number of edges, vertices, girth, or conflict relations. The advantage is a clear objective and a result that a program can check.

    2. Parameter-improvement tasks. The agent must provide an upper bound, lower bound, or a better value for a parameter, together with a corresponding construction or proof. For example, in a combinatorial optimization or coding problem where the best published upper bound is 84, the agent is asked to reach 82 and supply verifiable evidence.

    3. Program-optimization tasks. Given an objective function and constraints, the agent must generate runnable code or an algorithm that reaches or beats a reference baseline. These tasks require compilation, execution, testing, and result comparison.

    4. Proof-assistance tasks. The agent must produce the proof of a proposition, or fill in a key lemma in an existing proof. If a formal verifier or independent review mechanism exists, the process can be constrained further.

    5. Counterexample-generation tasks. The agent must find a counterexample that refutes a conjecture or negates a conclusion. A counterexample is valuable precisely because it can be verified directly, which makes this task type well suited to automated research evaluation.

    What these tasks share is that the result does not stay at the level of natural-language description; it enters a well-defined verification loop.

    2. Designing the Evaluation System: Tasks, Budget, Feedback, and Review

    A mature evaluation system for research agents cannot rely on a single “did the model get it right once” check. It needs to weigh task design, budget control, verification feedback, and result review together.

    2.1 Task Definitions Must Be Precise

    If a research task is too open-ended, the model can easily give a vague, general answer. The evaluation set should therefore favor problems with clear boundaries. For example:

    • Input: structural constraints for a class of graphs, target values, and reference baselines;
    • Output: a graph object that meets the conditions, construction parameters, or proof steps;
    • Constraints: no forbidden substructures, and a required maximum edge count or minimum vertex count;
    • Verification: a program that checks whether the object satisfies every condition.

    The clearer the task definition, the better it reflects the agent’s real reasoning, search, and optimization ability.

    2.2 Budget Control Matters More Than a Single Success

    If the model is allowed only one attempt, results are heavily influenced by randomness. A more reasonable approach is to set an attempt budget, for example 20, 60, or more attempts per task, and to track:

    • final success rate;
    • number of problems covered;
    • average number of attempts;
    • first-attempt success rate;
    • success rate after correction.

    This design is closer to the real research process. Research is not a single lucky hit; it is repeated attempt, verification, correction, and convergence under a budget constraint.

    2.3 The Verifier Is the Core of the System

    The verifier determines whether the evaluation is trustworthy. Common verification methods include:

    • Syntax checks: whether the output follows the required format;
    • Structure checks: whether the constructed object satisfies all hard constraints;
    • Numerical checks: whether the target value meets or beats the baseline;
    • Execution checks: whether the program runs and produces stable results;
    • Proof checks: whether the proof steps are complete and free of logical gaps;
    • Independent review: re-verification by a separate system or a separate set of rules.

    Without a verifier, evaluation tends to degrade into text comparison. With a verifier, model output can be turned into an assessable research artifact.

    2.4 Multi-Metric Evaluation Avoids Single-Point Bias

    A sound evaluation framework should include multiple metrics:

    • Completion rate: whether the task ultimately produces a valid result;
    • Success rate: the share of valid results across multiple attempts;
    • Coverage: how many different problem types the agent can solve;
    • Stability: whether repeated runs still maintain a high success rate;
    • Correction efficiency: how quickly the model improves after verifier feedback;
    • Review cost: how much time a human expert or automated system needs to verify a result.

    Together, these metrics form a more realistic “capability map” rather than a simple leaderboard.

    3. Current Performance of Autonomous Research Agents

    From the perspective of verifiable evaluation, agents with strong reasoning today can already deliver substantial value on well-defined tasks, but they also expose clear limits.

    3.1 Clear Problems and Sufficient Budget Amplify Capability

    Given a precise problem, enough attempts, and verifiable feedback, some agents can complete fairly complex construction and optimization tasks. For example, in extremal-graph-theory-style construction tasks, an agent can generate graph objects with a specific number of vertices and edges while satisfying constraints such as triangle-free or quadrilateral-free. In one evaluation, an agent constructed a graph with 124 vertices and 630 edges, beating the task’s 629-edge reference. This shows that the agent has some capability for combinatorial search and local optimization.

    The value of such a result is not just “one more edge.” It demonstrates that the agent can generate verifiable objects under strict constraints. Such output is not vague natural-language reasoning; it is a construction that a program can check.

    3.2 Parameter-Improvement Tasks Show Reasoning Combined with Construction

    Another typical task is improving a parameter’s upper bound. For example, in a certain combinatorial-parameter problem, the best known upper bound is 84, and the agent attempts to reach 82 with verifiable evidence. This task is harder because the model must not only know how to construct a solution but also judge which direction is worth spending budget on.

    Being able to tighten an existing upper bound suggests the agent has potential in:

    • abstracting the constraints;
    • estimating the existing boundary;
    • filtering candidate constructions;
    • iteratively correcting based on verification results.

    However, this ability is often highly dependent on the task type. Once a task lacks a clear verifier, or requires cross-domain judgment, the agent’s stability drops noticeably.

    3.3 Without Intermediate Steps, Long-Chain Reasoning Remains a Weakness

    A notable pattern in evaluation is that success rates can be relatively high when the model is allowed to rely on intermediate lemmas or hints, but drop once it is required to complete a complex proof or construction directly, without intermediate lemmas. This indicates that current agents perform well on short-chain reasoning and local search but still need to improve on long-chain autonomous decomposition.

    Common capabilities in research work include:

    • breaking a large problem into subproblems;
    • judging which subproblem is most critical;
    • choosing an appropriate tool or construction method;
    • performing root-cause analysis after a failure;
    • adjusting strategy based on verifier feedback.

    These are precisely the areas where current agents still lag behind human experts.

    3.4 Capability Is Not Uniform but Terrain-Like

    The evaluation results show that differences between agents are usually not a matter of one system leading across the board or trailing across the board, but rather a difference in how capabilities are distributed. Some systems are good at construction, some at proofs, some are more stable at program optimization, and some perform better on specific constrained tasks. In one evaluation, a leading system reached a success rate of about 70% on some theorem tasks, some systems reached about 38%, and most of the rest stayed below 12%. But on another task related to coding reliability, the systems that performed well elsewhere did not necessarily lead, and different systems traded wins.

    This reminds us that an autonomous research agent is not a single capability dimension. It is a multi-dimensional system composed of reasoning, search, verification, correction, expression, and engineering ability.

    4. Why Verifiable Feedback Matters

    Verifiable feedback is what allows autonomous research agents to move from “chat-style output” to a “production-grade tool.” Its value shows up in four main areas.

    4.1 Reducing Hallucination Risk

    Without verification feedback, a model can confidently give a wrong conclusion. With a verifier, the model can tell whether an output satisfies the hard constraints, which reduces invalid generation.

    4.2 Enabling Closed-Loop Optimization

    Agents can iterate based on verifier feedback:

    • if the structure does not hold, adjust the construction parameters;
    • if the proof is incomplete, fill in the missing steps;
    • if the program fails to run, fix the boundary conditions;
    • if the target value is not met, redesign the search strategy.

    This closed loop is closer to a real R&D workflow than one-shot generation.

    4.3 Supporting Finer-Grained Reward Design

    In automated search or reinforcement-style training, verification results can be converted into reward signals. For example:

    • passing verification: high score;
    • close to passing: partial reward;
    • clear improvement: extra reward;
    • repeated failures: negative feedback.

    This helps improve how efficiently the system converges within a given budget.

    4.4 Improving Reproducibility

    Verifiable tasks usually require that output objects, programs, or proofs can be saved, executed, and re-checked. This provides a foundation for reproducing experiments, building evaluation sets, and accumulating research data.

    5. Engineering: Building a Research Agent Pipeline

    To turn verifiable research tasks into a system that runs sustainably, you need a complete agent pipeline. The following is a general architecture.

    5.1 Task Layer

    The task layer defines the problem, constraints, baseline, verifier, and budget. Each task should include:

    • problem description;
    • input parameters;
    • output format;
    • constraints;
    • verification script;
    • reference baseline;
    • maximum number of attempts.

    5.2 Reasoning Layer

    The reasoning layer generates candidate solutions. It can adopt different strategies depending on the task type:

    • combinatorial construction: candidate generation + constraint search;
    • program optimization: code generation + execution testing;
    • proof completion: step generation + logic checking;
    • parameter improvement: heuristic guessing + verification filtering.

    5.3 Execution Layer

    The execution layer runs candidate results in a controlled environment. It typically needs:

    • sandbox isolation;
    • resource quotas;
    • timeout control;
    • permission restrictions;
    • logging;
    • exception handling.

    This keeps unreliable output from affecting the main system and makes later auditing and reproduction easier.

    5.4 Verification Layer

    The verification layer is the core of the system’s trustworthiness. It should include:

    • format checks;
    • constraint checks;
    • execution checks;
    • proof checks;
    • result comparison;
    • a human-review interface.

    The verification layer should be as automated as possible, but should not fully exclude manual spot checks, especially for new or high-risk tasks.

    5.5 Data and Observability Layer

    The value of a research agent comes not only from a single result but also from long-term data accumulation. You need to retain:

    • task input;
    • model candidate output;
    • verification reports;
    • failure reasons;
    • success paths;
    • budget consumption;
    • version changes.

    On the engineering side, organizations can rely on general cloud capabilities such as those from Baidu AI Cloud to host this kind of system: for example, a container platform to run sandbox tasks, object storage to keep construction results and proof artifacts, a logging service to record execution traces, and monitoring and alerting to track task success rates, resource consumption, and abnormal states. The point is not to depend on any single component, but to bring task execution, result verification, and runtime observability into a unified governance framework.

    5.6 Illustrative Code

    The following is simplified pseudocode for an agent evaluation flow:

    1. for task in tasks:
    2. plan = build_initial_plan(task)
    3. best_result = none
    4. for attempt in range(task.budget):
    5. candidate = generate_candidate(task, plan)
    6. report = verify(candidate)
    7. if report.passed and report.score > best_result.score:
    8. best_result = candidate
    9. if not report.passed:
    10. plan = repair_plan(task, plan, report)
    11. if best_result exists:
    12. save_result(best_result, report)
    13. update_metrics(task, best_result)
    14. else:
    15. log_failure(task)

    This flow captures the core idea of verifiable research tasks: generate, verify, give feedback, correct, and accumulate.

    6. Current Boundaries: What Cannot Yet Be Fully Handed to AI

    Although verifiable evaluation shows the potential of agents, it is important to see the boundaries clearly.

    6.1 Autonomous Problem Selection Is Still Weak

    Real research is not only about solving a given problem; it also involves discovering problems, posing them, and judging their value. Current agents are better suited to well-defined problems than to autonomously proposing new directions of long-term value.

    6.2 Judging the Value of Results Still Depends on Humans

    A result that passes verification is not necessarily important. An agent can improve a parameter, but it may not be able to judge whether that improvement carries theoretical significance, engineering value, or publication value. That judgment requires domain knowledge, academic experience, and research taste.

    6.3 Major Open Problems Still Need Human Leadership

    The difficulty of open problems often lies in vague definitions, unknown paths, and sparse feedback. Agents can help with local subproblems, but they can rarely lead a complete research process on their own.

    6.4 The Verifier Itself May Have Blind Spots

    If the verifier does not cover all cases, the model may generate results that “satisfy the current checks but not the true intent.” The verifier therefore needs continuous iteration and joint maintenance with domain experts.

    6.5 Budget and Compute Shape the Result Distribution

    The effectiveness of a research agent is closely tied to its attempt budget. A higher budget may yield a higher success rate, but this does not mean the system already has autonomous research capability; it means that, with enough resources, search and optimization ability is amplified.

    7. Practical Recommendations for Developers and Enterprises

    For teams that want to bring autonomous research agents into business or R&D workflows, a few starting points help.

    7.1 Start with Verifiable Tasks

    Prioritize tasks that can be checked by a program or a set of rules, rather than challenging open-ended research problems from the outset. Code optimization, parameter tuning, structural construction, constraint satisfaction, and test-case generation are all suitable first-stage scenarios.

    7.2 Build Your Own Evaluation Set

    Organizations should not rely only on external public benchmarks; they should build evaluation sets tailored to their own business. An evaluation set should include:

    • typical tasks;
    • boundary tasks;
    • failure cases;
    • high-difficulty tasks;
    • verifiable tasks.

    Only then can you measure whether a system truly fits your scenario.

    7.3 Emphasize Human-AI Collaboration

    A sound collaboration model looks like this:

    • the agent generates candidate solutions;
    • the verifier filters them automatically;
    • human experts review key results;
    • the system accumulates successful paths and lessons from failures.

    This model is more robust than full automation and easier to put into practice.

    7.4 Version Your Results

    In research and R&D settings, results that cannot be reproduced sharply reduce trust. Every candidate result should be bound to:

    • task version;
    • model version;
    • parameter configuration;
    • verifier version;
    • run time;
    • resource consumption.

    This supports later auditing, retrospectives, and optimization.

    7.5 Treat the Verifier as a Core Asset

    In many systems, the model itself may iterate quickly, but how complete the verifier is determines the system’s ceiling. Organizations should treat verification rules, checking scripts, expert rules, and test cases as long-term assets and continue to invest in them.

    8. Conclusion: A Capability Map, Not a Declaration of Omnipotence

    The purpose of evaluating verifiable research tasks is not to simply declare that one agent surpasses humans across the board, nor to prove that AI can already do research independently. It is more like a capability map: when problems are clear, feedback is verifiable, and budget is sufficient, agents can complete some proofs, constructions, program optimizations, and parameter improvements; but on autonomous problem selection, value judgment, and tackling major open problems, human experts must still lead.

    For developers, the most worthwhile focus is not a single evaluation result but the methodology behind it: making research tasks verifiable, automating the verification process, closing the feedback loop, and engineering the execution environment. Only by building such a system can autonomous research agents move from “looking smart” to “being trustworthy,” and create lasting value in real R&D and research workflows.