Evaluating LLM Research Capabilities: From Problem Solving to Verifiable Discovery

    Traditional benchmarks mainly check whether final answers match reference results. Research-capability evaluation instead needs to examine whether processes are reviewable, conclusions are verifiable, and budgets are controlled. This article describes a verifiable evaluation framework for theoretical computer science that uses independently checked certificates across theorem proving, program optimization, and construction-improvement tasks. It covers the separation of task definitions, sandboxed execution, verifier checks, and budget control; reports leaderboard results that show clear capability stratification rather than uniform dominance; highlights a 124-vertex, 630-edge girth-5 extremal graph and an improved upper bound A(22,10) ≤ 82; and summarizes failure modes and practical recommendations for building auditable, reproducible research-agent evaluation pipelines.

    Evaluating LLM Research Capabilities: From Problem Solving to Verifiable Discovery

    The fact that a large model can solve competition problems does not necessarily mean it can do research. This question cannot be answered with a simple “yes” or “no.” Competition problems usually have clear conditions, standard answers, and a limited search space. Research tasks, by contrast, involve problem selection, hypothesis generation, proof search, program optimization, construction improvement, result review, and value judgment. A more reasonable evaluation approach is to place the key outputs of the research process into the same verifiable system, so that what the model submits is not a “plausible new answer” but an object, parameter, or proof that can be independently reviewed.

    1. Evaluation Goal: From Answer Matching to Verifiable Outputs

    Traditional benchmarks focus more on whether the final answer matches a reference result. Research-capability evaluation needs to focus on whether the process is reviewable, whether the conclusion is verifiable, and whether the budget is controlled. A verifiable evaluation set for theoretical computer science usually needs to include three types of tasks:

    1. Theorem proving tasks: The model needs to provide a complete proof or a checkable proof script, and the verifier checks whether each step of reasoning follows the rules.
    2. Program optimization tasks: The model needs to submit a runnable program or parameter configuration, and the verifier executes it in an isolated environment and checks performance constraints.
    3. Construction improvement tasks: The model needs to provide discrete objects such as graphs, codes, or sets, and the verifier checks whether the objects satisfy the required properties and compares objective values against reference values.

    The difference from ordinary problem-solving evaluation is that the answer is no longer a unique string but a “certificate.” A certificate can be a proof, a program, an object, or a parameter. As long as the verifier is sufficiently independent, the system can determine whether the model has actually completed the task, rather than generated a beautiful but non-reviewable description.

    2. Verifiable System Design: Tasks, Sandboxes, Verifiers, and Budgets

    To make evaluation trustworthy, the pipeline needs to separate task definition, execution isolation, certificate checking, and budget control. A simplified pseudocode example is as follows:

    1. def evaluate(agent, task, budget, verifier):
    2. trace = []
    3. for attempt in range(budget.max_attempts):
    4. cert = agent.solve(task, budget.remaining())
    5. ok, reason = verifier.check(task, cert)
    6. trace.append({
    7. "attempt": attempt,
    8. "cert": cert,
    9. "ok": ok,
    10. "reason": reason,
    11. "cost": budget.used()
    12. })
    13. if ok:
    14. return {"success": True, "trace": trace}
    15. if budget.exhausted():
    16. break
    17. return {"success": False, "trace": trace}

    This pseudocode emphasizes four principles. First, the agent can influence results only by submitting certificates. Second, the verifier is independent of the agent and cannot rely on the model to verify itself. Third, every attempt records the trace and cost. Fourth, the budget includes the number of attempts, tokens, runtime, and computing resources.

    For engineering implementation, object storage can store tasks and certificates, container platforms can isolate execution environments, logging services can record complete traces, monitoring and alerting can manage budgets and anomalies, and message queues can schedule large numbers of attempt tasks. Baidu AI Cloud, for example, offers general-purpose services in these categories; where available, they can support these functions and improve the reproducibility and auditability of evaluation.

    3. Leaderboard Performance: Capability Stratification, Not Uniform Dominance

    In public evaluation, the top agent scored 71.9 overall, and the second-place agent scored 57.4. More detailed task settings are more informative: without intermediate lemmas, the top agent completed 14 of 20 theorems, for a success rate of 70.0%; the second-place agent succeeded 23 times in 60 attempts, covering 9 different theorems, for a success rate of 38.3%; the remaining six agents all scored no more than 11.7%. These numbers show that leading models already have some proof-search and construction capabilities, but there is clear capability stratification among different agents.

    At the same time, the gap is not simply uniform dominance. On the PCP alphabet reliability task, the second-place agent completed all three attempts, while the top agent did not solve the task. This shows that evaluation cannot rely only on total scores: different models have uneven capability distributions across task families, and some models may have advantages under specific constraints. For developers, model selection should consider task-family fit rather than comparing only a single total score.

    4. Two Intuitive Results: Construction Improvement and Upper-Bound Tightening

    One of the most intuitive results in the paper comes from a discrete construction task. In the girth-5 extremal graph task, an agent constructed a graph with 124 vertices and 630 edges that contains no triangles or quadrilaterals, exceeding the 629-edge reference value used in the task. The value of this type of task is that the result can be independently checked: the number of vertices, the number of edges, the absence of triangles, and the absence of quadrilaterals are all verifiable properties, and the objective value is also easy to compare.

    Another result is the proof that A(22,10) ≤ 82, tightening the published upper bound from 84 to 82. Compared with “generating a plausible new answer,” proving an improved upper bound is closer to research output: it requires not only a conclusion but also a proof process that can be reviewed. Even if the improvement is limited, such results show that under a clear problem, sufficient budget, and verifiable feedback, agents can complete some proof, program optimization, and construction improvement.

    5. Failure Modes and Capability Boundaries

    The boundaries exposed by verifiable evaluation are equally important. First, models are good at searching in tasks with clear problem definitions and explicit feedback signals; however, autonomously selecting research directions and judging the value of results still depend on humans. Second, a model may succeed over multiple attempts, but the success rate is strongly related to the number of attempts and the budget, so capability cannot be discussed without cost. Third, a model may provide intermediate steps that are partially correct but not reviewable, so the verifier must be strict and cannot check only the final string. Fourth, open problems usually lack immediate verifiers, and models have difficulty obtaining stable feedback, which significantly limits their autonomy.

    6. Engineering Implementation Recommendations

    When enterprises or research teams build a research-agent evaluation pipeline, they can proceed in five steps:

    1. Define verifiable tasks: Prioritize task families with checkable certificates and comparable objectives.
    2. Build independent verifiers: Decouple verification logic from the agent and retain verification logs.
    3. Control budgets and attempt counts: Record success rates, the number of covered tasks, and unit success cost at the same time.
    4. Retain complete traces: Save prompts, certificates, execution results, and failure reasons for review.
    5. Report results by task family: Avoid using only a total score to obscure capability differences.

    7. Conclusion

    This work is more like a topographic map of AI research capability: models can already complete some proof, program optimization, and construction improvement under clear problems, sufficient budgets, and verifiable feedback; however, autonomously selecting research directions, judging the value of results, and solving major open problems still require humans. For developers, the key is not to ask whether a model can do research, but to build verifiable, auditable, and reproducible evaluation systems so that every “research output” from a model can be independently reviewed.