Evaluating LLM Scientific Research Capabilities: From Problem Solving to Verifiable Research
While large language models excel at solving competition problems with predefined constraints, true scientific research involves open-ended questions, long trial-and-error cycles, and subjective value judgments. This article explores the gap between solving problems and conducting research, introducing a verifiable evaluation system designed for research workflows. It covers capability differences, system design, result interpretation, engineering practices, and the boundaries of human intervention, offering a practical guide to building an LLM research capability evaluation pipeline.
Evaluating LLM Scientific Research Capabilities: From Problem Solving to Verifiable Research
Can a large language model (LLM) that solves competition problems be considered capable of scientific research? This is a question easily masked by “high scores.” Competition problems typically feature clear premises, closed answers, and short reasoning chains. Research tasks, however, involve open problems, direction selection, long-term trial and error, evidence organization, and value judgments. There is a significant capability gap between the two. Recently, a research team proposed a verifiable evaluation system tailored for research workflows, incorporating theorem proving, program optimization, and structural improvements into a unified, reviewable framework. It attempts to answer a more engineering-oriented question: Are the model’s outputs merely “plausible new answers,” or are they independently verifiable objects, parameters, or proofs?
The significance of such evaluation lies not in creating new leaderboards, but in decomposing “research capability” into a set of definable, executable, and verifiable tasks. This article will discuss how to build an evaluation pipeline for LLM research capabilities across five aspects: capability differences, system design, result interpretation, engineering practices, and human boundaries.
1. Capability Differences Between Competition Problems and Research Tasks
The core of a competition problem is “finding a unique or limited answer under known constraints.” Models can achieve high performance through pattern matching, step generation, and self-checking. However, research tasks introduce at least four additional layers of complexity:
- Incomplete Problem Definition: Research directions must be extracted from literature, experimental phenomena, and theoretical gaps; the premises are not always clear.
- High Verification Cost: A conjecture, construction, or optimization result may require formal proof, program execution, or mathematical review.
- Open Search Space: Models must not only provide answers but also decide which lemmas, construction strategies, parameter intervals, and counterexample directions to explore.
- Subjective Value Judgment: Whether a result is important, novel, or worth further investment often depends on the judgment of domain experts.
Therefore, an evaluation system cannot only check whether the final answer is correct; it must also examine whether the submission contains an independently verifiable evidence chain. For example, theorem proving requires checkable proof objects or formal fragments; program optimization requires runnable code, input/output, and performance records; construction tasks require explicit objects (like vertices, edges, parameters) for the verifier to check constraints.
2. Design Points of a Verifiable Research Evaluation System
A verifiable evaluation system typically consists of five parts: task definition, submission specification, verifiers, budget control, and statistical metrics.
1. Task Definition and Submission Standardization
Tasks should clearly define inputs, constraints, allowed resources, and success criteria. Submissions should not be limited to natural language conclusions but must include executable or checkable objects. For instance, theorem proving tasks require proof scripts, lemma lists, and dependencies; construction tasks require structured data; optimization tasks require code, execution logs, and result metrics. Standardizing submissions reduces manual review costs and facilitates batch processing by automated verifiers.
2. Verifiers and Budget Control
The verifier checks whether constraints are satisfied, proofs are reviewable, programs are runnable, and results meet the threshold. Budget control includes the number of attempts, execution time, token budget, and tool call frequency. Research is not infinite trial and error; the evaluation must record budgets, otherwise “taking the best of multiple attempts” will mask the true capability. Statistics should simultaneously report single-attempt success rate, multiple-attempt success rate, number of tasks covered, and failure types.
3. Metric Design
Metrics should avoid the misleading nature of a single total score. It is recommended to report simultaneously: total score, task success rate, success rates across different task categories, number of attempts, number of theorems covered, distribution of verification failure reasons, and manual review pass rate. For construction tasks, the constraint satisfaction rate and improvement magnitude relative to the reference value should also be reported; for proof tasks, whether intermediate lemmas are used, proof length, and verifiability should be reported.
4. Pseudocode Example
from dataclasses import dataclassfrom typing import Any, Dict, List@dataclassclass Submission:task_id: strartifacts: Dict[str, Any] # Proofs, code, construction data, parameters, etc.budget_used: Dict[str, int] # Number of attempts, runtime, tokens, etc.logs: List[str]@dataclassclass VerifyResult:passed: boolscore: floatreason: strevidence: Dict[str, Any]def evaluate(task, submission: Submission, verifier) -> VerifyResult:# 1. Check submission completenessif not verifier.check_schema(task, submission):return VerifyResult(False, 0.0, "Incomplete submission structure", {})# 2. Check if budget is exceededif not verifier.check_budget(task, submission):return VerifyResult(False, 0.0, "Budget exceeded", {})# 3. Execute verifiable validationresult = verifier.verify(task, submission)# 4. Record evidence and failure reasonsreturn VerifyResult(passed=result.passed,score=result.score,reason=result.reason,evidence=result.evidence)
The pseudocode above emphasizes three principles: structured submissions, auditable budgets, and verified results with evidence. In engineering, log retention, random seed recording, environment version locking, and manual spot-checking should also be included to ensure evaluation reproducibility.
3. Result Interpretation: Not a Simple Comprehensive Overwhelming
Public leaderboards show that a general-purpose model ranked first with a score of 71.9, while another model ranked second with 57.4. In theorem proving tasks, the former completed 14 out of 20 theorems without intermediate lemmas, achieving a 70.0% success rate; the latter succeeded 23 times in 60 attempts, covering 9 different theorems, with a success rate of 38.3%; the remaining six agents scored no higher than 11.7%. This data indicates that leading models can already complete some proof work under clear problems, sufficient budgets, and verifiable feedback.
However, the results are not a comprehensive crushing. In the PCP alphabet reliability task, the second-ranked model completed all three attempts, whereas the first-ranked model failed to solve it. This indicates that different models exhibit distinct capability differences across various task types: leading in the total score does not mean leading in every subfield. Evaluations should retain a fine-grained task view to prevent a single ranking from masking long-tail capabilities.
Two of the most intuitive results in the paper are also noteworthy: in the extremal graph construction task with a girth of 5, one model constructed a graph with 124 vertices and 630 edges without triangles or quadrilaterals, exceeding the 629-edge reference value; another model proved A(22,10) ≤ 82, tightening the published upper bound from 84 to 82. The key to these results is not that the “answers look new,” but that they submitted independently verifiable construction objects or proof parameters. The evaluation system transforms these outputs into checkable evidence through verifiers.
4. Why Verifiable Feedback is Currently Crucial
The advantage of LLMs in research tasks often stems from the “generate-verify-correct” closed loop. As long as a task provides clear feedback, the model can iterate within the search space: fixing lemmas after a proof failure, optimizing structure after a program timeout, or adjusting parameters when a construction fails to meet constraints. Verifiable feedback transforms natural language generation into an auditable search process.
However, verifiers themselves have boundaries. First, the verifier’s coverage determines the upper limit of the evaluation. Second, the cost of formal proofs is high, making fully automated verification difficult in many domains. Third, models might exploit verifier vulnerabilities for reward hacking. Fourth, statistics from multiple attempts must prevent “brute-force enumeration” from being misinterpreted as research capability. Therefore, evaluation systems should combine automated verification with manual spot-checking, record failure reasons, and disclose task constraints and budgets.
5. Engineering Implementation: Building a Reproducible Evaluation Pipeline
In evaluation platforms within enterprises or research institutions, task packages, submissions, verifiers, logs, and result dashboards can be decoupled. Task packages include prompts, constraints, reference baselines, and budgets; submissions are archived via object storage; verifiers run in isolated containers; logs and monitoring alerts record runtime status; dashboards display total scores, sub-task success rates, and failure distributions. Using general cloud capabilities such as container platforms, object storage, logging services, and monitoring alerts is sufficient to build a reproducible pipeline. The focus should be on auditability, replayability, and extensibility.
6. Conclusion: A Topographical Map of AI Research Capabilities
Returning to the original question, this evaluation acts more like a topographical map of AI research capabilities: models can already complete some proofs, program optimizations, and structural improvements under clear problems, sufficient budgets, and verifiable feedback. However, autonomously selecting research directions, judging the value of results, and solving major open problems still cannot be done without humans. For developers, a more pragmatic approach is to place LLMs into “verifiable subtasks,” using verifiers to constrain outputs, budgets to control costs, and manual review as a fallback, rather than directly equating high scores in competition problems with research capability.
