LLM Research Capability Evaluation: From Problem Solving to Verifiable Outputs
This article examines how to evaluate the research capability of large language models by focusing on independently verifiable artifacts rather than standard answers. It outlines design principles for a verifiable research benchmark, interprets representative task results, and discusses engineering and interpretation practices for building sustainable evaluation platforms.
LLM Research Capability Evaluation: From Problem Solving to Verifiable Outputs
1. The Starting Question: Does Strong Competition Performance Mean Research Ability?
The performance of large language models on math competitions, coding contests, and standardized exams can easily create an intuition: if a model can solve formal problems, can it also conduct research? The answer is not so simple. Competition problems are usually closed problems with clear conditions, a unique answer, and a clear verification path. Research work, by contrast, deals with open problems and involves selecting a topic, building models, proposing hypotheses, constructing lemmas, designing experiments, checking results, and judging value. The difference is not merely one of difficulty, but one of problem structure.
Therefore, evaluating a model’s research capability should not rely only on whether its final answer matches a standard solution. It should examine whether, under a given budget, the model can produce artifacts, parameters, or proofs that can be independently reviewed. This is also the core design idea behind a verifiable research evaluation system: place key outputs from the research process into the same verification framework and judge their validity through automated checks plus human audits, rather than letting the model provide a new answer that merely “looks reasonable.”
From a technical perspective, this kind of evaluation is closer to a “research output verification pipeline” than to traditional benchmark scorekeeping. It must answer at least three questions: what type of object the model produces; whether the verifier can independently review it at an acceptable cost; and whether the evaluation metrics can distinguish “success through accidental attempts” from “stable problem solving.”
2. Evaluation System Design Principles: Verifiability First, Layered Metrics
A research-capability evaluation system should follow the following design principles.
First, prioritize verifiable objects. Task outputs should, as much as possible, be proofs, programs, constructions, parameters, or executable experiments. For example, theorem proving can require a formal or semi-formal proof; program optimization can require runnable code and performance metrics; combinatorial construction can require vertices, edge sets, and check results showing the absence of specific subgraphs. The verifier should run independently of the model to avoid self-verification.
Second, use layered tasks rather than a single total score. Research capability is not a single dimension. Evaluation should cover at least theorem proving, program optimization, combinatorial construction, reliability tasks, and counterexample search, with separate metrics such as solve rate, attempt success rate, theorem coverage, construction improvement, and reviewability rate. A total score can be used for ranking, but it cannot replace diagnostic subscores.
Third, make budgets and attempt counts transparent. Research is not unlimited trial and error. Evaluation should fix the budget for time, tokens, tool calls, and attempts, and record model behavior within that budget. Otherwise, a model might “happen” to succeed through many attempts without possessing stable research ability. Metrics in public evaluation summaries—such as “23 successes in 60 attempts, covering 9 different theorems, with a 38.3% success rate”—are intended to distinguish coverage breadth from per-attempt success rate.
Fourth, prevent leakage and ensure reproducibility. Task repositories, reference solutions, verifier versions, and runtime environments should all be versioned. Hidden test sets, cross-validation, and manual spot checks should be combined to reduce the risks of data contamination and benchmark overfitting. Evaluation results should be reproducible, with at least inputs, outputs, verification logs, and resource consumption records retained.
A simplified evaluation pipeline can be represented by the following pseudocode:
def run_benchmark(runner, tasks, budget, verifier):report = []for task in tasks:attempts = []for step in range(budget.max_attempts):artifact = runner.solve(task.prompt, budget)result = verifier.check(task, artifact)attempts.append(result)if result.status == "verified":breakreport.append(summarize(task, attempts))return aggregate(report)
The key to this pseudocode is not the interface but the process: each attempt generates a verifiable artifact, the verifier makes an independent judgment, and results are aggregated by task. Engineering implementation also requires sandbox isolation, timeout control, resource quotas, log auditing, and result storage.
3. Representative Tasks and Result Interpretation: A High Total Score Does Not Mean Comprehensive Dominance
In a set of public evaluation summaries, the highest score on the full leaderboard was about 71.9, and the second-highest was about 57.4. More noteworthy are the subscores: the leading model completed 14 of 20 theorems without intermediate lemma hints, for a 70.0% success rate; another model succeeded 23 times in 60 attempts, covering 9 different theorems, for a 38.3% success rate; the remaining six agents each scored no more than 11.7%. These data show that strong models already have some research-assistance capability on verifiable proof tasks, but differences in attempt efficiency, coverage breadth, and stability are significant.
Another representative phenomenon is the reversal caused by task structure. On the PCP alphabet reliability task, the model with the second-highest total score completed all three attempts, while the model with the highest total score did not solve the task. This reminds us that research-capability evaluation cannot rely only on overall ranking; task type, verification form, and prompt settings can all significantly affect performance. A model may be strong in proof search but not dominant on a particular reliability or construction task.
Construction tasks better demonstrate the value of “verifiable outputs.” In an extremal graph task with girth 5, one model constructed a graph with 124 vertices and 630 edges containing no triangles or quadrilaterals, exceeding the task’s reference value of 629 edges. In a coding theory task, one model proved A(22,10) ≤ 82, tightening the published upper bound from 84 to 82. The significance of such results is not that “the model replaces mathematicians,” but that, given a clear problem, sufficient budget, and verifiable feedback, a model can complete part of the proof, program optimization, and construction improvement. What it provides is a reviewable object, not an unverifiable natural-language assertion.
4. Engineering Implementation: Turning Evaluation into a Sustainable Platform
To turn the above evaluation from a paper experiment into a sustainable platform, engineering support is needed. A task repository is responsible for versioning problems, reference solutions, and verifiers; a sandbox execution environment isolates model code and external dependencies; object storage saves artifacts, logs, and metrics; logging services and monitoring alerts record runtime status; and a container platform elastically schedules resources by task. General-purpose cloud services—such as container platforms, object storage, and logging services—can support the computing, storage, and observability needs of such an evaluation pipeline, but the specific implementation should remain interface-neutral and portable.
The metrics dashboard should include at least: total score, per-task score, solve rate, attempt success rate, theorem coverage, average number of attempts, reviewability rate, human review pass rate, and resource consumption. For data governance, it is necessary to prevent test set leakage, record verifier versions, and conduct manual spot checks on high-risk tasks. For security, network and file system access should be restricted, and timeouts and resource limits should be set to prevent models from “reward hacking” through unexpected means.
5. How to Interpret It Correctly: A Terrain Map of AI Research Capability
A more reasonable interpretation is that this work depicts a terrain map of AI research capability. Models can already complete part of the proof, program optimization, and construction improvement under clear problems, sufficient budgets, and verifiable feedback; but autonomously choosing research directions, judging the value of results, and solving major open problems still require humans. Competition scores cannot be directly equated with research capability, and a single leaderboard cannot replace multidimensional diagnosis.
For enterprises and developers, the practical recommendations are: first define verifiable tasks, then design verifiers; first build subscores, then total-score rankings; treat models as research assistants and verification aids rather than replacements for researchers; establish human-in-the-loop review processes; and continuously update task sets to prevent benchmark overfitting. Only in this way can evaluation move from “score chasing” to “capability diagnosis” and truly serve engineering practice in research assistance, code optimization, and complex problem solving.
