Evaluating LLM Research Capability: From Problem Solving to Verifiable Research Output
Can large language models actually do scientific research? This article examines a verifiable research benchmark that reframes the question: instead of relying on exam-style scores, research output is broken down into independently checkable artifacts such as proofs, constructions, code optimizations, and parameter searches. It covers benchmark task design, verifier architecture, anti-reward-hacking measures, and how to interpret results as a capability map rather than a leaderboard, along with practical engineering guidance for building reproducible research evaluation systems.
Evaluating LLM Research Capability: From Problem Solving to Verifiable Research
1. Why Problem-Solving Skill Is Not Research Skill
Large language models perform impressively on competition problems, coding challenges, and standardized exams. This naturally leads to an assumption: if a model can solve formalized problems, does it also possess genuine research capability? The answer is not straightforward. Competition problems typically have clear boundaries, unique or limited answers, well-established solutions, and immediate feedback. Research tasks, by contrast, involve open-ended problems, long-horizon planning, value judgments, experiment design, failure analysis, and result presentation. The two share some reasoning skills, but research capability additionally requires a model to continuously produce verifiable intermediate artifacts under uncertainty, with final outputs that can be independently reviewed.
So whether a model “can do research” should not be judged by whether it produces a plausible-sounding answer. The real test is whether it can produce reviewable proofs, constructions, parameters, or program improvements given a well-defined problem, a sufficient budget, and verifiable feedback. A more sensible evaluation approach places the key outputs of the research workflow into a single verifiable system: does the intermediate lemma hold, does the construction satisfy all constraints, does the program optimization genuinely improve the metric, can the proof script be machine-checked, and can the conclusion be reproduced within a defined budget?
This is exactly the value of a verifiable research benchmark. Rather than summarizing all of research with a single problem, it decomposes “research capability” into observable dimensions, forming a capability map.
2. Benchmark Design: Turning Research Output into Verifiable Objects
2.1 Task Types and Verification Targets
Verifiable research benchmarks typically cover several task categories: theorem proving, combinatorial construction, program optimization, reliability verification, parameter search, and counterexample construction. Each requires the model to submit objects that can be independently checked, not just natural-language explanations. For example:
- Theorem proving tasks: submit a proof script or formal proof; the verifier checks whether each step is valid.
- Construction tasks: submit vertices, edges, parameters, or structural objects; the verifier checks whether all constraints are satisfied and compares against reference values.
- Program optimization tasks: submit code patches or configurations; the verifier runs tests and performance benchmarks in an isolated environment.
- Reliability verification tasks: submit objects, parameters, or proofs; the verifier checks whether the property holds stably across multiple sets of inputs.
The core principle is “reviewability.” What the model submits must not merely look like a novel answer—it must be an object, parameter, or proof that can be independently reviewed. The stronger the verifier, the closer the evaluation gets to true scientific “testability.” The weaker the verifier, the easier it is for fluent language to mask a lack of substance.
2.2 Scoring Logic: Beyond a Single Score
A verifiable research benchmark should not rely on one aggregate score. A total score reflects overall capability but can mask task heterogeneity. A more complete scoring system should include:
- Total score and per-task weighted scores;
- Number of successful tasks, theorems solved, and coverage breadth;
- Attempt budget versus success rate;
- Whether intermediate lemmas or hints were provided;
- Whether submissions passed independent verification;
- The distribution of failure types, such as constraint violations, incomplete proofs, budget exhaustion, or verifier rejections.
In published results, top models may lead on total score, but the gap is not a clean sweep. For example, in a setting with no intermediate-lemma hints, one model completed 14 of 20 theorems (roughly 70% success), while another succeeded 23 times out of 60 attempts, covering 9 distinct theorems (about 38.3% success); most other agents did not exceed 11.7%. These numbers show that differences among models appear not only in “whether they can solve a task,” but also in “how many task types they cover,” “how reliably they complete work within a given budget,” and “whether they need hints.”
3. Verifier Architecture: Making “Looks Reasonable” Become “Checkable”
3.1 The Basic Workflow
A verifiable research evaluation system can be abstracted into five components: task definitions, a runtime environment, submissions, verifiers, and a scorer. Here is the pseudocode:
class ResearchTask:def __init__(self, constraints, budget, verifier):self.constraints = constraintsself.budget = budgetself.verifier = verifierdef validate(self, submission):# submission contains reviewable objects: artifacts, proof, params, patch, etc.if not self.check_budget(submission.attempts, self.budget):return {"pass": False, "reason": "budget_exceeded"}if not self.constraints.check(submission.artifacts):return {"pass": False, "reason": "constraint_violation"}result = self.verifier.check(artifacts=submission.artifacts,proof=submission.proof,params=submission.params,patch=submission.patch)return {"pass": result.ok, "detail": result.detail}
This pseudocode emphasizes three principles: first, submissions must be structured objects; second, budgets must be tracked and enforced; third, verification results must include interpretable failure reasons. Without these three, an evaluation easily degenerates into “the model talking to itself.”
3.2 Isolated Environments and Reproducibility
The verifier must run submissions in an isolated environment, preventing the model from “cheating” by reading test files, modifying verification logic, or relying on external state. From an engineering perspective, task snapshots, submissions, verification logs, and scoring results can be stored separately, along with the environment image, random seeds, dependency versions, and runtimes. Such a system can be deployed on general-purpose infrastructure—a container platform, object storage, logging services, and monitoring/alerting—using container isolation for the runtime, object storage for task snapshots and submissions, and logging services for the verification trail, improving the reproducibility and auditability of the evaluation.
3.3 Guarding Against Reward Hacking
A verifiable system is not automatically a fair one. You must guard against models exploiting verifier loopholes to game the reward—for example, submitting special cases, exploiting floating-point errors, bypassing constraint checks, or probing test cases within the program. Countermeasures include cross-checking with multiple verifiers, hidden test sets, randomized constraints, limits on runtime and memory, static checks on submissions, and manual sampling of failed cases. The closer the verifier gets to real-world peer review and reproduction, the more meaningful the evaluation.
4. Interpreting Results: A Capability Map, Not a Leaderboard
4.1 Theorem Proving: Hint Dependence and Coverage
Theorem proving tasks best illustrate the value of intermediate lemmas. If a model completes many theorems without intermediate-lemma hints, it demonstrates some capacity for long-horizon reasoning and proof organization. But success rate and theorem coverage matter equally. One model might repeatedly attempt a small number of theorems at high budget, achieving a seemingly decent success rate with limited coverage; another might have a lower per-attempt success rate yet cover more task types. Evaluations should report attempt counts, successes, theorem coverage, and failure reasons together.
4.2 Task Heterogeneity: Local Upsets Show There Is No Clean Sweep
Different tasks demand different capabilities. In one alphabet-based reliability verification task, one model succeeded on all three attempts, while the top-scoring model failed to solve it. This shows a leaderboard is not a total domination: a model with a high overall score may be stronger on mainstream proof tasks but have blind spots on tasks with specific constraints, specific verification methods, or specific constructions. For engineering teams, this heterogeneity is more informative than a single score, because it reveals opportunities in model selection, task routing, and prompting strategies.
4.3 Constructions and Bound Improvements: The Direct Value of Verifiable Output
Construction tasks yield the most tangible results. For example, on an extremal graph task with girth 5, one model constructed a graph with 124 vertices and 630 edges containing no triangles or quadrilaterals, beating the task’s reference value of 629 edges; in a combinatorial design task, one model tightened a previously published upper bound from 84 to 82. The value of such results lies in the fact that the submission is not “something that looks like a new answer” but an object or proof that satisfies all constraints and can be independently reviewed. Even though these results remain far from major open problems, they demonstrate that with a well-defined problem, sufficient budget, and verifiable feedback, models can complete some proofs, program optimizations, and construction improvements.
5. Engineering Guidance: Building a Reproducible Research Evaluation
5.1 Task Tiering and Budget Control
Organize tasks into four tiers: basic verification, combinatorial construction, theorem proving, and open-ended exploration. Basic verification calibrates the verifier; combinatorial construction assesses constraint satisfaction and search ability; theorem proving assesses long-horizon reasoning; open-ended exploration observes model strategies under weak feedback. Give each tier an independent budget so that high-budget tasks do not dominate the overall score.
5.2 Verifier Quality Before Model Rankings
In the early stages of building an evaluation, prioritize verifier quality over model rankings. The verifier should cover constraint checking, independent recomputation, boundary testing, randomized testing, and manual sampling of results. For program optimization tasks, check both correctness and performance; for proof tasks, check the validity of every step; for construction tasks, check all constraints rather than sampling.
5.3 Human-in-the-Loop Review
Models can generate candidate proofs, constructions, and patches, but judging the value of results, choosing research directions, and tackling major open problems still require humans. A recommended workflow is “model generates → verifier pre-screens → human reviews → reproduction experiments.” The model expands the search space, the verifier filters out unreviewable output, and humans judge novelty, significance, and follow-up directions.
5.4 Data Contamination and Fairness
Evaluations must guard against training-data contamination. Tasks should include newly constructed problems, parameter perturbations, and hidden test sets; public leaderboards should distinguish “has seen similar problems” from “unseen problems”; and for high-budget attempts, complete trajectories should be logged rather than reporting only successful samples. Fairness also means consistent budgets, consistent prompts, and consistent runtime environments across models.
6. Conclusion: Models Can Produce Some Verifiable Research Output, But Open Research Still Needs Humans
Returning to the original question: being able to solve competition problems does not mean a large language model can do research. A more accurate statement is this: given a well-defined problem, sufficient budget, and verifiable feedback, models can already complete some proofs, program optimizations, and construction improvements. They can submit objects, parameters, or proofs that can be independently reviewed, rather than merely producing plausible-sounding answers. But autonomously choosing research directions, judging the value of results, posing major open problems, and persisting over the long term under weak feedback still depend on human involvement.
A verifiable research evaluation is therefore better understood as a capability map than a simple leaderboard. It shows which tasks models can already handle, which tasks still require hints, where their blind spots lie, and what loopholes remain in the verifier itself. For developers and enterprise users, the more valuable approach is not chasing a single score, but building verifiable task sets around your own business problems, running isolated environments, logging complete trajectories, and positioning the model as a “candidate generator” and “reviewable-output generator”—then continuously calibrating its capability boundaries through a systematic, engineering-driven evaluation process.
