Evaluating LLM Research Capabilities: From Problem Solving to Verifiable Discovery
Strong performance on competition problems does not automatically translate into research ability. This article examines why frontier models can excel at well-defined tasks yet still struggle with open-ended scientific work, and introduces a verifiable evaluation framework built on auditable outputs—proofs, programs, constructions, and parameter bounds—checked through deterministic verifiers. It covers the framework's four-layer design, task families, budget protocols, notable results such as improved combinatorial constructions and tightened upper bounds, and the engineering practices needed to build a reproducible evaluation pipeline. Readers will learn how to measure and interpret LLM research capabilities beyond a single leaderboard score.
Strong performance on competition problems does not automatically imply research capability. Competition problems typically have clear statements, fixed answers, and established solution methods. Research tasks, by contrast, require working in open-ended spaces: proposing testable intermediate conclusions, constructing objects, optimizing procedures, or tightening parameter bounds—with results that must be independently reviewable. A verifiable evaluation framework addresses this gap with one core idea: place all research outputs—proofs, programs, constructions, and parameters—into an auditable, reproducible verification system, replacing “does it look plausible” with “can it be independently checked.”
1. Problem Definition: Research Capability Requires Verifiable Outputs
When discussing whether large language models can do research, the first pitfall to avoid is treating “does the answer look right” as the evaluation standard. The value of a research output lies not only in producing a new conclusion, but in whether others can review, reproduce, and build upon it. The evaluation target should therefore be verifiable artifacts rather than a piece of natural-language prose, for example:
- Proof artifacts: formal or semi-formal proofs, key lemmas, and their dependency graphs;
- Program artifacts: executable code, optimized parameter configurations, test cases, and run logs;
- Construction artifacts: graphs, matrices, codes, or other combinatorial structures satisfying specific constraints;
- Parameters and bounds: numerically checkable upper bounds, lower bounds, and interval tightening results.
The corresponding verifiers should be as deterministic or auditable as possible: constraint satisfaction checks, independent program testing, numerical verification, proof checkers, counterexample search, and so on. The framework must also record protocol information such as attempt counts, budgets, tool permissions, and whether intermediate lemmas are provided—otherwise results across models are not comparable. In short, research capability evaluation is not a single-turn Q&A leaderboard, but a closed-loop system of “task registration — execution — verification — audit.”
2. Framework Design: Task Families, Verifiers, and Budget Protocols
A practical evaluation framework typically involves four layers.
The first layer is task family classification. Tasks can be grouped by capability type: theorem proving, algorithmic construction, program optimization, reliability tasks, and parameter bound tightening. Different families probe different abilities: proving tasks examine lemma organization and logical chains; construction tasks examine constraint search and combinatorial intuition; program optimization examines the balance between performance and correctness; reliability tasks examine stability under edge conditions.
The second layer is verifier registration. Every task must be bound to one or more verifiers, whose output should include pass/fail status, failure reasons, review logs, and a reproducible experiment entry point. Verifiers must not rely on model self-reports—they must execute independently.
The third layer is budget and protocol control. For the same task, attempt counts, compute budgets, tool-call permissions, and whether intermediate lemmas are provided must all be recorded for each model. In the original evaluation, “no intermediate lemmas provided” and “multiple attempts allowed” significantly affected success rates, so the budget protocol is part of result interpretation.
The fourth layer is audit and human review interfaces. Passing automated verification does not establish scientific value; human judgment is still needed on problem selection, method novelty, and result significance. The system should retain complete logs for expert review.
A schematic interface:
interface Task {id: stringfamily: stringstatement: stringbudget: { attempts: int, toolCalls: int, timeLimit: int }verifier: Verifier}interface Verifier {check(artifact): { passed: boolean, reason: string, log: string }}run(task, model):for attempt in 1..task.budget.attempts:artifact = model.solve(task.statement, task.budget)result = task.verifier.check(artifact)record(task.id, attempt, artifact, result)if result.passed: return successreturn failure
The key point of this design: models submit verifiable artifacts rather than “final answer text,” and the system records not a single score but attempt counts, task coverage, success rates, and failure reasons.
3. Interpreting Results: Capability Layering and Complementarity
Public evaluation results show clear capability stratification. On the overall score dimension, first-tier models scored about 71.9 out of 100, while second-tier models scored about 57.4. Without intermediate lemmas provided, first-tier models completed 14 of 20 theorems, a 70.0% success rate; second-tier models succeeded 23 times across 60 attempts, covering 9 distinct theorems, a 38.3% success rate; all other evaluated agents scored no more than 11.7%. These numbers show that top models can already complete some proving and construction tasks given well-defined problems, sufficient budgets, and verifiable feedback—yet there remains a clear gap to stable research output.
More importantly, the results are not a simple across-the-board dominance. On a PCP alphabet reliability task, second-tier models completed it within three attempts, while first-tier models failed to solve it. This reveals complementarity across task families: a leading total score does not mean leading every subtask, and failure on a single problem does not mean weak overall capability. Evaluations should report total scores, per-task success rates, theorem coverage, and attempt counts together, rather than letting a single ranking obscure structural differences in capability.
4. Representative Results: What Construction Improvements and Tighter Bounds Mean Technically
The most tangible results fall into two categories. The first is improved combinatorial constructions: on a classic extremal graph theory task, a second-tier model constructed a graph with 124 vertices and 630 edges containing no triangles or quadrilaterals, beating the task’s 629-edge reference value. The value of such a result lies in independent checkability: vertex count, edge count, and forbidden subgraph conditions are all deterministically verifiable. The model did not produce a “plausible-sounding new answer”—it submitted a reviewable graph structure.
The second is parameter bound tightening: a first-tier model proved A(22,10) ≤ 82, improving the published upper bound from 84 to 82. Upper bound proofs require checkable logical chains, and the verifier can review key steps. Regardless of ultimate mathematical significance, the technical meaning of such output is that models can generate verifiable proof artifacts around explicit goals and complete part of the intermediate reasoning within budget.
5. Engineering Implementation: Building a Reproducible Evaluation Pipeline
For enterprises and research teams, the engineering focus of verifiable research evaluation is not chasing a one-off leaderboard position, but building a reproducible pipeline. Recommended practices include:
- Task registration and version management: record each task’s statement, verifier version, budget protocol, and reference baseline;
- Isolated execution environments: run model-generated programs in container platforms with restricted network and file permissions;
- Independent verifiers: keep verifiers separate from model execution environments so models cannot influence the verification process;
- Logging and auditing: store proofs, code, run logs, and verification results in object storage, and use logging services and monitoring alerts to track budget consumption and failure distributions;
- Human review queues: route automatically verified outputs to expert review, focusing on problem selection, method novelty, and result significance.
At the infrastructure level, generic capabilities such as object storage, container platforms, logging services, and monitoring alerts can be used to assemble the pipeline—but the core remains the task protocol, verifier quality, and audit mechanisms.
6. Boundaries and Conclusion: Models Can Do Some Research, But Not Replace Research Judgment
Returning to the original question: an LLM that solves competition problems is not an LLM that does research. Current evaluations are more like a topographical map of AI research capability: given well-defined problems, sufficient budgets, and verifiable feedback, models can already complete some proofs, program optimizations, and construction improvements. But when it comes to autonomously choosing research directions, judging the value of results, or posing major open problems, human oversight remains indispensable. The purpose of an evaluation framework is not to manufacture a “comprehensive surpassing” narrative, but to turn model outputs into independently reviewable artifacts—making capability boundaries, budget effects, and failure reasons visible, comparable, and reproducible.
