Can Large Language Models Do Research? What Verifiable Benchmarks Reveal About AI's Capability Boundaries
It is now common to see large language models score full marks on math competitions and win programming contests. This article asks a sharper question: does solving hard competition problems mean a model can conduct independent research? Using a research-grade benchmark in theoretical computer science (TCS) as a case study, it shows how verifiable evaluation differs from vague comparisons of "who thinks more like a human"—by placing every research output into one system that can be independently checked. It breaks down the benchmark's task design, scoring, and result interpretation (including a leader scoring 71.9 versus 57.4 for the runner-up, theorem-proof success rates, an improved extremal graph construction, and a tightened coding-theory bound), and extends the discussion into how enterprises can build their own AI-assisted research workflows with clear problems, automatic verification, and controlled budgets.
Introduction: Solving Competition Problems Is Not the Same as Doing Research
Large language models that score full marks on math competitions or win programming contests are no longer rare. This raises an important question: when a model can solve high-difficulty competition problems, does it already possess the ability to conduct research independently?
Among the various approaches to evaluating the research capability of large models, one line of thinking is worth attention: instead of comparing “who looks more like they are thinking,” it places the outputs of every stage of the research process into the same verifiable system and directly tests “whether the model produces results that stand up to independent review.” This approach is being advanced jointly by university researchers and industry leaders, bringing real research activities—theorem proving, mathematical constructions, tightening parameter bounds, and program optimization—into a unified benchmark.
This article uses a research-grade benchmark targeting theoretical computer science (hereafter the “TCS benchmark”) to unpack its task design, scoring methods, and result interpretation, and then extends the discussion to how enterprises can build their own “AI-assisted research” workflows.
1. Why “Strong at Problem Solving” Does Not Equal “Able to Do Research”
To answer the question in the title, first look at the fundamental difference between competition problems and real research problems:
| Dimension | Competition problem | Real research problem |
|---|---|---|
| Source | Given by the setter; well-defined and closed in scope | Proposed or chosen by the researcher; boundaries may be fuzzy |
| Goal form | A single or limited set of standard answers | A stronger construction, a tighter bound, a better algorithm, or a new proof |
| Evaluation | Judged against a standard answer | Requires independent review, sometimes peer review |
| Core challenge | Finding a clever solution within a time limit | Forming hypotheses, trial and error, judging the value of a result |
Competition scores reflect “reasoning and computation ability given a clearly defined goal.” Research, by contrast, requires closing the full loop under conditions of directional uncertainty and insufficient feedback: posing a question, designing a method, verifying results, assessing value, and continuing to iterate.
This leads to a key distinction: “generating a plausible-looking answer” and “delivering a result that can be independently reviewed” are two different things. The former requires only internal consistency at the language level; the latter demands that proofs can be checked, constructions can be verified, and performance can be reproduced. Therefore, measuring research capability requires one core criterion: whether the output can be independently reviewed.
2. How a Verifiable Research Benchmark Is Designed
The design logic of the TCS benchmark can be summarized in three points: tasks that closely mirror the real research process, scoring based on objective verification, and results broken down by capability dimension. Each part is expanded below.
2.1 Tasks: Covering the Core Research Activities of Theoretical Computer Science
Theoretical computer science (TCS) is one of the most suitable fields to serve as a “litmus test” for research capability. On one hand, most of its results can be automatically judged through formal methods or programmatic checks. On the other hand, it simultaneously demands rigorous reasoning, creative construction, and algorithmic thinking, aligning closely with the elements of research activity. The TCS benchmark mainly covers the following task types:
- Theorem proving: deriving that a proposition holds under given conditions; some tasks require the model to work without intermediate lemmas provided in advance (i.e., without “scaffolding”).
- Mathematical constructions: finding objects that satisfy constraints in problems such as graph theory, for example extremal graphs with no short cycles.
- Tightening parameter bounds: improving existing upper or lower bounds, for example the upper bound on the maximum code size in coding theory.
- Program optimization: improving the efficiency of a concrete computation, reducing time or resource usage while keeping the result correct.
These tasks share one feature: the result has a clear form that a program can judge as correct or incorrect.
2.2 Scoring: Verifiability as the Sole Judge
This kind of benchmark no longer relies on subjective judgments like “whose proof is more elegant,” but instead lets a verification program act as the judge:
- Proofs must be complete and valid, and must pass formal checks or rigorous logical review;
- Constructions must satisfy every constraint, point by point;
- Improved bounds must be supported by a proof, not merely by a number;
- Optimized programs must pass functional tests, with actual performance measured.
As a result, what a model truly delivers is an object, parameter, or proof that can be independently reviewed—not a plausible-looking new answer. Because the verifier runs independently, it both reduces the error from subjective scoring and avoids the bias introduced by “one model judging another.”
2.3 Budget: Bringing “Trial-and-Error Cost” into the Evaluation
In real research, a researcher’s time, funding, and computing resources are all limited. The TCS benchmark therefore introduces budget control, with different attempt limits for different tasks:
- Some theorem tasks give the model only a single attempt, disallowing “trying a few more times to get lucky”;
- Some tasks allow multiple attempts, while recording both the success rate and the number of theorems covered.
This design is closer to real research: it evaluates “research output under a given budget,” rather than “the best score after unlimited trial and error.”
2.4 Recording Results: From a Total Score to a “Capability Profile”
The benchmark’s output is not just a leaderboard, but a set of fine-grained metrics:
- Overall score: aggregated by task type with weighting;
- Scaffold-free theorem completion: the success rate when no intermediate lemmas are provided, testing the ability to derive independently;
- Success rate across multiple attempts: the per-unit success rate under budget constraints;
- Coverage breadth: the number of distinct theorems solved, avoiding inflated scores from “repeatedly cracking the same problem”;
- Per-task pass status: reflecting the specific structure of capabilities.
Together, these metrics form the model’s “research capability profile.” For comparison, traditional math or code benchmarks are usually dominated by a single accuracy figure, whereas research benchmarks emphasize the combination of verifiability, budget constraints, and diversity.
3. Results: A “Capability Terrain Map,” Not a Ranking
On the TCS benchmark’s overall leaderboard, the top-ranked model (referred to below as Model A) scored 71.9, while the second-ranked model (Model B) scored 57.4, a gap of 14.5 points. The difference on theorem-proving tasks is even more intuitive:
- Model A: completed 14 of 20 theorems without intermediate lemmas, a success rate of 70.0%;
- Model B: succeeded 23 times across 60 attempts, covering 9 different theorems, a success rate of 38.3%;
- The other participating models: none of the remaining six models exceeded an 11.7% success rate on this task.
3.1 A Real Lead Exists, but It Is Not a Total Sweep
By overall score and theorem completion, Model A’s advantage is clear. But a closer look at per-task results reveals that Model B solved the alphabet-reliability problem for probabilistically checkable proofs (PCP) on all three attempts, while Model A failed to solve it.
This shows that a model’s research capability is not uniformly distributed. Some models are more stable at independently deriving long proofs, while others are more resilient on specific classes of construction problems. Looking only at the total score would obscure this structural difference.
3.2 Two Intuitive Results: Outputs That Can Be Independently Reviewed
The two most intuitive results in the TCS benchmark come from Model B and Model A, respectively:
- Extremal graph construction (Model B): In the “extremal graph with girth 5” task, the model constructed a graph with 124 vertices and 630 edges that contains no triangles and no quadrilaterals, surpassing the 629-edge reference value used by the task. The verifier only needs to check the edge count, the vertex count, and the two constraints of “no triangles, no quadrilaterals” to determine its validity.
- Tightening an upper bound on coding parameters (Model A): The model proved A(22,10) ≤ 82, tightening the previously published upper bound from 84 to 82. Here, A(n,d) denotes the maximum number of codewords achievable in a binary code of length n with minimum Hamming distance d.
These two kinds of results are valuable precisely because they are not “something that sounds reasonable,” but deterministic conclusions that others can independently review and reuse.
3.3 Three Points for Reading These Results
When interpreting results of this kind, three easily overlooked points deserve attention:
- A score gap does not equal a capability chasm. The models differ by 14.5 points overall, yet they split wins and losses on individual tasks, indicating a state of “overall lead” rather than “full coverage.”
- Success rates must be read together with the budget. Model A’s 70.0% was achieved under the strict condition of “one attempt, no intermediate lemmas,” while Model B’s 38.3% was achieved across 60 attempts. The comparison reflects a difference between “single-shot reliability” and “ultimate achievability after multiple attempts,” and does not directly determine who is stronger.
- Coverage breadth reflects research patience. Model B covered 9 different theorems across 60 attempts, showing that it did not fixate on individual problems but continued to obtain different results through repeated exploration. This is closer to the “broad exploration, gradual accumulation” pattern seen in real research.
4. From Benchmark to Practice: Building an AI-Assisted Research Workflow
The TCS benchmark offers more than a “ranking.” At its core, it demonstrates a reusable engineering path: define the problem → generate candidate results → verify automatically → select and improve. Enterprises can readily borrow this approach to embed large models into their R&D processes.
4.1 Step 1: Turn “Open Problems” into “Verifiable Tasks”
Take a typical task as an example. When using a large model to explore “finding better parameters or constructions,” you can organize the prompt along the following structure:
[Background] The best known result under the current constraints: parameter = X, sourced from published results.[Objective] Find a feasible solution better than X without violating any constraint.[Constraints]1. Must satisfy conditions C1, C2, and C3 (list each one);2. Must not rely on external conclusions that cannot be verified.[Output requirements]1. The specific parameters or explicit construction of the solution;2. A point-by-point explanation of how the solution satisfies each constraint;3. If a theoretical bound is given, include the proof.
The key to prompt design lies in three points:
- State the baseline clearly: spell out “what the best known result is” to avoid the model unknowingly repeating an existing conclusion;
- Make constraints checkable: each constraint should map to a piece of verification code;
- Require self-justification: force the model to output reviewable reasoning, not just a conclusion.
4.2 Step 2: Build an Automatic Verification Layer
The verification layer is the core of the entire workflow. Common verifier types include:
- Formal proof checkers: for mathematical propositions, confirming that every step of a proof follows the logical rules;
- Constraint-validation programs: for construction problems such as graphs, codes, and schedules, checking each constraint in turn;
- Functional tests and benchmarks: for code optimization, first confirming correctness, then measuring performance;
- Unit and integration tests: for screening candidate implementations in general software development.
The sample code below illustrates the “generate–verify–select” loop. The function names and interfaces are examples only:
def ai_assisted_search(problem, generate, verify, score,budget=100, threshold=0.95):"""Iterative solving framework with a large model as the generator and an automatic verifier as the judge.problem : task definition (includes explicit constraints and a known best baseline)generate : wrapper function that calls the large model to generate a candidate solutionverify : automatic verifier; returns True if the candidate fully satisfies the constraintsscore : quality score function for the candidate (e.g., magnitude of performance improvement)budget : maximum number of attempts, used to control costthreshold : target pass rate; stops the search once reached"""best = Nonepassed, attempt = 0, 0while attempt < budget and passed / max(attempt, 1) < threshold:attempt += 1candidate = generate(problem) # the model only "proposes"if not verify(candidate, problem): # the verifier "judges"continue # discard immediately if it fails; do not proceed furtherpassed += 1if best is None or score(candidate) > score(best):best = candidate # keep only verified solutions that are better# best can be fed back to generate as the new known baseline, forming iterative improvementreturn {"best": best, "attempts": attempt, "pass_rate": passed / attempt}
This code embodies an important principle: the model only “proposes,” and the verifier “judges.” Candidate results that have not passed automatic verification should never enter production directly; this significantly reduces the risk of “looks correct but is actually unusable.”
4.3 Step 3: Constrain the Budget and Quantify Input Versus Output
Following the TCS benchmark’s example, enterprises should set the following in advance:
- Call budget: the number of calls, token consumption, and cost cap per task;
- Time budget: the maximum runtime for a single task;
- Pass-rate threshold: for example, requiring the verification pass rate to be no lower than a given threshold before a model’s result may enter the formal process.
At the same time, each run should record metrics such as the number of attempts, the number of passes, and the number of distinct problems covered. This data is the basis for continuously assessing a model’s value, optimizing prompt strategies, and controlling costs.
4.4 Step 4: Build Evaluation and R&D Pipelines on Cloud Platforms
In a general public cloud environment, this kind of workflow can be split into layers by responsibility:
- Task generation layer: the large model service handles the generation of candidate solutions;
- Verification execution layer: a container platform runs the verification programs, providing isolated, elastically scalable computing resources;
- Data management layer: object storage holds the task definitions, candidate results, and verification logs, ensuring full traceability;
- Message decoupling layer: a message queue handles the asynchronous tasks between generation and verification, supporting large-scale concurrency;
- Observability layer: logging and monitoring with alerting uniformly track call volume, pass rates, latency, and cost.
For example, on Baidu AI Cloud, enterprises can compose the capabilities above on demand from general cloud product categories to build their own “AI-assisted research” and “automated evaluation” pipelines. Whatever large model service is used, the verification layer should remain independent and standardized, avoiding tight coupling to any specific model implementation.
4.5 Step 5: Define the Human-Machine Division of Labor
The TCS benchmark also reveals a practical reality: models perform best under the conditions of “clear problem, sufficient budget, and verifiable feedback.” The sensible division of labor is therefore:
| Stage | Better suited to | Reason |
|---|---|---|
| Proposing research directions, judging a problem’s value | Human | Requires domain insight and strategic judgment |
| Large-scale attempts and construction search | Model | Can run in parallel, low cost, unaffected by fatigue |
| Result verification and quality control | Automatic verifier + human | Objective rules are executed by programs; edge cases are reviewed by people |
| Interpreting results and external communication | Human | Requires judging significance and integrating context |
5. Capability Boundaries and Risks
While recognizing the progress, it is equally important to stay clear-eyed about the current boundaries of a model’s research capability.
- Autonomously choosing a research direction remains difficult. Models excel at finding better solutions given a target, but struggle to judge “what problem is worth studying.” In the TCS benchmark, all tasks are defined by the benchmark authors; the model’s role is closer to that of an “executor” than a “problem poser.”
- Judging the value of a result depends on humans. Whether tightening an upper bound from 84 to 82 is significant enough, or whether a construction is worth publishing, ultimately still requires assessment by domain experts.
- Major open problems remain unsolved. Current results are concentrated at the level of “improving known results” and are still a clear distance away from resolving long-standing major problems.
- Verification costs cannot be ignored. Formal proofs and constraint validation both require additional engineering investment, and verifying some frontier research results is itself a hard problem.
- Single-domain conclusions cannot be generalized. TCS tasks are highly verifiable, so the benchmark’s results cannot be simply extended to experimental sciences, the humanities and social sciences, or other fields with higher verification costs.
6. Conclusion
Returning to the original question: if large models can solve competition problems, does that mean they can do research?
Based on the TCS benchmark’s results, the answer is closer to this: models can already complete some proof derivations, parameter tightening, and construction improvements under conditions of a clear problem, a sufficient budget, and verifiable feedback, producing results that can be independently reviewed. But they still mainly play the role of an “efficient research assistant” rather than an “autonomous researcher.”
This benchmark is more like a terrain map of AI research capability: it marks both the high ground that models can reliably reach today and the uncharted areas that still require human exploration. For enterprises, rather than asking “can AI replace researchers,” the better question is how to place AI into a workflow of “clear problem, verifiable result, controllable budget,” letting large models leverage their scale advantages in verifiable stages while people focus on choosing directions and judging value. This may well be the most pragmatic path for human-machine collaboration in research at the current stage.
