LLM Research Capability Evaluation: How to Build Verifiable Benchmarks
Benchmark scores for math, code, and logic keep climbing, yet none of them tell us whether a model can actually do research. This article explains why solving ability and research ability are different things, why grading—not problem selection—is the real bottleneck in research evaluation, and why theoretically verifiable fields such as Theoretical Computer Science (TCS) make an ideal testbed. It then describes a four-level task design (from core knowledge to open problems), outlines an automated evaluation flow with strict separation between generation and verification, and closes with practical takeaways for developers and technical decision-makers who are introducing LLMs into R&D workflows.
1. Scores Keep Rising, Yet We Still Don’t Know If Models Can “Do Research”
Over the past two years, LLM capability benchmarks have been refreshed every month or two. Math, code, and logical reasoning leaderboards trade places constantly, and the numbers get prettier each round. But the moment a company actually wants to put a model into its R&D process and hand it work that requires “zero to one” creativity, an awkward question surfaces: what do those high scores really stand for?
The honest answer: they mostly measure problem-solving ability, not research ability.
That distinction matters. Solving a problem is like cooking from a recipe—the steps are given, the ingredients are specified, and there is a widely accepted standard for what the finished dish should taste like. You will almost never see a grading rubric that accepts chocolate added to a guō bāo ròu as a valid dish. Research is the opposite: no prescribed steps, no guarantee that any given path leads anywhere, and even the person posing the question may not know the final answer. Judging whether a piece of research succeeded does not depend on comparing it against a reference answer, but on whether it genuinely pushed the boundary of what is known.
Using solving-oriented leaderboards to infer a model’s research level is therefore a systematic misjudgment. To make evaluation reflect real research, we need a different methodology.
2. The Real Bottleneck in Research Evaluation: Not Writing Problems, but Grading Them
Many people assume the hard part of evaluation is making problems difficult enough and novel enough. It isn’t. The real bottleneck is grading.
Picture this: a model is given an open conjecture that has stood unresolved for decades. It outputs hundreds of pages of dense derivations and finally claims, “the conjecture is proven.” The reviewer now faces a nearly intractable problem:
- Who checks hundreds of pages line by line? The labor cost is enormous, and it requires top-tier experts;
- Even an expert may not be able to judge validity within a reasonable time;
- A single error can hide somewhere inconspicuous on page two hundred, and the conclusion collapses at the very first step.
This is the fundamental difficulty of open-ended research evaluation: whether a conclusion is correct is often hard to verify automatically, cheaply, and reliably. When the output of a field cannot be efficiently re-checked by machines, evaluation degrades into “subjective expert scoring”—expensive, hard to scale, and far from objective or reproducible.
So the first step in building a credible research benchmark is not picking the hardest problems. It is answering one question: in which fields can conclusions be verified by machines as a matter of nature?
3. Why Formally Verifiable Fields Are the Ideal Testbed
This leads to a key insight: not every discipline is suitable for research evaluation, but there is a special class of fields where the correctness of a conclusion can be checked automatically by a program.
The judgment logic resembles Sudoku: filling it in is hard, but checking right and wrong takes one look. Solving a complex logic puzzle may take you hours, yet once it is filled, a verifier needs only seconds to confirm whether every row and column is valid. The difficulty of producing an answer and the difficulty of verifying it are highly asymmetric—exactly the property a good benchmark needs.
Among all disciplines, Theoretical Computer Science (TCS) is an almost perfect candidate. Sitting at the intersection of mathematics and computer science, its core subject has always been computation itself: which problems are solvable, how fast a problem can at best be solved, and which problems are simply “doomed to fail.” It cares less about how to fix a machine or how to write production code, and more about the limits and the essence of computational power.
Three reasons make this a strong choice:
- Research is mainly done through proofs. Proofs can be formalized, and every inference step can be checked individually by a verifier;
- Conclusions are machine-decidable. A proof either holds or it does not, with almost no gray area, unlike open-ended writing where reasonable people differ;
- It covers real research work. It can test foundational knowledge, creative proof construction, and genuine open problems.
Put differently, if a model performs poorly in this field, we have no “the scoring was subjective” excuse. The machine decides whether the conclusion is right.
4. Designing an Exam That Feels Like Real Research
Once the field is chosen, the next question is how to decompose “research” into a set of tasks that is evaluable, tiered, and verifiable. A practical approach is to let problems progress from “foundational” to “frontier,” covering the full spectrum of a researcher’s daily work. A reasonable layering looks like this:
- Level 1: Concepts and foundational knowledge. Tests whether the model has command of definitions, theorems, and basic tools in the field. This is like a graduate qualifying exam—the ground everything else is built on.
- Level 2: Solving standard problems. Problems with well-defined conditions and a unique answer, testing the model’s ability to reason rigorously and combine knowledge correctly.
- Level 3: Constructing and writing proofs. The model must not only produce an answer but also a rigorous proof that a verifier can check line by line, testing rigor and completeness.
- Level 4: Exploring open problems. Facing conjectures that remain unresolved today, testing whether the model can propose genuinely new directions—even partial progress counts.
This layering has two benefits. First, it pinpoints where a model’s capability actually sits: has it memorized knowledge, can it solve problems properly, or can it construct proofs? Second, judgment at every level can be automated, with no reliance on expensive human review.
As for scale, a set of a hundred or more problems spanning multiple research directions usually reflects the true distribution of model capability better than scattered small samples, and is less susceptible to chance or data contamination.
5. What a Reproducible Evaluation Framework Looks Like
To put the ideas above into practice, the evaluation system itself should be reproducible and automated. The following illustrative structure shows the core flow and how “machine grading” is organized in engineering terms:
# Illustration: automated flow of a verifiable research benchmark (pseudocode)from dataclasses import dataclassfrom enum import Enumclass Level(Enum):KNOWLEDGE = 1 # Concepts and foundationsSOLVING = 2 # Solving standard problemsPROOF = 3 # Proof construction and verificationOPEN = 4 # Exploring open problems@dataclassclass Task:task_id: strlevel: Levelprompt: str # Problem statementverifier: object | None # Automated verifier for the conclusion/proofdef run_benchmark(model, tasks):report = {lv: {"total": 0, "passed": 0} for lv in Level}for task in tasks:report[task.level]["total"] += 1answer = model.generate(task.prompt)# Key: verification is decoupled from generation and handed to# an independent, trusted verifierif task.verifier is not None and task.verifier.check(answer):report[task.level]["passed"] += 1# Open-ended tasks without an automated verifier are flagged# separately in the manual review queuereturn report
A few design principles in this structure are worth emphasizing:
- Generation and verification are strictly separated. The model only produces output; verification is done by an independent, trusted, deterministic verifier. This avoids the circular reasoning of a model judging itself and keeps results reproducible.
- The verifier is the credibility of the benchmark. The verification logic itself must be correct, stable, and auditable. Once a verifier is wrong, the credibility of the whole leaderboard collapses—which is why verifiers need more rigorous testing than the problems themselves.
- Open problems are handled separately. For tasks that genuinely cannot be verified automatically, one should not force a “pass/fail” verdict. They should enter a manual review queue and be clearly marked in the report, rather than dressing up uncertainty as certainty.
6. The Core Conclusion: The Value of a Cold Shower
When such a verifiable benchmark actually runs, the results tend to be sober and pragmatic. One observation recurs: when facing open conjectures that remain unresolved to this day, nearly all participating models fail to produce a genuinely effective solution on their own. This is not surprising—it is extremely valuable.
Its value lies in clarifying the boundary between capability and performance. Many of today’s models are strong at retrieving, combining, and reasoning within an existing knowledge framework; genuinely original research—posing entirely new questions, opening paths no one has walked—remains territory where human intelligence is hard to replace. Acknowledging this does not deny the progress models have made. It gives us a more accurate expectation of what a model can do now and what it still cannot.
For developers and technical decision-makers, this yields a few practical implications:
- Choose models based on “performance on verifiable tasks.” Where conclusions can be machine-checked, capability differences are immediately clear, with no need to rely on vendor-published scores.
- Treat verification capability as a first-class concern. If your business allows it, prioritize tasks that are “easy to generate, hard to verify” as evaluation and deployment scenarios—they expose a model’s real weaknesses most clearly.
- Human-machine collaboration, not wholesale replacement. In open research settings, treating the model as an “idea amplifier” that proposes candidate directions while leaving the final judgment to human experts is currently the more robust engineering pattern.
7. Final Words
Evaluating an LLM’s research capability is, at heart, answering a simple but difficult question: is it a student who got better at taking exams, or a collaborator who can do research?
The answer does not lie in piling up more and harder problems. It lies in choosing fields where conclusions can be verified efficiently by machines, and using an engineering structure that fully decouples generation from verification to approximate the real shape of research. Use verifiable disciplines as the testbed, cover the full spectrum of research work with layered tasks, and guard the credibility of the results with rigorous verifiers—this methodology helps us escape the trap of “scores keep climbing while judgment gets blurrier,” and brings every evaluation closer to the truth. For teams introducing LLMs into their R&D process, that may matter more than any single score.
