LLM Research Capability Evaluation: How to Build Verifiable Benchmarks
LLM benchmark scores keep climbing, but a strong result on competition problems, coding challenges, or math puzzles does not necessarily mean a model can do original research. This article examines how to evaluate whether large language models possess genuine research capability rather than just problem-solving ability. It explains why theoretical computer science (TCS) works as a natural verifiable testbed, how to build a verifier-based evaluation that aligns tasks with deterministic checks, how to layer tasks by research activity type, how to prevent answer leakage and memorization, and how to design metrics that go beyond a single accuracy number. It closes with key findings, capability boundaries, and a practical selection checklist for enterprises and R&D teams.
1. Scores Keep Rising: What Are LLM Research Benchmarks Actually Measuring?
Discussion around evaluating large language models’ research capability has clearly heated up over the past two years. Many teams ask the same questions: What is verifiable evaluation? Why does being able to solve problems not equal being able to do research? If a model can score high on competition problems, coding tasks, and math problems, does that mean it can do theoretical research? And how do you evaluate open research problems without being misled by answers that merely “look reasonable”?
Most traditional leaderboards are “problem-solving tests with a given question”: the prompt is clear, the answer is known, and the scoring rule is fixed. Strong model performance means the model is good at retrieval, composition, and inference within an established paradigm. But real research is different. Research often has no fixed procedure, the process is open-ended, and the outcome is unpredictable in advance — even the person posing the question may not know the final answer. Solving problems emphasizes matching a standard answer, whereas research demands proposing new questions, designing proofs, arguing about boundaries, and bearing the cost of verification. So the real question is: how do you evaluate whether a large model has original research capability, rather than only whether it can solve existing problems?
2. Drivers vs. Road Builders: Why Theoretical Computer Science Makes a Good Research Yardstick
An analogy helps here. An ordinary programmer is like a skilled driver: familiar with how the car works, able to drive fast and smooth. A theoretical computer science (TCS) researcher is more like a transportation-system planner, who cares about questions such as: What is the theoretically fastest way to get from point A to point B? Is there a route that simply cannot be traversed? Are some problems fundamentally impossible to solve efficiently?
Theoretical computer science sits at the intersection of mathematics and computer science, and its core object is “computation.” Its research output is usually not a product but proofs, constructions, lower bounds, counterexamples, and formal arguments. A TCS task list suited to testing research capability typically includes:
- Algorithm design and analysis: Propose a new algorithm and prove its correctness, complexity, or conditions of applicability.
- Characterizing complexity lower bounds: Argue that a problem cannot fall below some boundary of computational resources.
- Constructive proofs: Construct objects, strategies, or data structures that satisfy specific constraints.
- Counterexamples or impossibility arguments: Show why a class of methods is infeasible, or why a certain goal cannot be reached.
- Formalization of proofs: Convert informal reasoning into machine-checkable formal proofs.
These tasks are closer to the research setting than ordinary engineering problems: they require not only an answer, but also an explanation of why the answer holds and where the boundaries lie.
3. Proofs Can Be Machine-Verified, but Open Conjectures Are Hard to Grade
A major advantage of mathematics and TCS is that the correctness of a conclusion can be checked automatically by a machine. It is like Sudoku: filling the board is hard, but once it is filled, a program can quickly judge whether it is correct. Even a very complex proof, if properly formalized, can be handed to a verification tool and checked line by line.
Open conjectures are another matter. For problems with a definite known answer, verification is relatively easy; for genuinely unresolved public problems, there is no ready answer to compare against and no widely accepted verification path. If a model claims to have solved a long-standing hard problem, outside observers cannot just judge that it “sounds right” — they need to see a complete proof, formalization evidence, or verification material that experts can independently re-check. Otherwise, the evaluation risks mistaking “fluent expression” for “research capability.”
4. Theoretical Computer Science as a Natural Verifiable Testbed
The value of TCS lies in combining two features: it is research-intensive, and its conclusions are verifiable. Algorithmic complexity, decidability, constructive proofs, and lower-bound arguments can all be formalized and mechanically checked by a machine. This makes it a rare and reliable testbed for evaluating LLM research capability.
It can both force a model to confront open-ended problems and turn the question “does the answer hold?” from a subjective judgment into an executable verification. In other words, TCS turns “research capability” from an abstract slogan into observable, reproducible, and traceable engineering metrics.
5. How to Build a Verifiable LLM Research Benchmark
5.1 Align Task Selection with the Verifier
Every problem should come with an automatically executable verifier. The verifier must not be merely a “reference-answer comparator”; it should be a deterministic decision tool. Before presenting any code illustration, the verifier contract should be made explicit:
- Input: Problem statement, problem spec, formal standard, input/output constraints, resource limits, allowed dependencies.
- Output: Whether it passed, failure reason, proof or trace, elapsed time, resource usage, logs.
- Failure types: Format errors, logic errors, invalid proof, timeout, memory limit exceeded, verifier undecidable, dependency version conflict.
- Dependency notes: The theorem library, compiler, solver, and language version used, along with their version numbers.
- Execution environment: Timeout, memory ceiling, random seed, filesystem permissions, network isolation.
The following is only a flow illustration, not a complete runnable system:
# Illustration: verifier-based evaluation pipeline (pseudocode)def evaluate(model, problem):# The model only sees the statement and spec; no known answer is providedsolution = model.generate(statement=problem.statement,spec=problem.spec)# Key: a deterministic verifier produces an objective verdict,# rather than another model's subjective scorepassed, proof_trace = verifier.check(solution=solution,spec=problem.spec,timeout=problem.timeout,memory_limit=problem.memory_limit)return Result(problem_id = problem.id,passed = passed,proof = proof_trace,failure_reason = solution.failure_reason,tokens = solution.token_count,latency_ms = solution.latency,verifier_version = verifier.version)
5.2 Layer by Research Activity Type
The evaluation set should not lump everything together as a pile of “hard problems”; it should be layered by task type. Different task types expose different model weaknesses.
| Task type | Verification method | Common failure modes | Suggested metrics |
|---|---|---|---|
| Algorithm design and analysis | Formal proof, correctness check, complexity-derivation review | Fake proofs, missed edge cases, overestimated complexity | pass@k, proof validity rate |
| Complexity lower bounds | Formal argument or counterexample construction | Logical leaps, unproven key lemmas | lower-bound validity rate, invalid step count |
| Constructive proofs | Automatically construct objects and check constraints | Non-unique constructions, parameter boundary errors | construction success rate, constraint satisfaction rate |
| Counterexamples or impossibility arguments | Automated counterexample search, formal checking | Invalid counterexamples, overly broad scope | counterexample validity rate, misjudgment rate |
| Formalization conversion | Formal tool checking | Syntax errors, inconsistent definitions | conversion success rate, manual repair cost |
This layering reveals structural differences: some models are good at construction but poor at arguing impossibility; some can generalize but lack rigorous reasoning.
5.3 Prevent Answer Leakage and Memorization
Research benchmarks must guard especially against models “memorizing problems.” Executable practices include:
- Deduplication search: Compare candidate problems against public papers, competition problems, problem solutions, code repositories, and Q&A communities.
- Similarity thresholds: Use 8-gram or 13-gram overlap rates and embedding similarity as an initial filter. For example, if continuous 13-gram overlap exceeds 5%, or vector similarity is above 0.85, send the item to manual review.
- Parameterized generation: Generate different scales, parameters, constraints, and random seeds for the same class of problem, so the model cannot simply recall a fixed answer.
- Publication cutoff: Set a cutoff date for problem publication, and prioritize recent or rewritten problems.
- Private problem sets and random sampling: Keep some problems unpublished and rotate them periodically to reduce the risk of contamination by training data.
- Rewriting and transformation: Change variable names, input order, constraints, and proof targets while keeping the mathematical structure consistent.
5.4 Go Beyond Accuracy in Metric Design
A single accuracy figure can easily hide problems. A more complete set of metrics should include:
- pass@1 and pass@k: Measure the success rate of a single attempt and across multiple samplings.
- Proof step validity rate: The proportion of valid steps out of total reasoning steps.
- Difficulty-tiered pass rate: Report separately for four tiers — easy, medium, hard, and open.
- Multi-sampling consistency: Whether results are stable across repeated generations of the same problem.
- Invalid proof rate: The proportion of proofs that look reasonable but fail verification.
- Timeout rate and resource consumption: Reflect whether complex reasoning becomes unaffordable.
- Error-type distribution: Distinguish format errors, logic errors, citation errors, and resource errors.
These metrics say more about whether a model approaches research capability than a single sweeping percentage.
6. Key Findings, Capability Boundaries, and Enterprise Selection Guidance
The conclusions in this article are based on a synthesis of public leaderboards and verifiable-evaluation practice; they do not list specific models, datasets, or scoring rules. The scope of observation is limited to publicly available theoretical computer science problems, discussions of open conjectures, and enterprise evaluation cases from recent years. The sample is drawn from public evaluation practice and does not cover unpublished models, private problem sets, or all time slices. The following judgments should therefore be understood as an engineering-level summary, not absolute claims about every model.
Within the scope of this verifiable evaluation, no model was observed to independently crack the open conjecture. However, frontier models can already perform reasonably high-quality technical reasoning on well-designed closed-form TCS problems. The capability boundary can be summarized as: models have strong reasoning ability within a clear paradigm, but still fall noticeably short at independently discovering new problems, constructing original proofs, and handling long-standing open problems.
For enterprises and R&D teams, selection can proceed along this checklist:
- Target capability: First clarify whether you are testing engineering efficiency, problem-solving ability, or deep research-level reasoning.
- Evaluation type: If the goal is research capability, prioritize verifiable problems that stay close to research structure, rather than relying only on general leaderboards.
- Verifier requirements: There must be deterministic verification, failure-reason recording, version traceability, and log tracking.
- Risk awareness: Watch for answer leakage, subjective scoring, misjudgment of open problems, and misleading single-metric accuracy.
- Result interpretation: For conclusions on open problems, require the model to provide evidence that can be independently verified by a machine or an expert.
Common mistakes to avoid: relying on accuracy alone, depending on another model’s subjective scoring, ignoring answer leakage, failing to distinguish open problems from closed problems, not recording failure reasons, and not treating the verifier as evaluation infrastructure.
7. Conclusion
The value of verifiable evaluation is not to rank models one more time, but to turn the vague concept of “research capability” into a testable, reproducible, and traceable engineering proposition. Only when the evaluation itself stands up to scrutiny can we truly see the capability boundary of large models’ research ability — what has already been crossed, and what is still on the eve of a breakthrough.
