Evaluating the Research Capabilities of Large Language Models: Building a Verifiable Assessment Framework

    This article argues that a competition-style accuracy score cannot measure whether a large language model (LLM) can actually do research. It explains why verifiability should be the foundation of any research-capability evaluation, showing how to decouple generation from a model-independent verifier and why the number of attempts is itself a metric. It then abstracts research output into verifiable objects—proofs, programs, and mathematical constructions—proposes a multi-dimensional metric system, and offers guidance on interpreting results correctly. Finally, it outlines the current capability boundaries of LLMs and gives practical steps for developers building their own evaluation systems.

    1. First, a Premise: Solving Problems Is Not the Same as Doing Research

    Before asking whether a large language model (LLM) can do research, we need to clear up a common misconception: treating the accuracy rate on competition problems as if it were a measure of research ability.

    Competition problems and real research tasks differ fundamentally in structure. Competition problems are typically well-defined, have a closed goal, and come with a single ground-truth answer; the model only needs to complete a derivation within fixed rules. Research tasks, by contrast, involve multiple stages—problem formulation, method selection, solution validation, and value judgment—and even defining “what counts as a good answer” itself requires human judgment. For this reason, using a model’s scores on math or coding competitions to infer whether it “can do research” is logically unsound.

    A meaningful evaluation should not ask “how many problems did the model get right?” Instead, it should ask: given a clearly defined problem, a sufficient sampling budget, and a verifiable feedback mechanism, can the model produce independent, reviewable research results? This shifts the focus of evaluation from “generating plausible-looking answers” to “generating objects that can be verified”—which is the first principle for building a rigorous evaluation system.

    2. Verifiability: The Cornerstone of the Entire Evaluation System

    Many evaluations are distorted because they rely on “matching against reference answers” or “subjective human scoring”—the former cannot cover open-ended output, while the latter is costly and hard to scale. A sustainable, scalable evaluation of research ability must be designed around verifiability as a core constraint.

    The key to verifiability is that the verification logic must be decoupled from the object being tested. The model is responsible for generating candidate results, while a verifier that is independent of the model decides whether the result holds. This brings three benefits:

    1. Objectivity: the verification conclusion does not depend on the scorer’s subjective preferences;
    2. Scalability: the verifier can run automatically in batches, supporting high-budget evaluations;
    3. Resistance to “plausible fakes”: what the model produces must be an object that passes strict checks, not a piece of text that merely sounds convincing.

    The following sample code shows the skeleton of a “decoupled generation and verification” evaluation:

    1. from dataclasses import dataclass
    2. from typing import Callable, Any, Tuple, Optional
    3. @dataclass
    4. class VerificationTask:
    5. problem: str # A clear, closed-form problem definition
    6. budget: int # Sampling / attempt budget
    7. artifact_type: str # Artifact type: proof / program / construction
    8. verifier: Callable[[Any], bool] # Model-independent verification function
    9. def evaluate(model, task: VerificationTask) -> Tuple[Optional[Any], int]:
    10. # Key: the verifier is fully decoupled from the model under test and only accepts reviewable results
    11. for attempt in range(1, task.budget + 1):
    12. artifact = model.solve(task.problem)
    13. if task.verifier(artifact): # Strict check, not "looks reasonable"
    14. return artifact, attempt # Return the result and the attempts needed for the first success
    15. return None, task.budget # Budget exhausted with no verifiable result produced

    This skeleton embeds an important design intent: the number of attempts (the sampling budget) is itself a metric. Given two models that both succeed, one completing the task within a single attempt and the other requiring dozens of samples to hit, their research efficiency is not the same.

    3. Abstracting Research Output into “Verifiable Objects”

    To make the evaluation cover the diversity of research, we need to abstract different types of research output into a unified notion of a “verifiable object,” and configure a corresponding verifier for each type. There are three common categories of output.

    3.1 Theorems and Proofs

    For a mathematical theorem, an acceptable output is not a passage of “reasoning that looks correct,” but a proof that can be formally checked. Ideally, the proof should be verifiable step by step in a proof assistant, or checked automatically in settings that have a strict decision procedure. During evaluation, one can further distinguish “whether intermediate lemmas are provided”—because the ability to autonomously break down key lemmas often reflects deeper reasoning than applying ready-made steps. Whether intermediate steps are given should be treated as a quantifiable observation dimension, not an implicit assumption.

    3.2 Programs and Optimization

    In algorithmic or code-optimization tasks, the verifiability of the output is especially clear: whether a piece of code is correct can be determined with test cases + complexity constraints + benchmarks. The evaluation looks not only at “can it pass,” but also at whether the post-optimization resource consumption (time, space, number of calls) has genuinely improved. Here the verifier is essentially a combination of “compiler + test suite + performance benchmark,” which offers very high objectivity.

    3.3 Mathematical Constructions

    “Construction” tasks in combinatorial optimization and extremal problems produce a concrete mathematical object—for example, a graph that satisfies specific constraints, a set of parameter values, or a new upper/lower bound. The value of such results lies in being an independently reviewable instance: the verifier only needs to check “whether the construction satisfies all constraints” and “whether its size/numeric value is better than the known reference value” to give a definitive conclusion. The “reference value” here should come from public, citable existing results to ensure a fair comparison.

    4. Evaluation Dimensions and Metric Design

    Once “verifiable objects” are unified, we need a multi-dimensional metric system that reflects research ability. A single aggregate score easily masks differences in the structure of ability. It is recommended to expand across at least the following dimensions:

    • Success rate: the proportion of tasks that produce verifiable results within the given budget;
    • Coverage: how many different problems across the task set the successful results cover (to avoid gaming the score on the same type of problem);
    • Efficiency: the average number of attempts needed for the first success, reflecting the efficiency of the “sample–verify” loop;
    • Capability profile: statistics computed separately by task type (proof / optimization / construction), rather than giving only a single aggregate score;
    • Difficulty stratification: distinguishing “guided” from “unguided” tasks, to observe how the model performs when it lacks intermediate scaffolding.

    It is worth stressing: a leaderboard ranking is not the endpoint—the capability profile is. Two models with similar aggregate scores may differ in that one excels at construction and the other at proofs; a model losing on an individual task does not mean it is comprehensively behind. The value of evaluation lies in revealing structure, not in ranking.

    5. How to Correctly Interpret Evaluation Results

    A common misreading is: one model ranks first on the aggregate score, so people conclude it “dominates across the board.” A proper interpretation should follow several principles:

    1. Value both individual items and the aggregate score: a model may have a unique advantage on a very small number of hard problems (for example, completing a particularly difficult construction, or performing well on a specific reliability task), while the leader may not have the edge on these points. Such differences are precisely the most informative signals.
    2. Distinguish “unsolved” from “done wrong”: an unverifiable output and an erroneous output are two different things; the former indicates a lack of ability, while the latter may simply be a sample that missed. Evaluation reports should count the two separately.
    3. Focus on “verifiability” rather than “fluency of expression”: being able to give an independently verifiable object (a proof, parameters, a construction) is more valuable than a fluent but uncheckable narrative.
    4. Acknowledge randomness: a success rate under large-budget, repeated sampling reflects real ability better than a single run; a single result has large variance and should not be treated as definitive.

    6. Capability Boundaries: What AI Can Do Now, and What Is Still Missing

    Overall, when the three conditions of a clear problem + sufficient budget + verifiable feedback are all in place, LLMs can already take on substantive work in several areas: completing partial proofs, performing program optimization, and producing improved mathematical constructions. These all fall under “solving and optimizing given a defined goal.”

    But three types of ability remain beyond the reach of current evaluations and are still mainly borne by humans:

    • Autonomously choosing a research direction: judging “what problem is worth studying”;
    • Assessing the value of results: judging whether a result is important or meaningful;
    • Tackling major open problems: long-term exploration in the absence of a clear path and verification scaffolding.

    Therefore, a more fitting positioning is this: AI can already provide verifiable contributions at the “solving” level, but at the levels of “defining the problem” and “judging value,” humans remain indispensable. This also points to the next stage of evaluation design—not only evaluating “how well it solves,” but also exploring how to evaluate “whether it chose correctly.”

    7. Practical Advice for Developers

    If you want to build a similar evaluation for research/reasoning ability in your own setting, you can refer to the following implementation points:

    1. Build the verifier first, then choose the model: the verifier is the foundation of the whole system; prioritize its correctness, independence, and level of automation;
    2. Enforce the “generation–verification” decoupling: never let the object under test be both the athlete and the referee;
    3. Record the complete process: save the number of attempts, reasons for failure, and successful paths, so that ability can be attributed;
    4. Use public reference values as comparison anchors: so that conclusions like “amount of improvement” are grounded;
    5. Report a capability profile rather than a single score: present it by dimension, by difficulty, and by task type to avoid misreading;
    6. Quantify the uncertainty: give confidence intervals or the distribution over multiple runs, rather than reporting only a point estimate.

    Putting these principles into practice turns an evaluation system from a “leaderboard tool” into a “dashboard for understanding the boundaries of model capability”—which is the correct way to answer the question of whether large models can do research.