Evaluating LLM Research Capabilities: From Competition Problems to Verifiable Workflows

    Large language models (LLMs) increasingly achieve high scores on math contests, coding competitions, and standardized exams, which can create the intuition that they are ready for research. This article argues that contest performance and research capability are fundamentally different, and it proposes a layered framework for evaluating LLM research capabilities through verifiable workflows. It covers task design, sandboxed environments, independent verifiers, metrics, engineering practices, failure modes, and the boundary between human and model work. Readers get a practical way to assess where models can produce independently checkable research artifacts and where human judgment remains essential.

    Large language models continue to achieve high scores in math competitions, coding contests, and standardized exams. This naturally leads to an intuition: if a model can solve difficult problems, does it already have research capability? The answer is not simple. Competition problems are usually closed problems with clear inputs, unique answers, and well-defined evaluation criteria. Research, by contrast, deals with open problems. It requires choosing directions, defining constraints, decomposing lemmas, designing experiments, constructing counterexamples, and judging whether a result has scientific value. Equating “can solve contest problems” with “can do research” significantly overestimates model capability and can mislead engineering teams in designing evaluation metrics.

    A more reasonable approach is to turn key outputs in the research workflow into objects that can be independently reviewed: proofs, constructions, programs, parameters, counterexamples, or experiment records. Then, under fixed budgets and verifiable feedback, we can observe whether a model can complete parts of the research process. This article focuses on “verifiable research workflow evaluation” and discusses the evaluation framework, task spectrum, metric design, engineering implementation, failure modes, and the boundary of human-model collaboration.

    1. Fundamental Differences Between Competition Problems and Research Tasks

    The first difference between competition problems and research tasks is the objective function. Competition problems usually require a definite answer within a limited time, and evaluation focuses on correctness and speed. Research tasks require testable candidates in an uncertain space, and evaluation focuses on novelty, reproducibility, improvement magnitude, and problem value. A model can complete some proofs and constructions when the problem is clear, the budget is sufficient, and feedback is verifiable. However, “choosing which problem is worth working on” still depends heavily on human judgment.

    The second difference is the success criterion. Competition problems often have standard answers, while research outputs can be multi-solution. A proof can use different lemma paths, a construction can make different parameter trade-offs, and a program optimization can pursue different performance goals under correctness constraints. Therefore, evaluation cannot only check whether the final string matches. It should check whether the output satisfies formal constraints, whether it can be independently verified, and whether it relies on undeclared intermediate assumptions.

    The third difference is verification cost. Competition problems can be verified by answer comparison, while research outputs require toolchains such as proof checkers, construction validators, program property tests, and numerical boundary rechecks. Verifiers themselves can have vulnerabilities, so the evaluation system must isolate generation from verification and prevent models from “proving themselves correct.”

    The fourth difference is value judgment. Even if a model produces a verifiable new construction or new upper bound, humans still need to judge whether the result matters, whether it duplicates existing work, and whether it deserves further investment. Verifiability addresses “is it correct,” but it does not address “is it worth doing.”

    2. A Layered Framework for Verifiable Research Workflow Evaluation

    A practical evaluation framework can be divided into four layers: task layer, environment layer, verification layer, and analysis layer.

    The task layer breaks the research workflow into decidable subtasks. For example, a theorem-proving task requires producing a proof under given axioms and definitions. It can have two modes: “intermediate lemmas allowed” and “intermediate lemmas not provided,” to observe the model’s ability to discover lemmas autonomously. A construction-improvement task requires producing combinatorial objects under constraints and comparing them with reference constructions. A program-optimization task requires improving performance while satisfying correctness properties. A reliability-verification task requires checking boundary conditions such as alphabet reliability in probabilistically checkable proofs. A counterexample-search task requires finding instances that violate a conjecture.

    The environment layer provides sandboxes, resource quotas, attempt limits, and dependency management. Fixed budgets are a prerequisite for fair evaluation. If attempts are unlimited, a system may hide insufficient reasoning ability through massive search. If the budget is too tight, the evaluation may underestimate the model’s potential on long-horizon tasks. Therefore, the budget should be recorded and reported as part of the evaluation configuration.

    The verification layer consists of independent verifiers. Proof tasks use proof checkers; construction tasks use constraint validators; program tasks use hidden tests and property checks; numerical tasks use interval rechecking. Verifiers should be isolated from the generation environment, use structured records for inputs and outputs, and preserve an evidence chain. An illustrative pseudocode example is shown below:

    1. verify(task, artifact, budget):
    2. if not schema_check(artifact): return fail("invalid format")
    3. if exceeds_budget(budget): return fail("budget exceeded")
    4. evidence = independent_check(task, artifact)
    5. if evidence.valid: return pass(evidence)
    6. return fail(evidence.reason)

    The analysis layer records attempt traces, successful tasks, covered objects, failure types, and reproducibility. Only by recording both the “final answer” and the “process evidence” can we distinguish accidental success from stable capability.

    3. Task Spectrum and Capability Dimensions

    Research capability is not a single score but a combination of multiple capabilities. At minimum, evaluation should observe the following dimensions: formalization level, openness, verification cost, search depth, cross-step dependency, and budget efficiency. The higher the formalization level, the easier verification becomes, but the task may drift away from real research. The higher the openness, the closer the task is to real research, but the harder evaluation becomes. Therefore, an evaluation set should include gradient tasks from semi-formal to formal.

    In proof-oriented tasks, whether intermediate lemmas are provided is critical. When lemmas are provided, the task is closer to “proof completion.” When lemmas are not provided, the model must propose usable intermediate conclusions autonomously, which is significantly harder. Construction-oriented tasks examine the model’s ability to search a constraint space and improve reference values, for example, constructing graphs with girth constraints that avoid certain short cycles, or tightening parameter bounds in coding theory. Program-optimization tasks require the model to improve performance under correctness constraints, and the verifier must check both output correctness and whether the optimization objective is genuinely improved.

    A common phenomenon in public evaluations is that the system with the best overall success rate may not lead on every subtask. Some systems with ordinary overall rankings may pass all instances of a specific reliability subtask, while an overall leading system may fail to solve it. This shows that evaluation should report layer-specific results rather than only a single leaderboard. Different systems may have complementary strengths in proofs, constructions, program optimization, and boundary-condition checking. A single total score can easily hide the structure of capabilities.

    4. Metric Design: From “Scores” to “Reviewable Artifacts”

    Metric design should avoid counting only final correctness rates. At minimum, the following metrics are recommended:

    1. Success rate: successful attempts divided by total attempts, but the attempt cap and budget must also be reported.
    2. Coverage: the number of distinct theorems, construction objects, or program modules covered by successful tasks, to avoid repeatedly solving the same easy task.
    3. Budget efficiency: the number of attempts, time, or resources required for success, reflecting search efficiency.
    4. Verifiability rate: the proportion of outputs that can be checked by an independent verifier, separating “looks plausible” from “verifiably correct.”
    5. Dependency level: whether a proof relies on undeclared intermediate lemmas, whether a construction violates constraints, or whether a program passes only examples.
    6. Robustness: the variance in success rate across repeated runs, to observe whether results are accidental.
    7. Failure classification: pseudo-proofs, out-of-bounds constructions, example overfitting, numerical errors, budget exhaustion, non-reproducibility, and so on.

    These metrics should be released together with failure examples. Failure examples often explain capability boundaries better than success examples: whether the model loses constraints in long-range dependencies, treats approximate values as exact values, or makes mistakes on verifier boundary conditions.

    5. Engineering Implementation and Preventing Reward Hacking

    When building a verifiable evaluation system, the engineering focus is not “running the model” but “ensuring that verification is trustworthy.” First, formalize and version tasks so that every evaluation uses the same task definitions and verifier versions. Second, develop and test verifiers independently, using known correct and incorrect examples for regression testing. Third, use sandbox isolation and resource quotas to prevent cross-task contamination and resource abuse. Fourth, record traces and ensure reproducibility by saving inputs, outputs, verification evidence, and random seeds. Fifth, require structured outputs and evidence chains so that models submit checkable objects rather than natural-language assertions. Sixth, conduct human spot checks by sampling and reviewing automatic verification results. Seventh, prevent reward hacking through hidden test sets, randomized boundary cases, property checks, attempt limits, and auditing of unusually high-frequency paths.

    At the infrastructure level, generic cloud containers, object storage, logging services, and monitoring and alerting capabilities can host sandboxes, evidence archiving, and runtime observability. However, the core remains task design and verifier quality. Without reliable verifiers, a larger evaluation scale will only amplify noise.

    6. Common Failure Modes and Human-Model Collaboration Boundaries

    Common failure modes include pseudo-proofs, where steps look coherent but lack key justification; implicit dependencies, where undeclared lemmas are used; out-of-bounds constructions, where objects violate constraints; program overfitting, where examples pass but properties are violated; numerical misjudgment, where floating-point approximations are treated as exact conclusions; non-reproducibility, where repeated runs differ significantly; and verifier vulnerabilities, where incorrect outputs are misjudged as correct. Corresponding mitigations include independent verification, hidden tests, formal constraints, budget limits, audit logs, and red-team spot checks.

    The boundary of human-model collaboration should also be clear. Tasks suitable for models are well-defined subtasks: proof drafts, construction search, program optimization, counterexample attempts, and boundary-condition checks. Tasks suitable for humans are: choosing research directions, defining problem constraints, judging the value of results, reviewing the evidence chain, and deciding whether to continue investment. Verifiable evaluation only shows that an output is reviewable; it does not show that the result has scientific significance. Therefore, a more realistic positioning is this: under verifiable feedback, models can complete parts of the research workflow output, but autonomous topic selection, attacks on major open problems, and value judgment still require humans.

    Overall, this kind of evaluation is more like a capability map than a simple win-loss ranking. It helps teams identify which tasks models can already handle with stable, reviewable outputs, which steps still fail, and which subtasks require human-model collaboration. For engineering practice, the most important thing is not to chase a single leaderboard score, but to build a verifiable, auditable, and budget-controlled evaluation and collaboration loop.