# How to Evaluate LLM Research Capability: A Verifiable Task System Breakdown

    This article examines how to evaluate the research capabilities of large language models using verifiable tasks, drawing on TCSExam, a benchmark suite built around theoretical computer science research tasks. It explains how research outputs can be turned into independently checkable objects, what makes a research task well-defined and verifiable, and how a task-pool checklist, difficulty tiers with lemmas as scaffolding, and budget controls shape a benchmark. It also lays out a three-layer verification architecture covering sandboxed execution, rule-based adjudication, and result aggregation, walks through concrete results in extremal graph construction and coding-theory bounds, and separates attempt success rate from theorem coverage rate. Finally, it assesses what LLMs can and cannot yet do in research and provides a five-step guide for building your own verifiable research benchmark.

    Introduction: When LLMs Start “Doing Research”

    Over the past two years, LLMs have improved steadily on benchmarks for mathematical reasoning, competition-level programming, and code generation. Most of these benchmarks, however, follow a “problem plus reference answer” paradigm and measure problem-solving ability on closed-form questions. Real research differs in three ways: it is open-ended (no single correct answer), long-horizon (conjecture, construction, and verification span multiple steps), and produces diverse outputs (a proof, a set of parameters, a program, or a concrete construction). Multiple-choice questions and unit-test-style evaluation alone therefore struggle to reflect how capable a model is of participating in research.

    Addressing this gap, a group of researchers working with university research teams proposed TCSExam. The name points to research tasks from theoretical computer science (TCS), and the core idea is to place all kinds of research outputs into one verifiable system.

    Note on data (verify before citing)

    Item Convention in this article
    Data source TCSExam public leaderboard (Full track). When citing, add the link to the official leaderboard and the evaluation cutoff date.
    Model IDs Models are referred to throughout by anonymous IDs (M1, M2, …); vendors and version numbers are not disclosed. When citing publicly, include the versions of participating models.
    Scoring convention Scores use a 100-point scale and represent weighted composite scores across tasks; they are not equivalent to success rates.
    Leaderboard results The top model scored 71.9, the runner-up 57.4, and the other six participating agents all scored no more than 11.7 (out of 100), a clear gap below the leaders.

    Scores are always expressed as “points” and never mixed with percentages, so that readers do not misread 11.7 points as an 11.7% success rate.

    1. Core Idea: Turn “Research Outputs” into “Verifiable Objects”

    Traditional evaluation relies on reference answers, yet frontier research problems precisely do not have them. TCSExam changes the object of evaluation: a model’s output is no longer a solution that merely “looks reasonable,” but an entity (an object, a set of parameters, or a proof) that can be independently checked. Scoring does not depend on subjective human judgment but on strictly defined verification rules.

    Task type Typical question Deliverable Verification method
    Extremal construction Given a vertex count, construct a graph with as many edges as possible that contains neither triangles nor quadrilaterals Complete graph structure (vertex and edge lists) A program checks for forbidden subgraphs, counts the edges, and compares against a reference value
    Parameter optimization Given a template in a constraint satisfaction problem, find parameter values A set of parameters Analytic or numerical checks that the constraints are satisfied
    Probability and check reliability Given constraints, determine the alphabet reliability parameters Symbolic algorithms or value-selection strategies Formal or exhaustive arguments based on the definitions

    This delivers value at three levels: results are objective (correctness is decided by algorithms); the process is auditable (a third party can recompute it using the same rules); and capabilities are pinpointable (different tasks correspond to different stages of the research pipeline).

    2. Task Design: Clear Boundaries Make for Meaningful Challenges

    A qualified research-oriented task meets at least four conditions:

    • Well-defined: inputs, outputs, and constraints have formal definitions.
    • Strictly verifiable: there is a decision criterion that can be executed within reasonable resources.
    • Discriminative: the task differentiates among models by difficulty.
    • Reliable reference point: it provides either the best known result from the literature or a widely accepted baseline.

    Take the extremal graph task as an example: construct an undirected graph with 124 vertices and as many edges as possible that contains neither triangles nor quadrilaterals; the reference baseline is 629 edges. Verification works by checking whether each edge is legal, whether any forbidden subgraphs exist, counting the total number of edges, and comparing the total against the reference value. There is no subjective scoring step anywhere in the process.

    Task-pool selection checklist (ready to use)

    1. Input/output schema: fields, types, value ranges, and serialization formats.
    2. Verifier interface: input paths, return structure, and conventions for timeouts and error codes.
    3. Reference baseline: literature values or accepted optimal values, with sources cited.
    4. Budget settings: limits on the number of attempts, compute, and wall-clock time.
    5. Counterexamples and forbidden-pattern list: explicitly prohibited substructures, invalid values, and degenerate outputs.

    3. Budget and Difficulty Tiers: Calibrating Challenge Strength with “Intermediate Lemmas”

    The benchmark has two modes that form a difficulty gradient.

    In Full mode, no intermediate lemmas are provided, and the solving model must carry out problem decomposition and derive intermediate propositions on its own. The score gradient of 71.9 / 57.4 / ≤ 11.7 reflects this difficulty. In Theorem mode, a dedicated proof agent first runs and verifies lemmas in a sandbox, then passes the lemmas along with the problem statement to the solving model. This is effectively “scaffolding.” This tiering separates the observation of two capabilities, research decomposition and local solving, and also keeps difficulty from spiraling out of control. Weaker models can still make progress on subtasks, and since every lemma is verified by actual sandbox runs, the scaffolding itself is also an object of verification.

    Budget control is equally essential: models must deliver within a defined number of attempts or compute budget, with success rate = successful attempts ÷ total attempts. Without budget constraints, an evaluation easily attributes differences to “whoever has more compute.”

    4. Making Verifiability Work: Three Layers of Safeguards

    Sandbox execution layer. Model-generated code is untrusted and must run in a restricted environment with limits on CPU, memory, and runtime (isolation). Pinning dependency versions and random seeds ensures that repeated verification of the same output produces the same conclusion (reproducibility). Executing logs, runtimes, and failure reasons are recorded in full (observability).

    Rule-adjudication layer. Verifiers are independent programs that are agnostic to the model. Following a “generate and test” approach, they check whether graph structures are valid and free of forbidden subgraphs, whether parameters fall within allowed ranges, and whether algorithms and probability claims meet theoretical conditions. Every model output passes through the same gate, which is what makes fair cross-model comparison possible.

    Result aggregation layer. Scores are stored together with metadata such as model IDs, task IDs, attempt counts, and runtimes, forming experiment records that can be queried and replayed. A common cloud setup is a container service for batch verification, object storage for task specifications and artifact logs, message queues for smoothing out traffic peaks, and logging plus monitoring and alerting to track success rates.

    Common mistakes:

    • Running verifiers in the same environment as model output.
    • Leaving random seeds unfixed, which causes results to drift.
    • Omitting budget limits, which distorts compute comparisons.
    • Not retaining logs, which makes failures impossible to attribute.
    • Reporting only a total score and not the number of tasks covered.

    5. Evaluation Examples: Understanding “Research Capability” through Concrete Results

    Improved construction. One participating model delivered a graph with 124 vertices and 630 edges that contains neither triangles nor quadrilaterals, surpassing the 629-edge reference baseline adopted by the task. The model did not restate an existing result; it found a better solution within the feasible region, and the verification was done entirely by a program, so anyone can check it.

    Proofs and tightened bounds. Another participating model established A(22,10) ≤ 82 on a coding-theory bound problem. The previously accepted upper bound for this value, in publicly available coding-theory tables (the type of literature maintained by Brouwer et al.), was 84. Results like this are rigorous conclusions that can be checked formally or structurally, not approximate heuristics. A suggested four-step review process for proof-type results:

    1. Formal verification (using proof assistants or type checkers).
    2. Step-by-step review of the structured reasoning chain.
    3. Human spot-checks of key steps.
    4. Isolated execution of untrusted code.

    When citing such a result, state the specific literature version and the verification method used.

    Different strengths. On a PCP alphabet reliability task, an anonymous model, M3, completed all three attempts, whereas M1, ranked first overall, failed to solve it. Research problems have highly diverse solution paths: some depend on constructive intuition, while others depend on rigorous theoretical derivation. A single overall score reflects only aggregate performance and cannot capture strengths in specific directions. A capability profile is more informative than a ranking.

    Do not conflate two metrics. Under a budget of 60 attempts, M1 solved 14 of the 20 theorems, giving a theorem coverage rate of 14/20 = 70.0% and a single-attempt success rate of 14/60 = 23.3%. The runner-up, M2, succeeded 23 times and covered 9 distinct theorems, giving an attempt success rate of 23/60 = 38.3% and a coverage rate of 9/20 = 45.0%. In other words, M2’s single-attempt success rate is actually higher, but its breadth of coverage lags clearly behind. Coverage measures “how many kinds of problems can be solved,” while attempt success rate measures “how many repetitions it takes to succeed once.” Research scenarios need both: conclusions must be reproducible, and the range of problems covered also matters.

    6. Capability Boundaries: AI as “Assistant,” Not Yet “Leader”

    What models can already do:

    • Carry out formal or structured reasoning (local proofs) given a problem and intermediate lemmas.
    • Search for better parameters or implementations (program optimization) under explicit constraints.
    • Produce constructions (construction improvement) on concrete mathematical objects that outperform known baselines.

    The common preconditions are that the problem is clearly defined, the budget is sufficient, and there is a verifiable feedback signal.

    Remaining limitations:

    • Models cannot autonomously judge “which problem is worth working on” (research direction selection).
    • It is difficult for models to judge whether a given improvement has academic significance (assessing the value of a result).
    • Major open problems still require the long-term accumulation and theoretical insight of human researchers.

    This benchmark resembles a topographic map of AI research capabilities: it marks which regions can already be explored autonomously and which still require human navigation. For developers, its value lies in guiding the division of labor between humans and machines. Hand the model the parts that are verifiable, iterable, and have clear criteria; leave problem selection, value judgment, and theoretical breakthroughs to humans.

    7. Practical Advice: Building Your Own Research-Oriented Evaluation System

    1. Define the task pool. Prioritize tasks with clear criteria that can be verified programmatically, and cover all three output types: constructions, optimizations, and proofs. Complete the schema, verifier interface, baselines, budgets, and forbidden-pattern items using the checklist in Section 2.
    2. Establish tiered difficulty. Adapt the intermediate-lemma approach to design scaffolding for models of different levels, and use controlled experiments to assess how effective the scaffolding is.
    3. Implement independent verifiers. Decouple verifiers from solving models, and keep verification fast, stable, and re-entrant.
    4. Set budget and statistical conventions. Apply uniform attempt-count and compute limits, and report both attempt success rate and theorem coverage rate to avoid overgeneralizing from a single number.
    5. Build up capability profiles. Aggregate results by task type and record them in a consistent table template. Use them to design human-machine division of labor rather than for a single ranking.
    Task type Model ID Attempts Successes Attempt success rate Theorems covered Coverage rate Average runtime Main failure reason
    Extremal construction M1 60 — — — — — Forbidden subgraphs not eliminated
    Bound proof M2 60 — — — — — Break in the reasoning chain

    8. Quick Answers to Common Questions

    Q: How does a verifiable research evaluation differ from an ordinary code benchmark?
    Ordinary benchmarks compare against reference answers. A verifiable research evaluation compares whether an output satisfies strict rules, and it allows answers that are better than the reference baseline.

    Q: How do I get the task pool started?
    Begin with 3–5 tasks that have public baselines and whose verifiers finish in minutes, covering constructions, optimizations, and proofs. Then expand the pool gradually.

    Q: How do I choose between models?
    Do not rely on a single overall score; look at the capability profile: coverage rate, attempt success rate, output per unit of budget, and the distribution of failure reasons.

    Conclusion

    TCSExam represents more than a model competition. It is an upgrade to research evaluation methodology, shifting the focus from “what the model said correctly” to “what the model delivered and whether it can be independently verified.” What truly deserves attention is not who came in first, but under what conditions a model can consistently deliver verifiable results.