Evaluating LLM Research Capability: How to Build Machine-Verifiable Benchmarks

    An AI benchmark has drawn attention for requiring large language models to prove theorems, construct combinatorial objects under strict constraints, and tighten known bounds in combinatorics and theoretical computer science. This article looks past the leaderboard to break down how to scientifically evaluate LLM research ability: what qualifies as a machine-checkable research task, how to build independent verifiers, how to read results under different scoring protocols (exact-match, all-budget, and one-shot), the statistical pitfalls of small samples, and how to assemble a reproducible evaluation pipeline. It also covers confidence intervals, contamination resistance, and practical engineering components.

    1. Introduction: Why “Research Capability” Is Hard to Evaluate

    A benchmark released by a large-model company in collaboration with well-known university researchers has drawn wide attention. It asks models to complete hardcore tasks in combinatorics and theoretical computer science—proving theorems, constructing combinatorial objects that satisfy strict constraints, and improving known parameter upper bounds—and reports impressive scores.

    Set aside the rankings. What is genuinely interesting about this benchmark is its evaluation philosophy: it does not judge a model by “how plausible its output sounds,” but requires the model to deliver artifacts that can be independently verified. This directly addresses one of the thorniest problems engineering teams face once large models enter the research-assistance workflow:

    • Natural-language output “reads convincingly,” but there is no way to tell whether it is true;
    • Research questions usually have no single correct answer, so human grading is costly and subjective;
    • Once a fixed question set is public, it can be “memorized” by training corpora—measuring recall rather than capability.

    This article abstracts away specific model and product names and dissects “how to scientifically evaluate an LLM’s research capability” across five layers: benchmark design, judge implementation, result interpretation, statistical pitfalls, and engineering deployment.

    2. Defining the Problem: What Counts as a “Research Task” Benchmark

    LLM evaluation has evolved from “perception and language” to “code and math” and now to “research.” The core change at each stage is how the correctness of an output is judged:

    Stage Typical tasks Judgment method Main limitation
    Language understanding Reading comprehension, commonsense QA Compare against reference answers Hard to reflect complex reasoning
    Math & code Competition problems, function completion Compare numeric answers, run unit tests High contamination risk; narrow task forms
    Research assistance Theorem proving, combinatorial construction, algorithm optimization Independently verifiable artifacts High judge-building cost; needs a sound budget definition

    At its core, a research-task benchmark tests whether a model can close the loop of search, construction, and verification within a “given problem space.” Here, verification is not the model talking to itself—it is a binary verdict (pass or fail) produced by a judge program that is independent of the model.

    Designing such a benchmark therefore requires answering at least four questions:

    1. Is the task objectively decidable? Is there a mechanically executable correctness standard?
    2. Can the artifact be verified automatically? Or does judging depend on subjective expert review?
    3. How is the sampling budget set? How many attempts is the model allowed, and how is the score tallied?
    4. Can the result resist data contamination? Will the question lose its discriminative power once it is public?

    3. Core of Benchmark Design: Making “Correct” Machine-Provable

    3.1 Three Machine-Checkable Output Forms

    The tasks in such a benchmark fall into three categories, and their common trait is that every output can be verified independently by a program:

    Output type Typical form Verification method Verification cost
    Proof Theorem proof, expressible formally, or accompanied by key lemma derivations Formal proof checker, or section-by-section review by lemma dependency Higher; depends on proof complexity
    Construction Graph, code, or combinatorial design satisfying several constraints, submitted as an edge list, matrix, etc. Program enumerates and checks that all constraints hold Low; usually sub-second
    Optimization / improvement Tighter parameter bound, better algorithm, faster program Recompute the metric against the given witness, or run test cases and benchmarks Low to medium

    Take combinatorial construction as an example. The judging process is entirely decoupled from how the model “thinks”: read the submitted graph, check the validity of vertices and edges, then verify constraints one by one—“no triangles,” “no quadrilaterals,” “edge count no lower than the reference value.” No matter how eloquently the model “explains” itself, a single offending edge makes the verdict a failure.

    3.2 Judge Example: Independently Verifying a Construction

    The sample code below implements two judges:

    • check_construction handles construction tasks under a girth constraint. “Girth” is the length of the shortest cycle in a graph; if the shortest cycle length is at least 5, the graph contains neither triangles nor quadrilaterals. The function first validates the submitted structure itself, then enumerates cycles of length up to 4.
    • check_edit_distance uses the classic dynamic-programming algorithm to turn an upper-bound-improvement task into “can we compute a result no greater than the target given these parameters,” decoupling the judgment from the algorithm the model submits.
    1. from itertools import combinations
    2. def check_construction(vertices, edges, min_edges: int, girth_lb: int = 5):
    3. """Verify a construction artifact: structural validity + combinatorial constraints + target metric"""
    4. # 1) Structural validity: valid vertex indices, no self-loops, no duplicate edges
    5. if any(not (0 <= u < vertices and 0 <= v < vertices) for u, v in edges):
    6. return False, "invalid vertex index"
    7. if len({tuple(sorted(e)) for e in edges}) != len(edges):
    8. return False, "self-loop or duplicate edge present"
    9. adj = [[] for _ in range(vertices)]
    10. for u, v in edges:
    11. adj[u].append(v)
    12. adj[v].append(u)
    13. # 2) Combinatorial constraints: when max_girth=4, equivalent to enumerating triangles and quadrilaterals
    14. for u in range(vertices):
    15. for v in adj[u]:
    16. if v > u and set(adj[u]) & set(adj[v]): # triangle
    17. return False, f"triangle present: {u}-{v}"
    18. for a, b in combinations(adj[u], 2):
    19. if a < b and set(adj[a]) & set(adj[b]) - {u}: # quadrilateral
    20. return False, f"quadrilateral present: {a}-{u}-{b}"
    21. # 3) Target metric: whether the edge count exceeds the reference value
    22. if len(edges) < min_edges:
    23. return False, f"edge count {len(edges)} < reference value {min_edges}"
    24. return True, f"passed: {vertices} vertices, {len(edges)} edges"
    25. def check_edit_distance(source: str, target: str, k: int) -> bool:
    26. """Independently recompute with classic dynamic programming; judgment is decoupled from the model's algorithm"""
    27. prev = list(range(len(target) + 1))
    28. for i, sc in enumerate(source, start=1):
    29. cur = [i]
    30. for j, tc in enumerate(target, start=1):
    31. cost = 0 if sc == tc else 1
    32. cur.append(min(prev[j] + 1, # deletion
    33. cur[j - 1] + 1, # insertion
    34. prev[j - 1] + cost # substitution
    35. ))
    36. prev = cur
    37. return prev[-1] <= k
    38. def judge(task, artifact) -> bool:
    39. """Unified judgment entry point; can be hooked into a CI pipeline"""
    40. if task["kind"] == "construction":
    41. ok, _ = check_construction(artifact["vertices"],
    42. artifact["edges"],
    43. task["min_edges"])
    44. return ok
    45. if task["kind"] == "threshold":
    46. return check_edit_distance(artifact["source"],
    47. artifact["target"],
    48. task["k"])
    49. raise ValueError(f"unknown task type: {task['kind']}")

    Three design lessons from this judge are worth borrowing:

    1. Judge logic is fully decoupled from the subject under test. The judge does not read the model’s reasoning; it reads only the structured artifact.
    2. Failure causes can be pinpointed. For example, “which edge introduced the quadrilateral,” which supports error analysis and follow-up attribution.
    3. It can serve as a quality gate in a CI pipeline. A passing judgment is the minimum entry condition for wiring a model into a research-assistance workflow.

    3.3 Output Interface: Make the Model “Deliver,” Not “Narrate”

    So that artifacts can be consumed by a program, the task prompt should specify the output format as a structured, machine-readable schema—for example, submitted in a structure like this:

    1. {
    2. "task_id": "extremal-graph-girth-5",
    3. "artifact": {
    4. "kind": "graph",
    5. "vertices": 124,
    6. "edges": [[0, 1], [0, 2], [1, 3]]
    7. },
    8. "claim": "edge count 630, exceeding the reference value 629"
    9. }

    Note that a model’s claimed conclusion can never serve as the basis for a verdict—it can only act as a locating clue or a reference for human review.

    4. Three Scoring Protocols: Exact-Match, All-Budget, and One-Shot

    For the same set of tasks, combining “whether hints are given” with “how many attempts are allowed” produces several tallying protocols, each with its own emphasis:

    Protocol Description Focus Applicable scenario
    Exact-Match (Full) Score only when the artifact fully passes the judge Complete one-shot solution Quality-sensitive research workflows
    All-Budget Count successes within a fixed number of attempts Search and self-correction ability Scenarios allowing repeated trials with ample compute
    One-Shot No intermediate hints, only one chance End-to-end independent problem solving Measuring the “first attempt” in real research

    For example, from the reported measurements:

    • Under exact-match, the leading model scored 71.9, second place 57.4, and the remaining six models all scored no more than 11.7;
    • Under one-shot, the leading model completed 14 of 20 tasks (70.0%);
    • Under all-budget, the second-place model succeeded 23 times in 60 attempts (38.3%), spanning 9 distinct theorems.

    The same group of models paints a different portrait under each protocol. So the correct way to read leaderboard conclusions is not “who has the higher total,” but “under which constraints, who can reliably deliver which kind of artifact.“

    5. Interpreting Results: What This “Capability Map” Shows

    5.1 A Score Gap Does Not Equal Total Dominance

    Two points in the public results are easy to over-interpret:

    1. The Pavl theorem proof task: under the second protocol, the second-ranked model completed it in all 3 attempts, while the leading model failed to solve it. This shows that “task difficulty” is not globally uniform—different models have clearly different capability distributions.
    2. Construction and bound improvement: these results beat existing reference values by only “one more edge” or “tightening by two units.” In combinatorics, this is still valuable progress, but it must be stressed: the increment is small, and it must be accompanied by a verifiable witness; otherwise its truth cannot be confirmed.

    5.2 Separate “Problem-Selection” Difficulty from “Execution” Difficulty

    A research-capability evaluation should split tasks into two layers:

    • Problem-selection difficulty: finding a problem that is worth doing, doable, and still unsolved;
    • Execution difficulty: completing a construction, proof, or optimization under a given problem.

    The problems in this benchmark are pre-assigned by the question setters, so they mainly test execution ability. Therefore, a high leaderboard score only means the model is strong at “executing solutions”—it does not mean the model can independently select research problems.

    5.3 Statistical Pitfalls: A Small-Sample “Counterexample” Proves Little

    Under all-budget, 23 successes in 60 attempts average about 2.6 attempts per solved theorem. This reveals two signals:

    • Instability: results vary noticeably across repeated attempts on the same problem;
    • Limited coverage: the 23 successes fall on only 9 distinct theorems, indicating uneven capability across problem types.

    Estimating with the binomial distribution, the 95% confidence interval for a 38.3% success rate is roughly 26%–51%, close to the boundary between success and failure. Hence, any conclusion of the form “model X is stronger on task Y” should rest on enough independent attempts. A one-time or three-time comparison (e.g., “3-for-3” versus “unsolved”) can only be recorded as a phenomenon—it is not sufficient as a conclusion.

    5.4 Verifiability Is the Real Currency

    What truly separates this benchmark from leaderboards that merely “look strong” is that it requires every conclusion to come with a verifiable object:

    • a graph with 124 vertices and 630 edges containing no triangles or quadrilaterals;
    • an argument that tightens a parameter bound from 84 to 82.

    Any team can independently run the judges using the methods in Section 3 and confirm whether these results hold. This is the key step by which research evaluation moves from “authority endorsement” toward “evidence-driven.”

    6. Statistics and Evaluation Methods: Making Leaderboard Conclusions Hold Up

    6.1 Replace Point Estimates with Confidence Intervals

    Point estimates under small samples carry large randomness. It is advisable to use the Wilson interval, which is better suited to small samples and extreme proportions (such as 0% or 100%) than the naive normal approximation:

    1. import math
    2. def wilson_interval(successes: int, total: int, z: float = 1.96):
    3. """Wilson confidence interval, suitable for small samples and extreme proportions"""
    4. p = successes / total
    5. denom = 1 + z ** 2 / total
    6. center = (p + z ** 2 / (2 * total)) / denom
    7. half = z / denom * math.sqrt(p * (1 - p) / total
    8. + z ** 2 / (4 * total ** 2))
    9. return center - half, center + half

    For instance, the point estimate for 23/60 is 38.3%, but the 95% confidence interval is roughly 26%–51%. When reporting results, always give both the point estimate and the interval, so readers do not mistake random fluctuation for stable capability.

    6.2 Distinguish “Model Capability” from “Sampling Strategy”

    Sampling parameters such as temperature, top-p, and maximum output length significantly affect the success rate. An evaluation report must fix and disclose these parameters; otherwise cross-model comparisons do not hold.

    6.3 Control Attempts and Cost

    The all-budget protocol naturally favors models with “more attempts.” A fair approach is:

    • fix the attempt budget per task (e.g., 60 attempts);
    • or fix the total cost (e.g., a token budget);
    • and report both “success rate per attempt” and “total number of successes.”

    6.4 Resist Data Contamination

    Once a question bank is public, it may enter the training corpora of later models. Practical mitigations include:

    • keeping a private holdout set and rotating it periodically;
    • auto-generating new problems via parameterized variants;
    • perturbing problems (changing parameter scale, boundary conditions) and re-testing.

    7. Engineering Practice: Building a Reproducible Research-Evaluation Pipeline

    7.1 Overall Architecture

    1. ┌────────────┐ task dispatch ┌───────────────┐ structured ┌──────────────┐
    2. │ Scheduler │ ───────────────> │ Sandbox │ ────────────> │ Judge cluster│
    3. └─────┬──────┘ └───────┬───────┘ artifact └──────┬───────┘
    4. │ │ │
    5. │ task / model version / params │ candidate artifact │ pass / fail
    6. ▼ ▼ ▼
    7. ┌─────────────────────────────────────────────────────────────────────┐
    8. │ Result storage & observability: per-attempt records + run logs + │
    9. │ monitoring alerts + cost tags │
    10. └─────────────────────────────────────────────────────────────────────┘

    7.2 Key Component Responsibilities

    • Scheduler: handles task dispatch, attempt quotas, timeout control, and retries.
    • Sandbox execution environment: provides an isolated runtime for model inference and the judges, limiting CPU, memory, and execution time to prevent a judge from “hanging” on maliciously crafted input.
    • Judge cluster: runs the judge logic as a stateless service or function, ensuring that judging the same artifact repeatedly yields a consistent result.
    • Result storage & observability: records each attempt’s inputs/outputs, verdict, latency, and cost, enabling retrospectives and regression comparisons.

    If a team runs on Baidu AI Cloud, it can optionally use general-purpose capabilities to support these stages:

    • object storage to hold the question bank and artifacts;
    • a message queue for task distribution and load smoothing;
    • a container platform to elastically scale sandboxes and judges;
    • a logging service and monitoring alerts to track judge failure rates and resource utilization.

    7.3 Pipeline Pseudocode

    1. def run_benchmark(tasks, model, budget: int, judge_fn):
    2. """Evaluation main loop under the all-budget protocol"""
    3. results = []
    4. for task in tasks:
    5. for attempt in range(1, budget + 1):
    6. artifact = model.solve(task) # call the model under test
    7. ok = judge_fn(task, artifact) # independent judgment
    8. results.append({
    9. "task_id": task["id"],
    10. "attempt": attempt,
    11. "passed": ok,
    12. "model_version": model.version,
    13. "seed": task.get("seed"),
    14. "latency_ms": task.get("latency_ms"),
    15. })
    16. return summarize(results, budget) # aggregate metrics across protocols

    7.4 Suggested Summary Metrics

    Metric Calculation Purpose
    Exact-match score Proportion where the first artifact passes Assess delivery quality
    All-budget success rate Successes / (tasks × budget) Assess search and correction ability
    One-shot pass rate Pass rate with no hints and a single attempt Assess end-to-end independent solving
    Coverage Share of tasks solved at least once Assess breadth of capability
    Stability Variance and confidence interval across repeated attempts Judge whether conclusions are trustworthy
    Cost-normalized success rate Successes / total cost (tokens or spend) Compare cost-effectiveness

    8. Conclusion and Recommendations

    The most important lesson from this benchmark is not “who took first place,” but that it pulls research-capability evaluation back from “narrative” to “evidence”:

    1. Verifiable artifacts: proofs, constructions, and optimization results can all be independently verified by a program—the cornerstone of evaluation credibility;
    2. Coexisting protocols: exact-match, all-budget, and one-shot each have their own emphasis, and a single total score easily masks capability differences;
    3. Statistical rigor: small-sample conclusions must carry confidence intervals, or a “counterexample” may just be random fluctuation;
    4. Engineering reproducibility: sandboxes, judges, versioned records, and observability systems are prerequisites for reproducible evaluation results.

    For teams preparing to bring LLMs into a research-assistance workflow, the recommendations are:

    • Build the judge first, then pick the model: treat “can the artifact be automatically verified” as a precondition for onboarding a task;
    • Choose metrics by protocol: for delivery-sensitive scenarios look at exact-match; for exploratory scenarios look at all-budget success rate;
    • Keep a human in the loop: let the model handle the “execution layer” of construction, proof, and optimization, while reserving problem selection, value judgment, and final sign-off for people.

    LLMs can already reliably produce some verifiable research artifacts under conditions of “clear problems, sufficient budget, and checkable feedback.” But when it comes to independently choosing research directions, judging the value of results, and cracking major open problems, human judgment remains indispensable. The purpose of an evaluation system is precisely to draw this capability boundary ever more clearly.