Verifiable Evaluation: A Quantitative Path to Measuring LLM Research Capabilities
This article explains how to evaluate large language models on open-ended research tasks by shifting from reference-answer matching to independently verifiable deliverables. It covers the core design principle behind verifiable evaluation, how to build task sets that span construction, bound-tightening, and conceptual-proof problems, a multi-dimensional scoring scheme, the engineering architecture of the evaluation pipeline, verifier quality assurance, common pitfalls in interpreting results, and cost and security considerations in practice.
1. Why Open-Ended Research Ability Is Hard to Evaluate
Mainstream LLM benchmarks mostly follow a “given input, compare against a reference answer” pattern: math problems compare the final numeric value, code problems run unit tests, and QA compares key-point match rates. This design is efficient and stable, but it carries an implicit assumption — that the correct answer to a task is unique and enumerable.
Once you swap the evaluation target to open-ended mathematical and theoretical research, that assumption breaks down. An extremal graph construction task may have infinitely many valid solutions; an improvement to a coding parameter bound depends on whether you found a better construction or a tighter derivation; a reliability proposition about the alphabet of a probabilistically checkable proof system tests precise understanding of the theorem’s structure, not formula plugging.
These tasks have three typical features that make them hard to evaluate:
- Non-unique answers: the same question admits many correct formulations, so string matching cannot decide correctness.
- Long chains with unreliable intermediate steps: when a model’s natural-language derivation says “clearly, it follows” or “by the lemma,” a reviewer can hardly tell whether this is real reasoning or filler.
- High subjectivity: when human scoring is involved, the scoring variance is often larger than the gap between models; different reviewers apply different standards for “completeness,” so conclusions are not reproducible.
This leads to a pragmatic direction: rather than chasing “evaluate like a human,” turn the research output into an object that can be independently re-checked. This is the premise of verifiable evaluation.
2. Core Design Principle: Output Must Be Independently Re-Checkable
Verifiable evaluation is essentially a shift in constraints: instead of asking the model to “explain clearly,” it asks the model to “deliver a recomputable result.”
In engineering terms, this means every task must be pre-bound to a verifier. The verifier is fully decoupled from the model and processes all submissions with the same deterministic program. It does not read the model’s self-description; it trusts only objective quantities that can be recomputed from the raw data.
Three mappings help illustrate this:
| Task type | Model deliverable | How the verifier judges |
|---|---|---|
| Graph structure construction | Vertex count, edge count, edge list | Rebuild the adjacency structure, detect forbidden subgraphs (triangles, quadrilaterals), and check whether the edge count meets the target |
| Parameter bound improvement | Code length, minimum distance, codeword set | Compare pairwise distances among codewords one by one and check whether the size exceeds a known reference value |
| Conceptual proof problem | Argument steps or a constructive explanation | Deterministically check whether all key conditions are fully covered |
The key insight of this table is not the specific tasks but the replayability of the judgment: anyone with the same submission and the same verifier gets exactly the same conclusion. The credibility of the evaluation no longer depends on the reviewer’s expertise or subjective preference, but on the correctness of the verifier.
3. Building the Task Set: Covering Different Capability Dimensions
An evaluation set composed only of “graph-drawing problems” cannot test the depth of theoretical understanding; one composed only of derivation problems cannot test construction ability. A reasonable task set should cover at least three categories:
Category 1: Construction tasks. Require building an object that satisfies all conditions under given constraints, for example a graph with girth 5 that contains no triangles or quadrilaterals. This category has the lowest judging cost and the hardest conclusions — the edge count either meets the target or it does not, and no matter how elegant the intermediate process, it cannot substitute for the final deliverable.
Category 2: Bound-tightening tasks. Require improving a known quantitative upper or lower bound. Its judgment is likewise recomputation: plug the delivered construction into the verifier, and if the size exceeds a published reference value, it counts as a valid improvement. These tasks are inherently falsifiable, and the model cannot bluff its way through with vague statements.
Category 3: Conceptual argument tasks. Require the model to correctly handle combinations of conditions in theorem statements, for example the effect of alphabet size changes on a reliability conclusion. These tasks often have the highest discrimination — they cannot be solved by search or trial and error, but depend on whether the model has truly internalized the domain’s system of definitions.
The proportions of the three categories determine the “profile shape” of the evaluation. If the goal is to assess general research ability, roughly equal weighting is recommended; if the goal is to assess a specific scenario (such as formal proof assistance), increase the weight of Category 1 and Category 3.
4. Scoring System: From a Single Score to a Multi-Dimensional Profile
Reporting only a single total score loses a great deal of information. It is recommended to output at least three dimensions simultaneously:
- Success rate: the proportion of successful completions across all attempts within a fixed attempt budget. This number measures stability.
- Coverage: the number of distinct problems successfully touched. Some models can repeatedly solve the same problem yet never reach a second one, inflating the success rate.
- Peak performance: with a relaxed attempt limit, the maximum number of problems solved. This reflects the capability ceiling rather than the average level.
These three dimensions must be viewed together. Consider an illustrative comparison: if an evaluated subject succeeds 23 times in 60 attempts, covering 9 distinct problems, its success rate is about 38.3%; while another subject completes 14 of 20 problems without being given intermediate lemmas, a 70% success rate. The former has worse exploration breadth, yet it may still pass a particular problem type on all three attempts, while the latter fails to deliver on exactly that problem. A total-score ranking would mask this kind of complementarity, whereas a per-dimension report exposes it — which is exactly where the real value of evaluation lies.
5. Engineering the Evaluation Pipeline
Putting the research process into one system requires four independently deployable components.
1) Task dispatch layer. Use a message queue to carry task delivery. Each task carries a unique identifier, a prompt template, a verifier identifier, and an attempt budget. The benefit is horizontal scaling: to increase evaluation concurrency, you only need to add consumer instances, without changing the scheduling logic.
2) Execution sandbox layer. Each evaluated subject runs in an isolated container with restricted resource quotas, network egress, and filesystem read/write scope. The container platform handles image distribution and elastic scaling, so the evaluation concurrency can adjust automatically with task volume.
3) Result retention layer. All submissions, intermediate constructions, and verification logs are written uniformly to object storage, archived by task identifier and timestamp. This is the prerequisite for reproducibility: any dispute over a conclusion can be traced back to the original deliverable at that time.
4) Monitoring and alerting layer. Verifier latency, sandbox timeout rate, and abnormal exit rate all need metrics and thresholds. The most common failure in batch evaluation runs is not a wrong model answer but a false negative caused by timeouts — mistaking an environment failure for a capability gap directly contaminates the entire batch of data.
A suggested verifier interface design (illustrative code, showing only responsibility separation):
from dataclasses import dataclassimport json@dataclassclass Task:task_id: strkind: str # construct | tighten | proveprompt: str # problem statement given to the subject under testverifier: str # verifier identifier, fully decoupled from the modelbudget: int = 3 # number of attempts allowedbaseline: dict = None # known reference value, used for tighten judgments@dataclassclass Outcome:task_id: strvalid: bool # whether it passes the hard checkexcess: float # improvement over the reference value, 0 if not passedattempts_used: intartifact_uri: str # location of the submission in object storagedef verify(task: Task, submission: dict) -> Outcome:"""Trust only objective quantities that can be recomputed from raw data."""if task.kind == "construct":ok = check_structure(submission["edges"], task.constraints)return Outcome(task.task_id, ok, 0.0, submission["attempts"], submission["uri"])if task.kind == "tighten":ok = recompute_bound(submission["witness"], task.baseline)gain = ok["size"] - task.baseline["size"] if ok["passed"] else 0.0return Outcome(task.task_id, ok["passed"], gain, submission["attempts"], submission["uri"])raise ValueError(f"unsupported kind: {task.kind}")
For construction tasks, the core logic of the verifier is to restore the edge list into a graph structure and then check each forbidden condition. The following illustrative code explains why the order of checks matters:
def check_structure(edges, constraints):n, forbid_tri, forbid_quad = constraints["n"], True, Trueadj = [set() for _ in range(n)]for u, v in edges:if u == v or not (0 <= u < n and 0 <= v < n):return {"passed": False, "reason": "illegal edge"}adj[u].add(v)adj[v].add(u)# Run the structural validity check first, then the property checks, to avoid wasting budget on invalid submissionsfor u in range(n):for v in adj[u]:if u < v and adj[u] & adj[v]:return {"passed": False, "reason": "triangle present"}# Quadrilateral detection: two distinct length-2 paths between any pair of points is a violationseen_pairs = set()for w in range(n):nbrs = sorted(adj[w])for i in range(len(nbrs)):for j in range(i + 1, len(nbrs)):pair = (nbrs[i], nbrs[j])if pair in seen_pairs:return {"passed": False, "reason": "quadrilateral present"}seen_pairs.add(pair)return {"passed": len(edges) >= constraints["min_edges"], "edges": len(edges)}
Two implementation suggestions: put the structural validity check first so obviously illegal submissions are rejected immediately, leaving the budget for promising attempts; reuse already-sorted intermediate results in property detection to avoid running a depth-first search for every pair of points, otherwise large graphs will turn the verifier itself into a bottleneck.
6. Quality Assurance for the Verifier Itself
The upper bound of evaluation credibility equals the upper bound of verifier precision. A carelessly written verifier systematically over- or under-estimates all evaluated subjects, and this bias is almost invisible in aggregate reports.
Three hard requirements:
- Independent implementation: the verifier must not share any code path with the execution environment of the subject under test. Even reusing a single utility function can introduce a collusion hole.
- Adversarial testing: prepare three fixed categories of test cases for each verifier — positive, negative, and boundary cases — especially approximate solutions that “look like they satisfy the conditions but actually don’t.” These cases should cover the most common failure modes.
- Recomputable judgment: the verification process must not rely on any probabilistic decision. Any step that requires “having a model re-evaluate once” should be regarded as the judgment standard for that problem not being fully designed.
7. How to Correctly Interpret Evaluation Results
Verifiable evaluation yields hard conclusions, but a hard conclusion is not the same as a complete one. Three common pitfalls are worth watching out for when interpreting results.
Mistaking the total score for capability. The total score is a function of the task set and weights. Swap the task set, and the ranking can completely reverse. The total score is only suitable for horizontal comparison under the same task set and the same verifier version.
Ignoring capability complementarity. Top of the ranking does not mean leading across the board. One evaluated subject may pass a conceptual problem about alphabet reliability on all three attempts, while a higher-ranked subject fails to solve it; conversely, the latter may be far ahead on long-chain construction tasks. The value of evaluation lies precisely in characterizing such differences, not compressing them into a single number.
Confusing “environment failure” with “insufficient capability.” Timeouts, sandbox crashes, and image-pull failures all lead to no submission, and if not flagged separately, they are silently counted as failures. This noise must be separable and counted separately.
8. Cost and Security Considerations in Practice
The invocation cost of research-oriented evaluation is significantly higher than that of conventional benchmarks: long contexts, multiple attempts, and repeated sampling. It is recommended to control this from three angles:
- Tiered sampling: first use a small budget for coarse screening, and only relax the attempt count and sampling temperature for subjects that enter the candidate pool.
- Result caching: cache submissions that pass verification, keyed by “hash of the problem content + model identifier + prompt version,” so that repeated batch runs get a direct hit.
- Quotas and circuit breakers: set a total-volume cap for each batch run; when exceeded, stop automatically while retaining the completed intermediate results, avoiding a single abnormal run wiping out the entire budget.
On security: the sandbox must be offline by default to prevent the subject under test from accessing external resources during execution and contaminating the evaluation conclusion; when archiving submissions, retain the original version and do not overwrite on write after verification, otherwise the judgment process cannot be re-checked afterward. The components above can be assembled on the cloud using general-purpose container platforms, object storage, message queues, logging services, and monitoring/alerting. Platforms like Baidu AI Cloud provide these general-purpose category capabilities, which do not constitute part of the evaluation methodology itself.
9. Conclusion
The evaluation of open-ended research ability ultimately comes down not to “how human-like the model’s answer is,” but to “whether its output can be independently re-checked.” Moving the research process into one verifiable system means three things must hold simultaneously: tasks have hard judgments, deliverables are recomputable, and the process is traceable.
The value of this method goes beyond ranking. When constructed graph structures, tightened parameter bounds, and fully covered argument chains are all settled in a recomputable form, they themselves constitute incremental assets within the domain. Evaluation shifts from a resource-consuming overhead into an engine that continuously produces verifiable results — this is where verifiable evaluation is truly worth investing in.
