Verifiable Benchmarks: How to Evaluate and Deploy Large Model Research Capabilities

    Verifiable benchmarks offer a practical way to measure how well large models perform research work. This article explains why single-turn Q&A evaluation has lost its discriminative power and why the more reliable approach is to turn research tasks into verifiable tasks whose outputs—proofs, programs, constructions, and code—can be checked automatically. It covers the evaluation system design (task spectrum, a unified verifier with four layers, attempt-budget semantics, and results presentation), interprets the findings through anonymized example data, and shows how engineering teams can adapt the same generate-verify-iterate methodology to real development workflows. Readers will also find a common-misuse checklist, a reference agent architecture, and a discussion of current limitations.

    Introduction: How Do You Measure a Large Model’s Research Ability?

    How do you measure a large model’s research ability? Answer quality alone is hard to judge. A more reliable approach is to convert research tasks into verifiable tasks, where proofs, programs, constructions, and coding results can all be rechecked by automated verifiers. In recent years, large models have performed increasingly well on standardized exams, competitive programming problems, and undergraduate math problems, so single-turn Q&A evaluation has gradually lost its ability to distinguish between models. The real question is this: when given an open task drawn from a real research setting, can a model produce a result that can be independently rechecked—for example, a complete proof, a better program, a stronger construction, or improved parameters?

    These outputs are fundamentally different from answers to traditional problems. First, the correct answer is not known in advance. A reference solution is often merely the best result already known; a better solution may or may not exist. Second, correctness cannot be judged by appearance. A derivation fragment in a formal proof may look reasonable yet contain type errors, missing steps, or circular references; a construction described in natural language may also fall “just short” of satisfying its constraints. Third, evaluation requires tools. Programs must compile, run, and pass tests; graph structures must be checked item by item against their definitions; and coding upper bounds must be verified by an independent proof.

    As a result, evaluating research ability requires closing the loop between generation and verification, with every conclusion decided by a repeatable checker. This article analyzes a verifiable research-task evaluation approach proposed jointly by an AI research team and academic researchers, examining its task design, verification loop, and result interpretation. It then explains how enterprise teams can put a generate-verify-iterate R&D workflow into practice under clear objectives, sufficient budget, and verifiable feedback.

    A note on data sources and terminology: This article is a methodology-oriented rewrite based on publicly available verifiable evaluation cases. The specific evaluation organizations, leaderboard names, and original data should be treated according to their public sources. To avoid misinterpretation, the models are anonymized here as Model A and Model B, and some figures are labeled as illustrative, used to explain differences between statistical calibers. This is not a final ranking judgment about any fixed leaderboard.

    Evaluation System Design: Putting the Research Process Inside a Verifiable Loop

    The core idea of this evaluation system is to place the various objects a model might produce during research into a single automated verification framework, rather than looking only at whether the final answer is right. A reusable design can be broken into the following five stages.

    2.1 Task Spectrum: Covering Multiple Forms of Research Output

    The task set aims to stay close to the way theoretical computer science and combinatorics research is actually done. Common tasks include:

    • Theorem proving: Given a proposition, produce a complete derivation. Difficulty tiers are distinguished by whether intermediate lemmas are provided. The low tier hints at key lemmas, constructions, or ideas; the high tier provides only the proposition itself. The latter is close to the situation a researcher faces with a real open problem.
    • Program-level problems: Express combinatorial structures, code tables, or search bounds as computational problems, and require the model to write a program that searches or optimizes, using the program’s output as the final evidence.
    • Combinatorial construction: Find new structures under a series of constraints, such as short-cycle-free graphs in extremal graph theory or balanced arrangements in combinatorial design.
    • Coding theory: Improve a known upper or lower bound for a class of codes, where the conclusion must be given as a construction or an independent proof.

    This task spectrum is closer to research production than plain Q&A. A model cannot output only an answer that “looks reasonable”; it must deliver an object that can be verified.

    2.2 A Unified Verifier: One Logic, Four Levels

    Each task is paired with a deterministic verifier, and any conclusion a model submits must pass it. Verification is usually divided into four levels:

    1. Syntax and format level: The deliverable can be parsed, and formal proofs pass the proof checker.
    2. Definition-constraint level: Check whether each condition in the definitions is satisfied. For example, for a graph that is “triangle-free and quadrilateral-free,” you must verify that at most one path of length no greater than 4 exists between any two vertices.
    3. Metric-comparison level: Compare against a known reference value or a published bound, such as the running quality of a program, the number of edges in a construction, or the upper bound on coding parameters.
    4. Independent recheck level: Proof-based conclusions must be rechecked by humans or formal tools, ruling out blind spots in the verifier itself.

    The value of the verifier lies in turning subjective judgment into a repeatable decision. A model does not need to “convince the reader”; it only needs to make the conclusion pass the check.

    2.3 Budget Semantics: One Attempt or Many?

    Research output is inherently experimental. Repeatedly exploring the same problem may succeed on the first try or fail many times. The evaluation system therefore defines two statistical calibers:

    • Unassisted single-attempt success rate: Each problem allows only one attempt, with no intermediate lemmas provided, measuring the model’s ability to “get it right in one shot.”
    • Multi-attempt success rate and coverage: Repeated attempts are allowed within a fixed number, while tracking both the number of successes and the number of distinct problems actually solved, reflecting “how far the model can get given enough budget.”

    The following pseudocode illustrates the relationship between the two calibers:

    1. # Pseudocode: TASKS holds example task names; max_attempts_per_task is the per-task
    2. # attempt cap, not a total attempt count
    3. TASKS = ["proof_01", "construct_01", "code_01"] # Example task names, for structure only
    4. MAX_ATTEMPTS_PER_TASK = 3
    5. def run_evaluation(model, tasks, max_attempts_per_task):
    6. attempts_left = {task: max_attempts_per_task for task in tasks}
    7. solved, records = set(), []
    8. while any(attempts_left[t] > 0 for t in tasks if t not in solved):
    9. for task in tasks:
    10. if task in solved or attempts_left[task] <= 0:
    11. continue # Skip tasks already solved or with no attempts left
    12. answer = model.generate(task.prompt)
    13. ok = verify(task, answer) # Unified verifier: deterministic decision
    14. attempts_left[task] -= 1
    15. records.append((task, ok))
    16. if ok:
    17. solved.add(task)
    18. success_rate = sum(ok for _, ok in records) / len(records) if records else 0
    19. coverage = len(solved) / len(tasks) if tasks else 0
    20. return {"success_rate": success_rate, "coverage": coverage}

    The key point in this code is that max_attempts_per_task is explicitly a per-task attempt cap, not a total attempt count. A statement in an evaluation report like “23 successes out of 60 attempts” should be understood as 60 cumulative generate-verify records under this setting, of which 23 passed verification—not that every problem was fixed at 60 attempts. Whether you adopt a per-task budget or a total budget, success rate and coverage must be tracked separately. Otherwise, a model could repeatedly rack up successes on the same problem and mask its true research ability.

    2.4 Presenting Results: From a Single Score to a Capability Terrain Map

    To illustrate the differences between calibers, this article uses anonymized example data. The task set is treated as 20 problems, and Model A and Model B are illustrative placeholders.

    Statistical caliber Model A (rank-1 example) Model B (rank-2 example) Notes on caliber
    Leaderboard total score 71.9 57.4 Anonymized and illustrative, not a specific leaderboard result
    Unassisted single-attempt success rate 70.0% (14/20) Not disclosed Applies to single-generation caliber
    Multi-attempt success rate Not disclosed 38.3% (23 passes in 60 records) Applies to cumulative multi-attempt caliber
    Multi-attempt coverage Not disclosed 9/20 problems Share of distinct problems solved
    Alphabet reliability task Not solved Completed in all 3 attempts Single-task example

    The other six evaluated agents form an independent group, all scoring below 11.7 in combined performance on the same task set. This group’s data is not merged into the Model A or Model B columns, to avoid having a single column represent different statistical calibers at once.

    This table says more than a single total score. Directly comparing the raw numbers under different attempt budgets is misleading, but combining the single-attempt success rate, the multi-attempt success rate, coverage, and single-task performance still yields a clearer judgment.

    Interpreting the Results: The Gaps Are Clear, but It Is Not a Total Rout

    3.1 The Top Model’s Advantage Comes From “Unassisted Single-Attempt Correctness”

    Without intermediate lemmas, Model A completed 14 of the 20 problems in one shot, a 70.0% success rate. Under the same conditions, the other six agents all scored below 11.7%. This gap shows that the ability to “start from the problem and make independent progress” is a key dimension for distinguishing models’ research levels.

    The results under the multi-attempt caliber are also worth noting. Model B produced 60 generate-verify records, with 23 successes, but covered only 9 distinct problems. That works out to roughly 6.7 records per solved problem on average. The evaluation adopts a “do not repeat solved problems” strategy, which already rules out artificially inflated numbers from deliberate repetition, yet the success rate still exceeds the coverage. This suggests a more reasonable way to evaluate: combine the single-attempt success rate, coverage, and required number of attempts into an overall judgment, rather than looking at a single number.

    3.2 Reversals on Individual Tasks Show the Ability Structure Is Not One-Dimensional

    Not all tasks were completed by the model ranked first overall. On a task about coding-alphabet reliability, Model B completed it in all three attempts, while Model A did not solve it. This is where the value of the multi-attempt mechanism shows: repeated exploration under a fixed budget can sometimes offset the variance of a single generation.

    From this we can conclude that a model’s research ability is a “terrain composed of task types,” not a linear ranking. Tasks related to computation and search favor multiple iterations, while tasks related to formal derivation favor the rigor of a single generation; different models rise and fall across these dimensions.

    3.3 Recheckability Matters More Than a “Nice-Looking Answer”

    The most important takeaway from this evaluation is that a model’s output is not a new conclusion that merely looks plausible, but an object, parameter, or proof that can be independently rechecked. A new construction from a graph structure can be checked edge by edge and pair by pair against its constraints; an improved upper bound on coding parameters can be verified by an independent program checking the minimum distance between codewords, or by checking a proof line by line. The verifier does not care what the model “said”; it only cares whether “the conclusion holds.”

    In effect, this redraws the boundary of AI’s role in research: when a result can be decided by tools, the AI’s output can enter an engineered “generate-verify-adopt” workflow; when it cannot be decided by tools, no matter how fluent the expression, it is only a lead that still needs to be verified.

    3.4 Two Landmark Results: A Construction Improvement and a Tightened Bound

    In the example data, the model ranked second, Model B, produced two representative results:

    1. Extremal graph construction improvement: In a task seeking an extremal graph of girth 5—that is, constructing a graph with no triangles and no quadrilaterals—Model B constructed a graph with 124 vertices and 630 edges, surpassing the 629-edge reference value used by the task. This kind of “break the reference value” result especially warrants caution against data snooping: the construction was verified item by item under fixed constraints, not something that accidentally triggered a check condition during the search.
    2. Tightened coding upper bound: In a task on the upper bound for a code of length 22 and minimum distance 10—commonly written as A(22,10), where n=22 denotes the codeword length, d=10 denotes the minimum-distance constraint, and the A-type notation is often used for the upper bound or known maximum size of codewords under this condition—Model B tightened a previously published upper bound of 84 to 82. This kind of improvement must come with an independent proof or a reproducible verification program; a bare conclusion is not enough.

    Together, these two examples show that under clear constraints and automatable feedback, AI can already complete real combinatorial construction and program optimization, and in some metrics even surpass existing public results. To avoid overextending the claim, this article treats them as illustrative results; real citations should still return to the original evaluation and verifiable records.

    3.5 The Boundaries Remain Clear

    It must be stressed that these results do not mean AI can complete research independently. The evaluation tasks themselves were designed, selected, and given reference values by human researchers; the budget and verification methods were also set by humans. Real open research still requires researchers to make their own judgments: which problems are worth studying; whether an improvement is important; and how to write the result into the field’s body of knowledge. The evaluation results are more like a terrain map than a finish line.

    How Do You Build a Verifiable Agent Loop?

    The methodology of this evaluation can be transferred to enterprise R&D. Teams do not have to wait for AI to “do research on its own”; instead, they can first shore up the verification steps in their work, letting agents iterate under clear feedback.

    4.1 Build the Verifier First, Then Talk About Agents

    Whether an agent can work reliably depends on whether the task is verifiable. It helps to sort R&D tasks by the following levels:

    • L1 Directly verifiable: Unit tests, type checking, static scanning, and data-consistency checks are available. Suitable for handing to agents for autonomous iteration.
    • L2 Partially verifiable: Involves metrics such as performance, compatibility, and security that require specific tools. Define quantitative thresholds first, then hand it to an agent.
    • L3 Hard to verify: Architecture selection, requirement trade-offs, and user experience design. These still require human leadership, with agents only assisting.

    A common mistake is to buy a model first and add verification later. A more sensible order is to define the acceptance criteria first, then choose the model and agent approach.

    4.2 Treat “Budget” as a Designable Variable

    The multi-attempt mechanism in the evaluation gives engineering practice a direct lesson: the success rate of a single generation is not the only metric; the expected output per unit cost is more worth watching.

    Let p be the success rate of a single generation-and-verification attempt, c_gen be the cost of a single generation, and c_verify be the cost of a single verification. If every attempt incurs both generation and verification costs, and the verification cost varies with the number of attempts, then the expected total cost to obtain one result that passes verification can be expressed as:

    Expected total cost = (per-generation cost + per-verification cost) / per-attempt success rate

    For example, a model with a 20% first-try success rate (p=0.2) needs about 5 calls on average to get one verified result; another model with a 5% first-try success rate (p=0.05) needs about 20. Once verification overhead and call costs are both counted, the former is often clearly more cost-effective.

    A note here: this formula assumes every attempt must be verified separately and that verification cost is tied to the number of attempts. If verification can be run in batches or by sampling, or if some checks are run only on the final candidate, then c_verify should be replaced with the actual average verification cost. In engineering, you can first batch-generate candidates, then use the verifier to filter and rank them. This is also a typical application of inference-time compute scaling: trading more samples for a higher final success rate.

    4.3 Prevent “Objective Misalignment”

    The verifier is not omnipotent; it shapes the model’s behavior. The “do not repeat solved problems” rule in the evaluation is, in essence, there to prevent models from inflating their success rate by grinding the same problems. In engineering, watch out for three types of risk:

    • Reward hacking: The model games the check conditions rather than actually solving the problem—for example, hard-coding a return value to make a test pass, or deleting code to make a metric look better. This requires that the tests themselves be reviewed and that constraints such as code coverage and change review be added.
    • Verification blind spots: What the verifier cannot cover is exactly where risk concentrates. For example, “running for 72 consecutive hours without interruption” cannot truly be verified by a single test and needs to be broken down into observable, staged metrics.
    • Data snooping: Once the evaluation set or reference values are exposed to the model, the results become artificially inflated. In engineering, use held-out hidden test sets and rotate them regularly.

    4.4 A Reference Architecture for Engineering Deployment

    Bringing the above ideas down to the system level, you can form a generate-verify-iterate pipeline:

    1. Task input (requirements / defects / optimization goals)
    2. │
    3. ▼
    4. ┌────────────────────────┐ Candidate results
    5. │ Agent generation service │──────────────┐
    6. │ (containerized) │ ▼
    7. └────────────────────────┘ ┌─────────────────────────┐
    8. ▲ │ Verifier cluster │
    9. │ failure reasons │ unit tests / compile │
    10. │ and logs │ static checks │
    11. │ │ perf benchmarks / │
    12. │ │ consistency checks │
    13. │ └─────────────────────────┘
    14. │ │ pass
    15. │ ▼
    16. └────────────── Result archiving and human review

    Each stage can be built on mainstream cloud platforms or cloud service providers; the specific product choices need to be verified against actual needs. Common capabilities include:

    • Container platforms: Run isolated sandboxes and cap the resources available to agents, preventing generated code from affecting the external environment.
    • Object storage: Save inputs, candidate results, and verification reports to keep results traceable.
    • Message queues: Split generation and verification into asynchronous tasks to smooth out peaks when task volume is high.
    • Logging services and monitoring/alerting: Record the time and failure reason for each attempt, catching stages where problems concentrate in time.
    • Elastic compute: Provide on-demand, scalable compute for multi-attempt and multi-model parallelism.

    The key to building this loop is not which platform you use, but whether you build the verification step solidly.

    4.5 Cost and the Human-Machine Division of Labor

    The cost of the multi-attempt mechanism is that call costs grow with the number of attempts, so budget control needs to be done alongside the expected-cost model. For example, cap the number of attempts for simple tasks, raise the budget for difficult tasks, and treat budget allocation itself as an optimization problem.

    At the same time, the human role needs to be repositioned. In the evaluation, human researchers design the tasks, select the problems, and set the reference values; agents handle search and construction within the constraints. In an enterprise, this division can be reflected as:

    • Engineers define the acceptance criteria and the verifiers;
    • Agents iterate quickly under verification feedback;
    • Engineers review the final output and make decisions on the parts that cannot be verified automatically.

    A Checklist of Common Misuses

    When drawing on verifiable evaluations, teams should also avoid the following misuses:

    • Looking only at the single-attempt success rate: This overlooks a model’s true ceiling under multiple attempts, model combinations, or more samples.
    • Looking only at the leaderboard total score: Scores across different budgets, task sets, and verifier implementations are not directly comparable.
    • Looking only at the multi-attempt pass rate: Without checking coverage, results can be distorted by repeated successes on the same problem.
    • Ignoring verification blind spots: Metrics a verifier does not cover should not be assumed safe by default.
    • Overlooking data snooping: Once the evaluation set, reference values, or test cases are exposed, results may be inflated.
    • Treating examples as conclusions: Anonymized and illustrative data can only help with understanding the method; it cannot replace a formal evaluation report.

    Limitations and Outlook

    The results of this evaluation still have clear boundaries. All tasks have well-defined mathematical definitions and automated verification, so the model does not need to judge whether a problem is meaningful, nor decide what to research next. Once you enter fields with fuzzy definitions, hard-to-verify outcomes, and a need for long-term accumulation, the applicability of current capabilities drops noticeably.

    In addition, conclusions from a single leaderboard must be interpreted with care. A model’s relative ranking changes with version updates, budget settings, and verifier implementations; between “completing a task once” and “using it reliably in production,” there is still a series of issues around engineering, observability, and cost control.

    Three directions are foreseeable:

    • Continual expansion of the task set: From pure math and theoretical computation to more industry-relevant fields such as scientific computing, materials screening, and chip design.
    • Automatic verifier generation: Researching how to derive check conditions automatically from requirement descriptions, lowering the cost of building verifiers by hand.
    • Combining formal and natural language: Letting the rigor of proof checkers and the generalization ability of large models complement each other.

    Conclusion

    A verifiable research-task benchmark does not give the simple answer of “who is the strongest,” but rather a terrain map of AI’s research capabilities. In regions where problems are well-defined, budgets are sufficient, and feedback is verifiable, models can already produce proofs, optimize programs, and improve constructions, with some results even surpassing reference values; in regions of autonomous problem selection, value judgment, and tackling major open problems, human researchers must still lead.

    For developers and enterprise teams, the more pragmatic approach is to first turn key tasks into verifiable tasks, then let agents keep iterating under verification feedback. This path does not depend on any single model’s ranking; it is built on engineering methods and a verification system, and it is more likely to deliver stable, sustainable returns.