Verifiable AI Research Evaluation: From Competition Problem Solving to Scientific Research
This article explains why answering contest-style questions does not prove that a large model can conduct research. It proposes a verifiable evaluation system that separates competition problem solving, semi-open research tasks, and open research judgment; outlines design principles such as object-based submissions, independent verification, explicit budgets, traceable processes, contamination resistance, and complementary tasks; and describes task families, metrics, architecture, verifier implementation, engineering deployment, anti-cheating measures, and human-AI collaboration. Readers learn how to turn the gray area between solving problems and doing research into observable, reviewable, and improvable engineering evaluations.
Large models can solve competition problems. Does that mean they can conduct research? The answer is not simple. Contest problems usually have clear boundaries, standard answers, and short reasoning chains. Research activities include problem selection, background understanding, hypothesis generation, proof or experiment design, result review, value judgment, and other stages. A model may complete a local proof, program optimization, or construction improvement within a limited scope, but that does not automatically mean it can conduct research independently. A more reasonable approach is to place key outputs from the research workflow into one verifiable system and use tasks, budgets, submission objects, independent verifiers, and process metrics to measure capability boundaries.
1. Boundaries Between Competition Problem Solving and Research Capability
Competition problem solving emphasizes finding an answer to a known problem, and the evaluation standard is relatively simple. Research emphasizes “proposing new objects that can be reviewed.” For example, a mathematical proof requires every step to be checkable; a construction problem requires submitting a legal object and verifying its properties; program optimization requires repeatable execution results; parameter improvement requires independent recomputation. A model should not provide only a “plausible new answer,” but a proof, construction, program, or parameter setting that can be independently reviewed.
Therefore, evaluation design should distinguish three types of capability. The first is closed problem solving, such as generating answers to standard problems. The second is semi-open research tasks, such as completing a theorem proof without intermediate lemmas or improving a construction within a certain budget. The third is open research judgment, such as choosing research directions, judging the value of results, and deciding whether to continue investment. Current systems are better suited to evaluating the first two types; the third still depends on human researchers.
2. Design Principles for a Verifiable Evaluation System
A verifiable evaluation system should follow these principles:
- Object-based submissions: Require models to submit checkable objects such as proof scripts, graph structures, program patches, and parameter vectors, rather than only natural-language conclusions.
- Independent verification: Separate the verifier from the generator and use deterministic checks, formal checks, test cases, or mathematical recomputation wherever possible.
- Explicit budgets: Limit the number of attempts, runtime, token budget, or computing resources to avoid hiding capability gaps through unlimited trial and error.
- Traceable processes: Record the input, output, failure reason, verification log, and resource consumption of each attempt.
- Contamination and overfitting prevention: Use hidden tests, random seeds, variant tasks, and submission signatures to prevent models from memorizing evaluation data.
- Complementary tasks: Different tasks test different capabilities; a single total score cannot support claims of across-the-board superiority.
3. Task Spectrum and Metrics
The task spectrum can include theorem proving with or without intermediate lemmas, constructive extremal problems, program optimization, reliability verification, upper-bound tightening, parameter search, and other task types. The metric system should not rely only on success rate. It should also include the number of theorems covered, number of attempts, proof reviewability rate, construction validity, improvement magnitude, budget consumption, and failure-type distribution.
As an abstract observation from a set of public evaluation samples: the top system scored about 71.9 in total, and the second about 57.4. Without intermediate lemmas, the leading system completed 14 of 20 theorems, a 70.0% success rate. Another system succeeded 23 times in 60 attempts, covering 9 different theorems, a 38.3% success rate. The remaining six agents each did not exceed 11.7%. This data shows that the capability distribution is not simply linear. One system completed all three attempts on an alphabet reliability task for probabilistically checkable proofs, while the leading system did not solve that problem. In an extremal graph construction task with girth 5, one system constructed a graph with 124 vertices and 630 edges containing no triangles or quadrilaterals, exceeding the reference value of 629 edges. Another system proved A(22,10) ≤ 82, tightening the published upper bound from 84 to 82. These examples show clear complementarity on local tasks.
4. System Architecture and Verifier Implementation
A practical evaluation system usually contains six parts: a task library, an execution sandbox, a submission repository, a verifier cluster, metric aggregation, and human review. The task library stores problem descriptions, allowed lemmas, budgets, and verification rules. The execution sandbox isolates model execution. The submission repository stores proofs, constructions, programs, and other objects. The verifier cluster performs independent checks. Metric aggregation generates leaderboards. Human review handles edge cases.
def evaluate_submission(task, agent, budget):submission = agent.solve(task.prompt, budget)assert submission.kind in {"proof", "construction", "program", "parameter"}if submission.kind == "proof":ok = verify_proof(task.statement, submission.proof, task.allowed_lemmas)elif submission.kind == "construction":ok = verify_properties(task.spec, submission.object)elif submission.kind == "program":ok = run_hidden_tests(submission.code, task.test_suite)else:ok = verify_parameter_bounds(task.constraints, submission.value)metrics = {"valid": ok,"budget_used": submission.budget_used,"attempts": submission.attempts,"failure_type": classify_failure(submission, ok)}log_result(task.id, agent.id, metrics)return metrics
Verifiers should be as independent as possible: use proof checkers or formal kernels for mathematical proofs, property checkers for construction problems, hidden tests and performance benchmarks for program optimization, and interval recomputation for parameter problems. For natural-language proofs, a hybrid approach of “formalizing key steps plus human spot checks” can help avoid mistaking fluent language for a correct proof.
5. Engineering Deployment and Anti-Cheating
For engineering deployment, a general-purpose container platform can provide isolated execution; object storage can store submission objects and logs; a message queue can schedule verification tasks; and logging and monitoring/alerting can record resource consumption and failure reasons. For example, on a general-purpose cloud platform, you can select the relevant general cloud service categories for deployment without relying on proprietary interfaces. For anti-cheating, restrict external network access, randomize test cases, hash-sign submission objects, keep complete attempt records, and manually review unusually high scores.
6. Human-AI Collaboration and Organizational Recommendations
A verifiable system is more like a “capability terrain map” than a replacement for research. When organizations use such a system, they can let models generate candidate proofs, constructions, and optimization plans, while humans handle topic selection, value judgment, and final review. When evaluating value, consider four dimensions: novelty, impact, verifiability, and cost. Common failure modes include “plausible but not reviewable,” “overfitting to the evaluation,” “budget abuse,” and “hidden proof gaps.” Human spot checks and failure-classification reviews should be built into the process.
In short, models can already complete some proofs, program optimizations, and construction improvements when problems are clear, budgets are sufficient, and feedback is verifiable. However, autonomously choosing research directions, judging the value of results, and solving major open problems still require humans. The value of building a verifiable evaluation system is to break the gray area between “can solve problems” and “can do research” into observable, reviewable, and improvable engineering problems.
