Verifiable Evaluation of LLM Research Capabilities: From Solving Problems to Completing Research Tasks
Standardized question-answering benchmarks are often insufficient for assessing whether large language models can support real research work. This article argues that research requires a chain of hypothesis generation, method design, experimentation, verification, and revision, and therefore proposes a verifiable evaluation framework that turns model outputs into executable, checkable, and comparable technical objects. It covers three core task categories: theorem proving, program optimization, and constructive search. It also explains why evaluation should report capability profiles rather than only aggregate scores, outlines a practical evaluation pipeline, discusses current capability boundaries, and provides recommendations for teams building their own model evaluation systems. The overall goal is to help organizations assess model behavior more objectively and build a more reliable foundation for automated research assistance.
1. From Solving a Problem to Completing a Research Task
In the past, large model evaluation often focused on standardized questions: the model was given a clear prompt, produced an answer, and was then compared against a reference result by humans or automated systems. This kind of evaluation can clearly show a model’s memory, reasoning, and code generation abilities, but it does not fully cover the complexity of real research activities.
Research is not the answer to a single isolated question. It is a sequence of judgments, constructions, verifications, and revisions. A researcher needs to propose hypotheses, design methods, choose experiments, analyze results, and iterate based on feedback. Even when facing a formalized problem, a model must complete multiple steps: understanding constraints, retrieving knowledge, generating candidate solutions, running verification, identifying failure causes, and providing a reviewable conclusion after multiple attempts.
Therefore, judging whether a large model has research capability cannot rely only on whether it gives a “seemingly reasonable” answer. It is more important to examine whether the model can produce an object, process, or result that can be independently reviewed.
For example:
- In theorem-proving tasks, a model should not only output text saying that a proof is complete; it should provide proof steps that can be checked by a verifier.
- In program optimization tasks, a model should not only claim performance improvement; it should provide runnable code, benchmark data, and resource consumption information.
- In constructive search tasks, a model should not only describe an idea; it should provide concrete parameters, structures, or counterexamples.
This is the core value of a verifiable evaluation system: it converts model output from natural-language claims into executable, checkable, and comparable technical objects.
2. Three Key Dimensions of a Verifiable Evaluation System
To assess whether large models are approaching real research scenarios, an evaluation system should cover at least three dimensions: proof, optimization, and construction. These task types represent different capability aspects of research work.
2.1 Proof Tasks: Logical Rigor and Completeness
Proof tasks are a core scenario in theoretical research. The model must work from a set of premises, definitions, and goals, and generate a reasoning chain. The difficulty lies in the fact that natural-language proofs may contain leaps, ambiguity, or unstated assumptions. Therefore, formal or semi-formal verification should be introduced as much as possible.
A reliable proof evaluation usually checks whether:
- the problem definition is complete;
- intermediate lemmas are clearly stated;
- reasoning steps are continuous;
- there is no circular reasoning;
- no conditions outside the given problem are assumed;
- the proof can be checked by an external verifier or rule engine.
For complex theorems, the model can also be required to introduce intermediate lemmas on its own. Real research is rarely completed in one step. Researchers often need to break a large problem into smaller subproblems that can be proved individually. The evaluation system can set different difficulty levels: whether the model can complete the task without intermediate lemmas; whether it can correct errors after receiving a small amount of guidance; and whether it can cover more target theorems across multiple attempts.
2.2 Optimization Tasks: Engineering Capability and Quantifiable Improvement
Program optimization tasks are closer to software engineering and algorithm research. The model needs to understand an existing implementation, propose improvement strategies, and ensure that the optimized result is runnable, measurable, and comparable.
The key is not simply “generating a piece of code,” but forming a complete closed loop:
- Read the original program or algorithm specification.
- Locate performance bottlenecks.
- Propose an optimization plan.
- Write a runnable implementation.
- Execute it in a unified environment.
- Record execution time, memory usage, correctness, and stability.
- Continue revising based on failure logs.
The evaluation system needs to ensure a consistent test environment. Otherwise, results may become incomparable due to different hardware, dependencies, or compilation settings.
For performance optimization, it is best to use multiple metrics, such as execution time, peak memory usage, throughput, error range, resource consumption, and code readability. If only one metric is considered, a model may produce a “local optimization” that improves one indicator while damaging overall quality.
2.3 Construction Tasks: Search Capability and Constraint Satisfaction
Construction tasks are common in graph theory, combinatorial optimization, algebraic structures, and experimental design. The model must find an object that satisfies given conditions under complex constraints. For example, it may need to construct a graph, matrix, set, or parameter combination with specific properties.
The advantage of these tasks is that results are usually easy to verify: as long as the constraints are given, a verifier can determine whether the construction is valid. However, the challenge is that the search space can be extremely large. The model therefore needs strategic exploration rather than simple enumeration.
A high-quality construction task evaluation should include:
- an explicit constraint set;
- an executable legality checker;
- a quality scoring function;
- multiple success thresholds;
- a multi-attempt mechanism;
- classification of failure cases.
The model can provide a concrete construction object, such as vertex count, edge count, parameter values, structure description, or a generation script. The evaluation system then checks whether all constraints are satisfied and records whether the result improves upon reference values.
3. Evaluation Should Show Capability Profiles, Not Only Total Scores
When comparing multiple models, total-score rankings provide an intuitive reference, but they can easily hide capability differences. Real research capability is not a single-dimensional strength or weakness. It is a performance distribution across different task combinations.
For example, one model may perform strongly on theorem-proving tasks but have limited advantages in program optimization or constructive search. Another model may have a high success rate on single tasks but cover fewer problems. Some models may have a lower overall score yet can complete difficult results that other models cannot.
Therefore, evaluation reports should present detailed dimensions rather than only a single aggregate score.
At minimum, the following metrics can be separated:
- Completion rate: the proportion of tasks where the target result is produced;
- Success rate: the proportion of all attempts that eventually pass verification;
- Coverage: the number of different problems or goals the model can address;
- Stability: whether results fluctuate across repeated experiments;
- Budget efficiency: the time, compute, and call cost required to obtain one valid result;
- Correctability: whether the model can revise its output based on failure feedback;
- Reviewability: whether the output contains enough information for humans or machines to verify it.
It is also important to distinguish between “success in one attempt” and “success after multiple attempts.” Some tasks allow multiple rounds of iteration, which is closer to the trial-and-error process in real research. If an evaluation only shows the final result, it may overestimate the model’s reliability in real scenarios. The evaluation system should clearly record the number of attempts, failure reasons, and convergence process.
4. A Practical Evaluation Pipeline Design
If a team wants to build a similar evaluation system, it can start with a small pipeline and gradually expand it into a private task library.
A basic process can be designed as follows:
def evaluate_task(task, model, verifier, max_attempts=3):history = []for attempt in range(max_attempts):prompt = task.build_prompt(attempt, history)answer = model.solve(prompt)result = verifier.check(task, answer)history.append({"attempt": attempt,"answer": answer,"passed": result.passed,"score": result.score,"error": result.error})if result.passed:return historyreturn history
In this process, task defines the problem and prompt strategy, model generates the answer, and verifier checks whether the answer is valid. The key is not the model itself, but whether the verifier is sufficiently strict, whether the task is sufficiently clear, and whether the logs are sufficiently complete.
To support more complex tasks, the evaluation system can also introduce the following modules:
- Sandboxed execution environment: ensures that programs run in isolation and untrusted code does not affect the host machine;
- Version management: records model versions, task versions, verifier versions, and configuration parameters;
- Metric storage: writes scores, execution time, and failure reasons from each run into a database;
- Monitoring and alerting: sends alerts when task success rates drop abnormally or verifiers report errors;
- Visualization dashboard: shows capability distributions across different tasks, models, and budgets.
If an organization wants to build such a system on cloud infrastructure, it can use general-purpose components, such as object storage for experimental artifacts, container platforms for sandbox execution, logging services for tracing execution processes, and monitoring and alerting for observing resource consumption and task failure rates. Cloud platforms such as Baidu AI Cloud often provide similar foundational capabilities and can serve as a general base for building experimental platforms.
5. Capability Boundaries in Real Evaluations
The most important value of verifiable evaluation is not to prove that one model is comprehensively superior. Its value is to help teams understand the capability boundaries of models.
Under conditions of clear problem definitions, sufficient budgets, and verifiable feedback, large models can already complete a considerable amount of research assistance work, such as:
- proving some theorems;
- generating candidate algorithms;
- optimizing local code;
- constructing examples that satisfy constraints;
- searching for improvable parameters;
- machine-verifying existing results.
However, research also contains more complex parts. At present, models still find it difficult to independently take on the following responsibilities:
- judging whether a research direction is worth investing in;
- evaluating whether a result has academic value;
- discovering truly original open problems;
- building research frameworks under vague goals;
- making comprehensive judgments about research ethics, social impact, and academic norms.
This means that large models are more like efficient research execution assistants than fully autonomous research subjects. They can accelerate verification, search, and implementation, but research topic selection, value judgment, and accountability still need to be completed by human experts.
6. Enterprise Recommendations: Start Narrow and Build Trusted Evaluation Step by Step
For developers and technical teams, building a large-model research evaluation system should not begin with an overly broad or comprehensive design. A more reliable path is to start with narrow tasks and gradually establish task libraries, verifiers, and feedback mechanisms.
6.1 First Choose Tasks with Verifiable Outcomes
Teams should prioritize tasks with clear inputs, checkable outputs, and clear success criteria. For example:
- Given constraints, check whether a program’s output meets the specification;
- Given functional properties, verify whether a mathematical conclusion holds;
- Given performance metrics, compare differences before and after optimization;
- Given a construction goal, check whether an object is valid.
It is better not to design open-ended research tasks at the beginning. Such tasks are difficult to score automatically and may introduce subjective judgment.
6.2 Build Traceable Logs
Every model attempt should record complete context, including the task description, prompt, model output, execution result, verification log, and failure reason. This not only helps reproduce experiments, but also makes it easier to analyze where the model failed.
6.3 Distinguish Different Budget Levels
Research tasks usually require multiple rounds of attempts. The evaluation system should support different budget settings, such as low budget, high budget, long-chain reasoning, and multi-round feedback. Through comparison, teams can observe whether models remain usable under low-cost conditions and whether increasing the budget brings clear benefits.
6.4 Introduce Human Review
Automated verification can improve efficiency, but it cannot fully replace human review. For proof tasks, experts can perform spot checks. For optimization tasks, code quality and maintainability can be reviewed. For construction tasks, evaluators can assess whether the result has potential for further extension.
6.5 Treat Failure Samples as Assets
Failure cases are often more valuable than success cases. Teams should classify the reasons for model failures, such as constraint misunderstanding, verifier misjudgment, prompt ambiguity, execution environment errors, or model hallucination. By managing failure samples, teams can continuously improve task design, verifiers, and prompt strategies.
7. Future Forms of Research Agents
As evaluation systems mature, the way large models participate in research will gradually move from single-turn question answering to multi-stage collaboration. Future research agents may take on roles such as:
- Literature review assistant: summarizing existing methods and extracting reusable hypotheses;
- Experiment design assistant: generating candidate experiment plans and control settings;
- Code execution assistant: running scripts, collecting metrics, and locating anomalies;
- Proof checking assistant: identifying reasoning gaps and generating counterexamples or revision suggestions;
- Result review assistant: cross-validating model outputs;
- Research record assistant: organizing experiment logs and forming reproducible reports.
In these roles, verifiers and task libraries will become key infrastructure. The reliability of model output will no longer depend on whether the natural-language expression is fluent. It will depend on whether the output can be executed, checked, and reproduced by the system.
8. Conclusion
Solving competition problems does not mean that a large model already has complete research capability. Competition problems usually have clear goals and scoring criteria, while real research also includes problem selection, value judgment, open exploration, and responsibility.
The significance of a verifiable evaluation system is precisely to break down these complex activities into executable, checkable, and reproducible subtasks, so that model capabilities can be observed more objectively.
From a practical perspective, teams should focus on the three task categories of proof, optimization, and construction. They should build an evaluation system that combines strict verification, sufficient logging, and multidimensional metrics.
In this way, large models can take on more verifiable research work when problems are clearly defined, while human researchers can focus on direction judgment, problem definition, and result value assessment. This division of labor will not weaken the importance of research. Instead, it will improve research efficiency and help teams build more robust, trustworthy, and automated research capabilities.
