From Problem Solving to Research: How to Evaluate Large Models' Research Capabilities Verifiably
Large language models (LLMs) excel at solving closed-ended problems but often struggle with open-ended research tasks. This article explains why test scores alone do not reflect true research ability, and outlines a practical framework for designing verifiable, objective evaluations of model research outputs. It covers key principles for building evaluation tasks, how to measure success across multiple attempts, and how to interpret results without overgeneralizing. The article also discusses current limitations of LLMs in autonomous research and provides concrete deployment tips for running large-scale evaluations in the cloud.
I. The Starting Point: Problem-Solving Ability Is Not Research Ability
Competition problems typically come with complete problem statements, standard answers, and well-defined scoring rules. Research problems, by contrast, have no ready-made answers: a researcher must first choose a topic, then model it, try multiple approaches, and finally judge whether the result has value. The fundamental difference lies in the openness of the process. Using only competition problems as evaluation benchmarks measures a model’s reasoning on closed problems, but does not reveal whether it can handle other stages of the research workflow.
A complete research workflow generally includes the following steps: propose a testable hypothesis, design a construction or proof path, verify intermediate conclusions, correct errors, and finally produce a deliverable result. Each step has its own failure modes. Therefore, evaluation must cover not just the final answer, but also whether each intermediate step can be independently checked.
II. Making Research Outputs Verifiable
The core of evaluation is verifiable feedback. A trustworthy research output should satisfy three conditions:
- Concrete outputs: The output consists of specific objects—such as graphs, sequences, parameters, or proof texts—accompanied by checkable evidence.
- Independent rechecking: The checking program can determine correctness without relying on the model’s own description.
- Comparable results: Given the same budget and constraints, outputs from different systems can be compared side by side.
Take the example of “constructing a graph that satisfies specific forbidden substructure constraints.” The task requires outputting the number of vertices and an edge list. The verification program checks whether the graph contains forbidden short cycles and counts the edges. This kind of evaluation does not depend on subjective judge opinions, and results are reproducible. Below is a sample verification script that detects triangles and quadrilaterals (3-cycles and 4-cycles):
from itertools import combinationsdef has_c3_or_c4(n, edges):"""Check whether an undirected simple graph contains a cycle of length 3 or 4."""adj = {i: set() for i in range(n)}for u, v in edges:adj[u].add(v)adj[v].add(u)# Triangle: an edge (u, v) exists, and u and v share a common neighborfor u, v in edges:if adj[u] & adj[v]:return True# Quadrilateral: two neighbors b, c of vertex a share another common neighbor d (d != a)for a in range(n):for b, c in combinations(adj[a], 2):for d in adj[b] & adj[c]:if d != a:return Truereturn Falsedef verify_candidate(n, edges, target_edges):if has_c3_or_c4(n, edges):return "Constraint violated"if len(edges) >= target_edges:return f"Passed, edge count {len(edges)}"return f"Target not reached, edge count {len(edges)}"
This script only handles simple graphs; if the input contains duplicate edges or self-loops, a preprocessing step for deduplication and validity checking is required.
III. Four Principles for Designing Evaluation Tasks
1. The problem can be clearly stated, but the answer cannot be directly derived from the problem statement. The task must be hard enough to distinguish real capability from pattern matching.
2. Budgets must be uniform. Limits on time, number of calls, and computational resources should all be fixed. Results obtained under different budgets are not comparable.
3. Allow multiple attempts and record success rates. Generative systems have randomness; a single result is not sufficient. It is recommended to record the number of attempts and successes per task, calculate the success rate as “successful attempts divided by total attempts,” and also report the number of subtasks covered.
4. Distinguish between “completed” and “correctly completed.” Beyond pass/fail, also record failure reasons, whether human hints were needed, and whether the output passed independent checks.
IV. How to Interpret Evaluation Results
A single ranking can conceal important details. A system that leads overall may not lead on every task type. For example, one system may have a significantly higher success rate on a batch of theorem-proving tasks, while another system completes all three attempts on a certain reliability task. This indicates that the capability distribution is not uniform. It is recommended to report results broken down by task type:
- Proof-type tasks: Check whether a complete, step-by-step verifiable derivation is provided.
- Construction-type tasks: Check whether the object satisfies all constraints and whether the metric exceeds the reference value.
- Optimization-type tasks: Check whether the improvement can be reproduced by a program on the same inputs.
Additionally, experimental settings change the meaning of conclusions. If intermediate lemmas are not provided, what is being measured is mainly the model’s ability to derive proof independently. If auxiliary materials are provided, the measurement reflects the ability to exploit existing resources. Results from these two settings cannot be directly added or compared.
V. Current Capability Boundaries
Under well-defined problems, sufficient budgets, and verifiable feedback, current models can already complete some proofs, program optimizations, and construction improvements. However, selecting research directions, judging the value of results, and attacking major open problems still require human researchers to lead the effort.
Evaluation conclusions should not be over-extrapolated. Success on a batch of closed problems does not imply that the model possesses open-ended research capability. “Plausible-looking answers” also require caution: a well-formed answer does not equal a correct conclusion. Independent verification is essential.
VI. Practical Implementation Recommendations
- Break down tasks: Decompose research-style problems into several verifiable subtasks, each with an independent checking program.
- Execute in parallel: For large-scale evaluations running in the cloud, use a general-purpose container platform for parallel scheduling, store output files in object storage, and use logging services and monitoring alerts to track failure rates and time anomalies.
- Keep full traces: Save the input, output, and verification result of every attempt to enable experiment reproduction and post-hoc audits.
- Perform spot checks: Automatically verified conclusions still need human sampling, especially for parts that require mathematical explanation or domain judgment.
