LLMs for Research: Using Verifiable Evaluation to Clarify Capability Boundaries

    Competition benchmarks measure performance within predefined constraints, not full research capability. This article explains why high scores can be misleading, why the research workflow should be divided into verifiable stages, and how concrete objects, parameters, and proofs can be independently checked, illustrated with a graph example.

    High Competition Scores Do Not Equal Strong Research Ability

    Competition problems have clear statements, standard answers, and a fixed solution space; evaluation only checks whether the result is correct. Research questions often start from a vague phenomenon. Researchers must first articulate the problem, then choose methods, and along the way judge whether a route is worth further investment. Three gates separate competition from research: formulating the question, selecting a route, and judging the value of results.

    Evaluating models only on competition problems tends to produce optimistic conclusions. A high score shows that the model can solve problems within a given framework, but it does not directly imply that the model can independently choose a topic, identify dead ends, or judge whether a result is valuable. A more reliable approach is to break the research workflow into stages and test model performance at each stage.

    This article discusses “verifiable evaluation,” which measures the ability to produce checkable outputs; it is not equivalent to full research capability. Direction selection, result judgment, and open problems require separately designed evaluations or judgment by domain experts.

    How Research Outputs Can Be Independently Verified

    Research outputs fall into three broad categories:

    1. Objects: specific constructions such as graphs, matrices, combinatorial structures, or algorithm instances.
    2. Parameters: an upper bound, constant, threshold, or optimal configuration.
    3. Proofs: a chain of logical reasoning that can be formally checked or reviewed step by step by experts.

    The first two categories can usually be checked directly. A checker can traverse all edges to confirm whether a graph satisfies a constraint, or substitute parameters to confirm whether a threshold is met.

    Consider a concrete example. Suppose a model claims: “The graph has no triangles, and its minimum cycle length is exactly 5.” It provides the following adjacency list:

    1. Vertex 1: [2, 5]
    2. Vertex 2: [1, 3]
    3. Vertex 3: [2, 4]
    4. Vertex 4: [3, 5]
    5. Vertex 5: [4, 1]

    The checking procedure is straightforward. The first step is to enumerate all triples to confirm that no triangle exists.