From Solving Problems to Doing Research: How to Evaluate the Research Capabilities of LLMs

    This article challenges a common misconception in AI evaluation: that strong performance on math benchmarks means a model can do research. It explains why leaderboard scores measure problem-solving rather than research ability, and why theoretical computer science (TCS) offers a practical way forward—most of its results can be machine-verified through formal proofs. The article outlines how to design a research benchmark that maps typical TCS activities into scorable tasks, how to keep the evaluation rigorous through contamination checks, proof verification, and difficulty calibration, and how to build a reproducible evaluation system on cloud infrastructure. The key takeaway: measuring AI should go beyond chasing scores and focus on whether the reasoning is rigorous and the conclusions are verifiable.

    1. A Question That Won’t Go Away: Does Being Good at Math Mean Being Good at Research?

    Lately, claims like “AI has made mathematics obsolete” keep resurfacing on social media. These discussions attract attention because math happens to be one of the domains where large language models (LLMs) most easily “score points.”

    The reason is straightforward: math problems usually have a single, definite answer, so right and wrong are obvious at a glance. It resembles solving Sudoku—filling in the cells is hard, but once they’re filled, checking them one by one immediately reveals whether the solution is correct. This “automatically checkable” property is what makes math the least controversial showcase for assessing LLM capabilities.

    Yet treating “solving problems” as the same thing as “doing research” is an implicit logical substitution. To see the issue clearly, we first need to distinguish the essential difference between doing problems and doing research.

    2. Why Benchmark Scores Can’t Measure Research Ability

    In today’s evaluation landscape, leaderboards are refreshed regularly and scores keep climbing. But these results only show that a model “can solve problems”—they say little about its actual “research level.” The gap comes from three dimensions:

    • Whether the steps are already known: Solving problems is like cooking from a recipe—inputs, steps, and timing all follow established guidelines, and “standard-answer” solution templates may even exist. Research has no ready-made process; the path must be carved out by the researcher, and no one knows in advance whether it will lead anywhere.
    • Whether the answer is already known: Problem-style tasks usually have a correct answer that exists beforehand. Genuine research questions, by contrast, sometimes have answers that even the person posing them doesn’t know—they may remain open and unresolved.
    • Whether mistakes are forgiving: Dropping chocolate into sweet-and-sour pork would be spotted instantly, but a “seemingly sound yet actually wrong” step in research is extremely hard to detect—a single logical flaw can invalidate an entire argument.

    A tougher paradox follows: the closer a problem gets to real research, the harder it is to score automatically. Open-ended problems have no single answer, so grading often depends on subjective judgment and resists scaling or standardization. This is precisely why evaluating AI research ability has long lacked a widely accepted, reliable benchmark.

    Theoretical computer science, as it turns out, offers a way out of this bind.

    3. Theoretical Computer Science: Why It Fits Research Evaluation

    Theoretical computer science (TCS) sits at the intersection of mathematics and computer science. Every problem centers on “computation” itself, and the primary research method is rigorous proof.

    A useful analogy: engineers are like drivers who care about implementation details—the engine, the interior, which framework to use, how to tune it. TCS is more like studying the entire transportation system—how fast one can travel from point A to point B in principle, and which routes are fundamentally impassable. It isn’t concerned with any single vehicle’s construction, but with the boundaries and limits of computation itself.

    The key reason TCS makes an ideal evaluation ground is a single property: most of its results can be machine-verified. The core output of TCS is formal propositions together with their proofs, and the correctness of a proof can be checked automatically or semi-automatically using formal proof systems or verification scripts. This gives us:

    1. Objective grading criteria that don’t rely on subjective expert scoring;
    2. Reproducible results—anyone who reruns the same proof reaches the same conclusion;
    3. Scalability—there’s no need to hand-write scoring rules for every problem.

    It is exactly this “machine-verifiability” that lets TCS reconcile the tension between “problems should resemble real research” and “answers must be gradeable automatically.”

    4. Designing the Benchmark: Turning Research Activities into Testable Tasks

    A high-quality research benchmark must abstract “what theoretical computer scientists actually do” into operational, scorable task types. Consider a benchmark of roughly 100 problems organized into four categories. Its core idea is to make each problem correspond to a typical activity in TCS research. These activities fall broadly into:

    • Designing algorithms (proving upper bounds): Given a problem, construct a method and show how fast it runs under specified resource constraints. This measures “how quickly the destination can be reached.”
    • Proving lower bounds (arguing impossibility): Show that a problem cannot, in principle, be solved below a certain cost—“this route is destined to fail.” This is the most challenging direction in TCS.
    • Reductions and complexity classification: Through transformations between problems, determine where a problem sits in terms of difficulty, or prove that it is equivalent to another known problem.
    • Structural analysis: Characterize and prove key properties of objects such as algorithms, graphs, and Boolean functions, providing support for subsequent results.

    This categorization covers the main fronts of TCS research, and because the output of each task ultimately reduces to “whether a specific proposition holds,” automatic grading becomes feasible.

    5. Ensuring the Rigor of the Evaluation

    A credible research evaluation can’t rely on the number of problems alone—it also depends on anti-cheating measures and the grading mechanism. In practice, several issues deserve attention:

    • Guarding against data contamination: An LLM’s training data may already contain public problems and their solutions, leading to “memorization” rather than “problem-solving.” When building the benchmark, prefer newly constructed or long-unpublished problems, and check model outputs for similarity against known solutions.
    • Verifiable proofs: For proof-based tasks, require the model to produce formal or structured proofs, then run them through a verification pipeline that checks whether each reasoning step holds—rather than grading solely on whether the conclusion is correct.
    • Difficulty calibration: Problem difficulty should be tested and calibrated by human experts, removing extremes where “everyone can solve it” or “no one can,” so that scores genuinely discriminate between models.
    • Weighing both process and result: Look at the final answer, but also scrutinize whether the reasoning chain is rigorous—so that “a correct conclusion reached through faulty reasoning” isn’t mistaken for genuine capability.

    The evaluation loop can be described with the following illustrative pseudocode:

    1. def run_benchmark(model, tasks, verifier):
    2. results = []
    3. for task in tasks:
    4. # 1. Call the model to generate an answer (including proof or construction)
    5. answer = model.generate(task.prompt)
    6. # 2. Check the result: conclusion + process
    7. conclusion_ok = verifier.check_conclusion(answer, task.ground_truth)
    8. proof_ok = verifier.check_proof(answer.proof) if task.requires_proof else True
    9. # 3. Record the category for later per-dimension analysis
    10. results.append({
    11. "task_id": task.id,
    12. "category": task.category, # Maps to one of the four research activities
    13. "conclusion_ok": conclusion_ok,
    14. "proof_ok": proof_ok,
    15. "passed": conclusion_ok and proof_ok,
    16. })
    17. return aggregate_by_category(results)

    The crucial design here is that the scoring function is deterministic and replayable, and check_proof inspects the process so that “faulty process with a correct conclusion” cannot slip through.

    6. Engineering a Reproducible Evaluation System

    From an engineering standpoint, a rigorous research evaluation system needs stable compute scheduling, reliable storage, and complete traceability. Building one can draw on mature cloud infrastructure:

    • Container platform: Package model inference and the verification pipeline into standardized images, keeping the evaluation environment consistent and elastically scalable to absorb compute spikes during concentrated runs.
    • Object storage: Centrally host the problem set, model outputs, and verification logs, making version management and result traceability straightforward.
    • Message queue: Decouple the “generation” and “verification” stages, enabling large-scale tasks to run asynchronously in parallel.
    • Logging, monitoring, and alerting: Record the full call chain of each evaluation and trigger alerts when anomalies occur (such as timeouts or a sudden spike in verification failures).

    Cloud infrastructure such as Baidu AI Cloud can provide these building blocks—container platforms for elastic inference, object storage for evaluation data, and monitoring and alerting to keep evaluations stable. By assembling these general-purpose capabilities, a team can build a reproducible, scalable, and auditable pipeline for evaluating AI research ability, without reinventing the wheel.

    7. Conclusion

    “Will AI replace mathematicians?” is a headline-grabbing question, but a more important one is how to objectively measure AI’s research ability. Traditional math leaderboards prove that a model “can solve problems,” yet they can’t answer “can it do research?” The value of theoretical computer science lies in using the property of “machine-verifiable proofs” to turn the vague notion of “research ability” into something quantifiable and reproducible.

    For developers and researchers, the real lesson is this: evaluating AI can’t stop at chasing scores. It should be designed around whether the process is rigorous, whether the conclusion is verifiable, and whether the discrimination between models is real. When evaluation shifts from “looking at scores” to “looking at reasoning,” the yardstick we use to measure AI finally comes closer to the essence of research.