Why Passing Benchmarks Isn't Doing Research: Lessons from Theoretical Computer Science

    Leaderboards for frontier models keep climbing, but they mostly measure whether a model can solve exam-style problems, not whether it can do research. This article explains the gap between the two, why theoretical computer science (TCS) is unusually well suited to evaluating genuine research ability, the four types of tasks a TCS evaluation covers, and why machine-verifiable results make such evaluation possible. It also outlines the engineering infrastructure—data and question-bank management, large-scale verification, observability, and contamination detection—needed to run a credible evaluation pipeline, and closes with practical takeaways for developers.

    Leaderboards Keep Climbing, Yet Leave One Question Unanswered

    Every month or two, the evaluation leaderboards for frontier models get refreshed. The scores pile higher and the rankings stretch longer, until it becomes hard to keep track. But almost every leaderboard answers the same question: can the model solve problems? They rarely touch a harder and more important question—can the model do research? The two sound similar, but their cores are worlds apart.

    A popular claim is that “AI has turned math into a solved game.” Discussions like this trend for a large part because math problems are so “friendly” to grade: like a Sudoku puzzle, filling in the answer takes effort, but once it’s filled, right or wrong is obvious at a glance. Because verification is cheap and disputes are few, tasks with clear-cut answers—math and code—have become the most eye-catching “trophies” in model capability showcases.

    The problem is that chasing leaderboard scores measures test-taking ability, not research ability.

    Problem-Solving and Research Are Two Different Skills

    Think of solving problems as cooking from a recipe: the steps are given, the heat is standardized, and there’s a reference for how salty it should taste. You’ll almost never find chocolate in a sweet-and-sour pork dish, because the recipe frames what’s possible and the standard answer fixes what’s correct.

    Research is an entirely different matter. There are no preset steps, the path is full of forks, and no one can say in advance whether it leads anywhere. More subtly, many research questions can’t even be answered by the people who posed them—they are open questions to begin with. The closer a problem is to real research, the harder it is to grade: if no one knows the standard answer, who is the judge?

    This is exactly the crack that model evaluation has always struggled to avoid. A model that scores high on standardized exams may not be able to pose new questions, construct new proofs, or judge whether a line of reasoning has run into a dead end. And that, precisely, is the dividing line between “research” and “grinding through exercises.”

    Why Theoretical Computer Science, of All Fields

    If evaluating research is so hard, why pick theoretical computer science (TCS) specifically?

    Here’s an analogy: programmers are more like drivers, focused on the “experience layer”—the engine, the interior. TCS, by contrast, cares about traffic itself—from point A to point B, how fast can you go at the absolute limit? Is there any road that is fundamentally, in principle, impassable?

    This sits at the intersection of mathematics and computer science, where every question revolves around “computation,” and the main tool of research is proof. It has two properties that make it especially suited to testing models:

    • Formalized conclusions: research results tend to be a proposition that can be stated precisely, rather than a vague opinion.
    • Verifiable conclusions: once a proof is written, it can be checked step by step, and its correctness is relatively objective.

    In other words, TCS naturally offers the combination of “hard to pose, feasible to grade,” compressing open-ended research into objects that can be checked as much as possible.

    One Hundred Problems, Four Kinds of Tasks

    A TCS evaluation designed for frontier models can be roughly split into one hundred problems, corresponding to four kinds of daily work for a theoretical computer scientist. Rather than fixating on the specific questions of any one test, the point is to understand how “research actions are broken into evaluable items”:

    1. Understanding and restatement: read an existing theorem or proof and accurately restate its core mechanism. This tests whether the model truly “understands” rather than reciting the conclusion.
    2. Construction and proof (upper bounds): given a problem, produce a new valid result along with its proof. This corresponds to “how to do it”—finding a path and arguing that it actually works.
    3. Lower bounds and impossibility: argue that some goal cannot be done better. This corresponds to “why it can’t be done”—proving that no faster path exists.
    4. Counterexamples and formalization: construct a counterexample to refute a false claim, or rigorously formalize a problem stated in natural language. This tests rigor and awareness of boundaries.

    These four categories cover the full chain from “understanding others’ results” to “producing your own results,” and they capture research ability better than simply applying standard answers.

    The Key Is “Verifiability”

    The biggest obstacle to open-ended research evaluation is grading. What makes TCS a rare exception is that most of its conclusions can be verified by a machine.

    Take proofs as an example. A formalized proof can be checked line by line by a checker:

    1. # Illustration: a minimal proof-checking loop (pseudocode, for explanation only)
    2. def verify_proof(proof_steps, axioms, rules):
    3. context = set(axioms)
    4. for step in proof_steps: # advance the proof step by step
    5. if not rule_applies(step, context, rules):
    6. return False # if any step is invalid, the whole proof fails
    7. context.add(step.conclusion)
    8. return goal_in(context) # did we successfully derive the goal proposition?

    Real systems are far more complex than this, having to handle types, quantifiers, auxiliary lemmas, and many other details, but the core is the same: turning “this looks plausible” into “verified item by item.” This is exactly why fields like math and TCS can lead the way in research evaluation for AI—they give the judge a reusable ruler.

    From the Exam to Engineering: How to Build an Evaluation System

    A credible research evaluation is far more than “writing one hundred problems.” To make it real, engineering support is equally critical:

    • Data and question-bank management: problems, reference answers, proofs, and verification rules all need version control to prevent answer leakage from contaminating training data. This typically relies on object storage to hold problems and proof artifacts, and a message queue to schedule concurrent verification jobs.
    • Large-scale verification compute: formal verification and proof checking are often compute-intensive, requiring a container platform to elastically spin up hundreds or thousands of parallel jobs and reclaim them once they finish.
    • Observability and traceability: every evaluation’s inputs, outputs, and verification results should be persisted, and the process reconstructed through logging services and monitoring alerts, to avoid “scores that can’t be reproduced.”
    • Anti-cheating and contamination detection: regularly checking whether the question bank has already entered a model’s training corpus is the prerequisite for maintaining a leaderboard’s credibility.

    In this sense, the general-purpose cloud capabilities for storage, containers, messaging, logging, and monitoring form the infrastructure for research evaluation. For example, general-purpose cloud capabilities like those offered by Baidu AI Cloud can let teams focus their energy on the evaluation methodology itself, rather than repeatedly building up the underlying environment.

    Takeaways for Developers

    For developers, the value of this approach lies not in any specific score, but in a rethinking of what evaluation means:

    • Separate capability tiers: measure “solving problems” and “doing research” separately, and don’t let high exam scores mask weaknesses in reasoning.
    • Prioritize verifiability: favor tasks whose conclusions can be checked by a machine when evaluating models, so the judge stays as objective as possible.
    • Focus on process, not just results: research ability shows up in proposing ideas, constructing proofs, and recognizing dead ends—not just in a single final answer.
    • Use engineering to ensure credibility: a reproducible, traceable, contamination-resistant evaluation pipeline matters just as much as the model itself.

    Large models are moving from “answering questions” toward “thinking.” Theoretical computer science offers a rare mirror—one that reflects not only where models stand right now, but also how far the road to “genuine research ability” still stretches.