From Problem-Solving to Research: A New Approach to Evaluating LLM Research Capabilities

    Abstract

    Public benchmarks for large language models mostly measure problem-solving rather than research, and as questions leak into training data, scores inflate while real capability stalls. This article examines a systematic evaluation that tests frontier models across the main directions of Theoretical Computer Science (TCS), and explains why TCS is a suitable touchstone: its conclusions can be machine-verified, it holds a deep stock of open problems, and its questions can be freshly generated to resist contamination. Built around 100 questions mapped to four types of TCS work, the evaluation combines strict automated grading, partial credit, dual-mode testing, multi-sample statistics, and human review. The key result is that no model independently solved any open conjecture—showing that models excel at retrieval and recombination but lack the ability to propose hypotheses, which positions them as amplifiers rather than replacements for researchers. The article closes with engineering guidance for building a contamination-resistant, judgeable evaluation pipeline, including design principles, a sample workflow, and a reference grading interface.