Evaluating LLM Reasoning: How to Design Truly Rigorous Test Tasks

    Toy-level benchmarks are quietly losing their ability to separate strong models from weak ones. This article explains why frontend page-generation tests are failing, what makes a genuinely rigorous task, and why process correctness and instruction-following matter more than surface completion. Using the classic "construct a regular heptagon with a marked ruler" task as a running example, it walks through a scoring model that prioritizes honest reasoning, a checklist-based evaluation method, the tradeoff between reasoning depth and output speed, and practical guidance for building a reproducible, tamper-resistant evaluation pipeline.

    Why Toy-Level Tests Are Failing

    For a long time, frontend page-generation tests were almost the default way to evaluate a large model’s coding ability. You give a natural-language description, the model generates an interactive HTML page, and as long as it “runs, moves, and animates” to the naked eye, it passes. These tests became popular because they have a low barrier to entry, intuitive results, and are easy to compare side by side.

    But as model capabilities have improved across the board, the discriminative power of these tests is decaying quickly. There are three reasons:

    • Pattern memorization leads to “reciting answers”: Training corpora are saturated with similar tutorial pages and demo animations, so a model can pass by pattern matching rather than genuine reasoning.
    • High surface completion: A page that “looks” usable does not mean it truly fulfills the requirement. A model can easily assemble the visuals while contradicting the underlying logic of what was asked.
    • Lack of verifiable hard constraints: Toy-level tasks often have no clear correctness criteria, so evaluators can only score on subjective impression, making it hard to reach objective, reproducible conclusions.

    As a result, evaluators have started deliberately hunting for threshold-style tasks that “look easy but can stop a top-tier model in its tracks.” The value of such tasks is that they force the model to demonstrate its actual reasoning chain and constraint-following ability, rather than superficial assembly.

    What a Task That Can Actually “Crash” Looks Like

    An ideal rigorous evaluation task usually has the following characteristics:

    1. Simple to state, harsh to implement: The requirement can be described in a sentence or two, but faithfully executing it demands strict step-by-step adherence.
    2. A clear correctness criterion: The result can be verified objectively, rather than “looking right.”
    3. Counterintuitive constraints: The internal logic of the task tempts the model to take shortcuts, exposing whether it understands the problem or is just memorizing.

    Take a classic geometric construction task: build a webpage that uses animation to demonstrate the process of “constructing a regular heptagon with a marked ruler.” The reason this is “hardcore” rests on solid mathematical grounds:

    A regular heptagon is one of the classic problems that cannot be constructed with compass and straightedge alone (an unmarked ruler plus a compass), because it requires solving a cubic equation, whereas compass-and-straightedge constructions can only resolve constructions tied to quadratic equations. This is precisely the essential difference between it and “constructible” figures like the equilateral triangle, square, and regular pentagon. To genuinely construct a regular heptagon, you must resort to a stronger tool such as a marked ruler (a ruler with graduations, i.e., neusis construction), or an equivalent advanced method.

    This means that any animation claiming to “draw” a regular heptagon using only ordinary compass-and-straightedge steps is mathematically invalid. A model that truly understands the problem will demonstrate, step by step, exactly the specified movements of the marked ruler and the alignment of its graduations; a model trying to take a shortcut will often quietly insert steps that do not exist in the prompt, or simply draw a heptagon that “looks like” one to force out a final result.

    What makes this kind of task clever is that it turns “is it correct?” into a question with a definite mathematical answer. Who cut corners and who followed the constraints faithfully becomes obvious at a glance.

    Correctness Over Surface Completion

    Faced with a task like this, results from different models may “all look decent”—each can draw an edge, a polygon, and an animation on the page. But this is exactly where evaluators get misled most easily.

    The key criterion is not “whether a heptagon appears on the page,” but whether it actually followed the construction path specified in the task.

    A typical failure looks like this: the output superficially satisfies the requirement and seems to produce a correct regular heptagon, but the implementation does not match the requirement at all. The model may:

    • Directly “fill in” seven points from known vertex coordinates, skipping the entire construction process;
    • Add auxiliary steps that do not exist in the prompt on its own, bypassing the real difficulty;
    • Simplify the “marked ruler” step into ordinary line drawing, sidestepping the core constraint.

    In other words, it forces a visually correct final result through extra, unrequested steps. These “right result, cheated process” outputs are the failure mode to watch for most closely in evaluation. They show that the model did not truly understand the problem, but learned to disguise the absence of reasoning with means that appear compliant on the surface.

    For this reason, when designing evaluation criteria, process correctness must rank above surface completion. A scoring logic like the following works well:

    1. score =
    2. correct_final_result (final result is correct) * 0.3
    3. + follows_specified_steps (strictly follows the given steps) * 0.4
    4. + no_fabricated_steps (no unrequested steps added) * 0.2
    5. + verifiable_process (process is objectively verifiable) * 0.1

    As you can see, the two heaviest weights both point to “whether the process is honest and follows the constraints,” not “whether it looks the part.”

    Instruction-Following Is the Core Metric That Separates Models

    On complex tasks, what truly widens the gap between models is often not “can it write some code,” but can it precisely satisfy every constraint one by one under multiple conditions.

    For a geometric construction task like this, the prompt usually contains several explicit constraints:

    • First take a square and draw the perpendicular bisector of the base;
    • With a specific vertex as center and the diagonal as radius, draw a circle;
    • Keep the “marked ruler” flush against the specified vertex, sliding its graduations along the circle until the condition is met;
    • Complete all the ordinary construction steps that were omitted;
    • Leave construction marks only where necessary;
    • Finally, add the ability to export the result as a GIF animation.

    A model with strong instruction-following maps each of these constraints to the implementation exactly—no more, no less. A model with weak instruction-following tends to lose track of the whole: it either drops a step or replaces a key step with whatever it assumes. Instruction-following is a touchstone for a good evaluation because it simultaneously tests the model’s understanding of long text, its ability to preserve step order, and its sensitivity to edge conditions.

    When evaluating, it helps to break the prompt down into a checkable constraint checklist and verify, item by item, whether the model’s output hits each one:

    1. Constraint checklist (excerpt):
    2. [ ] Draw the square correctly and mark its four vertices
    3. [ ] Construct the perpendicular bisector of the base correctly
    4. [ ] With the specified vertex as center and the diagonal as radius, draw a circle
    5. [ ] Keep the marked ruler flush against the specified vertex, sliding its graduations along the circle
    6. [ ] Determine the key intersection only when the second graduation crosses the perpendicular bisector
    7. [ ] Complete the omitted routine perpendicular-bisector and compass steps
    8. [ ] Keep construction marks only where necessary
    9. [ ] Provide the ability to export the animation as a GIF

    Once it is put into checklist form, evaluation shifts from “subjective impression” to “objective item-by-item verification,” and both reproducibility and persuasiveness improve significantly.

    The Tradeoff Between Reasoning Depth and Output Speed

    One counterintuitive but noteworthy phenomenon is this: models that output faster and are “lighter” in size do not necessarily lose to flagship or largest-parameter models on these hardcore tasks.

    The reason is that reasoning ability is not always linearly correlated with model size. Faced with complex constraints, whether a model is willing to think deeply enough, verifies each step strictly, and proactively checks whether its output has drifted from the constraints is often more decisive than sheer “size.” Some models show a choppy, halting feel in their thinking process during generation, or even spend long stretches reasoning without producing output. From an engineering standpoint, this usually means the model is doing deeper, step-by-step deduction internally rather than rushing to emit a result. That “slowness” buys higher accuracy and stricter constraint-following.

    Evaluators need to build the right mental model:

    • Output speed ≠ capability. Speed reflects only generation throughput and has no necessary connection to correctness.
    • Time cost must be factored into the tradeoff. The extra latency from deep reasoning is a real user-experience cost in interactive scenarios and should be weighed during model selection.
    • Focus on “stability,” not a single run. Running the same task multiple times and observing whether the model consistently follows constraints is more valuable than a one-off score.

    At the same time, some engineering phenomena deserve honest attention: for example, a model that “thinks for a long time without outputting” may trigger timeouts and hurt the interactive experience. In real deployments, such issues can be mitigated through timeout policies, streaming feedback, and progress indicators.

    Building a Reliable Evaluation System

    Pulling the discussion together, an evaluation system that can truly reveal a model’s capabilities should include the following elements:

    1. Verifiable hard constraints: The task must have an objective correctness criterion—for instance, a geometric construction can be verified by computing intersection coordinates, and an exported animation can be verified by actually playing it back—rather than scoring on visual impression alone.
    2. Anti-shortcut scoring logic: Focus on whether the process is honest, and explicitly penalize opportunistic behavior like “quietly adding steps” or “jumping ahead by filling in coordinates.”
    3. A checklist-based verification mechanism: Break the prompt into checkable items so that evaluation moves from subjective to objective and from one-off to reproducible.
    4. Multi-dimensional composite scoring: Cover at least four dimensions—result correctness, instruction-following, absence of hallucinated steps, and verifiability—rather than a single metric.
    5. A difficulty-tiered test set: Include both routine tasks and hardcore tasks capable of stopping top-tier models, balancing discriminative power with coverage.

    After collecting evaluation data, it helps to funnel every run’s input, output, latency, and constraint-hit status into a unified store via monitoring and logging services, making regression comparison and capability-boundary analysis straightforward. To run evaluations at scale, you can orchestrate evaluation tasks with a container platform, archive result artifacts in object storage, and schedule concurrent tasks with a message queue—building a repeatable, traceable, engineering-grade evaluation pipeline.

    Practical Engineering Recommendations

    When putting this into practice, the following guidelines are useful:

    • Verify with code, not the naked eye. Write validation scripts for results that can be formalized (coordinates, intersections, animation frames) and let the machine decide right from wrong.
    • Run repeatedly to gauge stability. Execute the same prompt multiple times and track the constraint-hit rate to avoid being misled by an occasional “good result.”
    • Focus on capability boundaries, not a single score. The goal of evaluation is not to produce a ranking, but to understand under which constraints and difficulty levels a model fails, so as to guide subsequent selection and optimization.
    • Treat “process honesty” as an independent evaluation dimension. Fold the identification of opportunistic behavior into routine evaluation, and track over time how stable different models are in this respect.

    Ultimately, the real difficulty of evaluating a large model’s capability does not lie in designing a flashy problem, but in designing tasks that are not fooled by surface appearance, can be verified objectively, and reflect genuine reasoning quality. Only when evaluators shift from “does the page look right” to “is the process correct” have they truly crossed the threshold into serious capability assessment.