Evaluating LLM Reasoning with a Marked-Ruler Construction: Drawing a Regular Heptagon with Two Marks
When templated problems leak into training data, benchmark tasks that only ask a model to "produce the answer" quickly lose their value. This article presents a benchmark built around geometric constructions, using the classic marked-ruler (neusis) construction of a regular heptagon as a worked example. It shows how to design tasks whose process cannot be gamed, how to score them by checking each construction step against the required algorithm, and what output signals to watch beyond the final image. Readers will find a reusable task template, a capability breakdown, a weighted scoring rubric, common grading mistakes, and an evaluation-record template for reproducible model comparisons.
Start with the conclusion: evaluate LLMs on tasks whose process cannot be faked
Once large numbers of templated problems make their way into training corpora, evaluations that only ask a model to “give the answer” become easy to defeat. A genuinely useful benchmark task should let the model neither memorize its way around the problem nor hide errors behind polished output.
This article lays out a design and scoring approach for a class of construction-based benchmark tasks. Using “constructing a regular heptagon with a marked ruler” as the example, it tests whether a model can strictly follow mathematical constraints, execute a specified sequence of construction steps, and deliver a runnable front-end page along with an exportable animation.
The value of this kind of task is not in whether the model draws something that “looks like” the target figure. It is in the fact that neither the result nor the process can be fudged. If a key construction step is missing, or if the model quietly swaps in a more familiar approximation, the evaluation should deduct points.
Problem design: constructing a regular heptagon with a marked ruler
The task can be phrased as a copyable template:
Build an HTML page that constructs a regular heptagon using a two-mark ruler construction. Start from a square, then construct the perpendicular bisector of its bottom edge. With the bottom-right vertex E as center and the diagonal as radius, draw circle E. Then, using the marked ruler, keep it flush against the bottom-left vertex D and slide the first mark along circle E until the second mark passes through the perpendicular bisector of the bottom edge. Finally, determine the center through a standard perpendicular-bisector construction and complete the regular heptagon. Fill in all the compass-and-straightedge steps that were omitted; the compass produces point trails, but marks should be left only where necessary; and add a feature to export the result as an animation.
Here, the “two-mark ruler” (二刻尺) can be understood as a straightedge carrying two fixed marks: the two marks simultaneously constrain two geometric objects, allowing a construction that pure compass-and-straightedge methods cannot achieve. A regular heptagon is not directly constructible in the classical compass-and-straightedge sense, so the model must understand and apply an extended neusis-like constraint rather than simply outputting approximate angle coordinates.
Before running the evaluation, it helps to apply a minimal verification pass:
- Check whether a square is drawn and its bottom-left point D and bottom-right point E are correctly labeled.
- Check whether the perpendicular bisector of the bottom edge is constructed.
- Check whether circle E is drawn with E as center and the square’s diagonal as radius.
- Check whether the marked ruler stays flush against D and whether it shows the two-mark constraint: the first mark sliding along circle E and the second mark passing through the bottom-edge perpendicular bisector.
- Check whether the center is determined through a perpendicular-bisector construction and whether the heptagon’s vertices are generated from it.
- Check whether the trail animation and static marks are distinguished: keep only necessary marks and avoid meaningless smearing.
- Check whether the exported animation can be opened and played and includes the key construction steps.
These seven items are the most basic acceptance bar. If any one of them fails, the evaluation should not pass the model on the basis that “the final shape is correct.”
Task breakdown: what this problem actually tests
This looks like a front-end drawing task, but it simultaneously probes mathematical understanding, instruction following, reasoning coherence, code completeness, and engineering delivery.
| Capability dimension | How it shows up in this problem | Typical failure mode |
|---|---|---|
| Math understanding | Whether it understands that a regular heptagon is not constructible with compass and straightedge alone, and uses the marked-ruler constraint to solve it | Directly dividing the circle into equal arcs, or claiming ordinary compass and straightedge suffice |
| Instruction following | Whether it strictly executes the specified order: square, circle E, the ruler’s two marks, perpendicular bisector | The result looks like a heptagon, but the construction does not match the required steps |
| Reasoning coherence | Whether the multi-step construction is self-consistent and traceable | Skipping steps, omitting key constraints, marks not matching steps |
| Code completeness | Whether coordinate computation, animation, interaction, and export are runnable | The page displays, but animation is missing, interaction fails, or export is unusable |
| Engineering delivery | Whether it delivers an openable, savable, reproducible page and animation | Only screenshots, no runnable code, or a corrupted export file |
To make scoring more executable, add a weighted rubric after the capability breakdown. Score each item as 0, 1, or 2, then convert by weight.
| Scoring item | Weight | 0-point criteria | 1-point criteria | 2-point criteria |
|---|---|---|---|---|
| Math construction | 30% | Doesn’t understand the marked-ruler constraint; outputs an approximate heptagon directly | Knows it isn’t constructible with compass and straightedge alone, but doesn’t correctly implement the two-mark constraint | Correctly expresses circle E, point D, the two-mark constraint, and their relationship to the final construction |
| Instruction following | 25% | Replaces key steps; doesn’t follow the specified vertices, center, and radius | Some steps are correct, but a key order is omitted or changed | Faithfully reproduces the specified construction order and geometric objects |
| Mark control | 15% | Leaves random trails, or no necessary marks at all | Has marks, but they are too cluttered to distinguish the key constructions | Leaves marks only where necessary, with animation and static marks clearly separated |
| Code completeness | 20% | Page doesn’t run, or core logic is missing | Displays a basic figure, but animation and interaction are incomplete | Coordinate computation, movement animation, and interaction logic all run correctly |
| Animation export | 10% | No export function, or the exported file can’t be opened | Has an export entry, but content is missing or the format is broken | The exported animation plays, includes the key steps, and can be reviewed |
The final score can be aggregated as “each item’s score ÷ 2 × weight.” A rubric like this prevents a model from scoring well on the final figure alone.
For example, a failing attempt might use code like this:
for (let i = 0; i < 7; i++) {const a = 2 * Math.PI * i / 7;pts.push([cx + r * Math.cos(a), cy + r * Math.sin(a)]);}
This code may generate a neat heptagon, but it bypasses circle E, the ruler’s two-mark constraint, and the perpendicular-bisector method for finding the center. It looks approximately correct but is not algorithmically valid, so it should not earn a high score.
Key observation: a correct result does not mean a correct process
In practice, the easiest thing to misjudge is exactly this “surface-correct” trap.
A model may take a fairly long time to generate, then produce a page that indeed renders as a neat regular heptagon. If you only look at the screenshot, there is almost nothing to criticize. But once you inspect the construction code step by step, you find that it never solved the problem with the “ruler flush against the vertex, two-mark constraint” method, but instead took a more familiar shortcut—for instance, directly dividing the circle by equal central angles, or directly outputting approximate coordinates.
This is the classic failure scenario: the output meets the visual expectation but violates the algorithmic essence. Such errors are especially deceptive in pure front-end rendering evaluations, because the nicer the page looks, the more it lulls you off guard.
Conversely, a truly strong model usually does the following:
- Faithfully reproduces every construction step in the problem;
- Clearly distinguishes “necessary marks” from “trail animation”;
- Correctly handles the fact that a regular heptagon is not constructible with compass and straightedge alone;
- Uses the marked-ruler constraint to lay out or solve the construction;
- Fully implements animation export and makes the animation able to verify the key steps.
So when scoring, you should not only ask “does it look right,” but “was it built with the specified algorithm.”
Three output signals worth watching during evaluation
Beyond the final result, the model’s output behavior is itself an important evaluation signal.
1. Coherence of the reasoning process
A high-quality model’s reasoning is usually smooth and traceable. It tends to develop step by step around the geometric objects, rather than frequently skipping ahead or repeatedly overturning itself. If the reasoning process shows clear discontinuities, or switches abruptly from one wrong direction to another several times, it usually means the model is struggling with multi-step geometric reasoning.
2. Thinking without output, and abnormal latency
Some models stay in the reasoning phase for a long time and are slow to produce any visible output; certain complex tasks can even take tens of minutes. Long latency is not necessarily a defect in itself, but if a model still doesn’t strictly follow the construction after thinking for a long time, what’s exposed is a capability gap, not “careful deliberation.”
3. Hallucination and instruction substitution
This is the most important issue to watch for. When uncertain, a model may “invent” a seemingly plausible construction, or quietly replace the specified method with a familiar alternative. For example, using equal angle division instead of the marked-ruler constraint, or using preset coordinates instead of a derived process. Recognizing this kind of “unauthorized substitution” matters more than spotting an obvious error.
Note that there are limits to how observable these signals are. Whether the reasoning process is visible depends on whether the model and the product expose the relevant logs or intermediate output; latency is also affected by sampling parameters, context length, the toolchain, the runtime environment, and concurrency. So when evaluating, record information such as the model version, temperature, top_p, number of repetitions, and runtime environment, to avoid mistaking random fluctuations for capability differences.
Reusable evaluation design principles
How to design benchmark tasks for LLMs
To make a benchmark task genuinely discriminating, consider the following principles:
- Process constraints that can’t be fudged: specify key steps precisely and make them non-substitutable, so the room to “fake it with a different method” is minimized.
- Distinguish visual correctness from algorithmic correctness: don’t grade from the rendered screenshot alone; verify line by line whether the implementation is faithful to the specified algorithm.
- Add engineering-completeness requirements: layer on requirements like export, animation, and interaction to test whether the model can deliver end-to-end code beyond reasoning.
- Record process metrics, not just results: fold in observations like reasoning coherence, latency, and whether hallucinated substitution occurred to form a multi-dimensional profile.
- Stay neutral and reproducible: base tasks on general mathematical knowledge, avoid depending on any particular product or platform features, and ensure conclusions reproduce across environments.
Common grading mistakes
Common mistakes include:
- Looking only at the final screenshot without checking the construction process;
- Checking only whether a heptagon was generated, without checking whether the marked-ruler constraint was used;
- Treating the trail animation as necessary marks, or conversely ignoring key marks;
- Overlooking whether the exported animation can be opened;
- Overlooking whether the model quietly replaced the specified method with an approximation algorithm;
- Not recording the model version and parameters, making the result non-reproducible.
Evaluation record template
For easier review, record at least the following fields:
| Field | Description |
|---|---|
| Model name | Identifier of the model being evaluated |
| Version or date | The model version at evaluation time |
| Parameter configuration | temperature, top_p, maximum output length, etc. |
| Number of runs | How many times the same task was run |
| Elapsed time | Time from input to final output |
| Output screenshot | The rendered result of the final page |
| Code diff | Key construction code or a diff |
| Failure reason | Math, instruction, mark, code, or export issue |
To run these evaluations in practice, you can rely on general-purpose infrastructure such as cloud containers, object storage, and log monitoring to host the evaluation environment, archive results, and track each model’s latency and output behavior. This raises the engineering maturity of the pipeline without depending on any single vendor’s proprietary tooling.
Conclusion
A common misconception is that if a model can “draw a figure,” it “can do geometry.” But a correct result never equals a correct process. Tasks like constructing a regular heptagon with a marked ruler are valuable because they simultaneously gate four things: mathematical understanding, instruction following, reasoning coherence, and code completeness. If any one link is fudged, it will likely surface during line-by-line inspection.
For developers and those making technology selections, rather than chasing toy problems that have already been memorized and are easy to fake, it is more productive to invest evaluation effort in construction-based tasks whose process cannot be cheated. Such evaluations usually reveal more clearly whether a model is capability-driven or performance-driven, and they provide a more reliable basis for selection and engineering practice.
