Constructing a Regular Heptagon with a Marked Ruler: A Hard Benchmark for LLM Reasoning
Simple HTML generation tests no longer separate strong LLMs from weaker ones. This article explains why neusis construction of a regular heptagon is a better benchmark: verifiable answers, long reasoning chains, and traps for fake success. It covers the math, prompt design, dual scoring of result vs. process, failure modes, and practical evaluation advice.
1. The evaluation trap: why simple HTML tests are losing power
Communities have long used a simple “generate a webpage” request as a quick proxy for model skill. As overall capability has risen, that test has lost discriminative power. Most mainstream models can produce a clean layout and smooth animation. “Can generate a page” is no longer a bar.
The core issue is that easy tasks do not expose real differences. A useful benchmark needs three properties:
- Verifiable answers: correctness can be checked mechanically or semi-automatically, not by taste.
- A long reasoning chain: many linked steps; one error fails the whole result.
- Fake-pass traps: the output can look finished while violating the actual constraints.
Tasks that meet these three points can serve as a hard yardstick for model capability.
2. What makes a real bar: the math of a marked-ruler heptagon
Classical geometric construction is a strong candidate because it is verifiable and requires a long chain of steps. A representative case is constructing a regular heptagon with a marked ruler (two fixed marks on a straightedge).
A regular heptagon cannot be constructed with unmarked straightedge and compass. Algebraically, the key quantity (2\cos(2\pi/7)) is a root of the cubic
[
x^3 + x^2 - 2x - 1 = 0,
]
which is irreducible over the rationals. The corresponding points lie outside the field extensions obtainable by compass and unmarked ruler.
A marked ruler (neusis construction) changes this: the ruler must pass through a given point while its two marks lie on two given curves. That operation is equivalent to solving a cubic, so it can do constructions impossible with classical tools—angle trisection, doubling the cube, and a regular heptagon. This is the classical Archimedean idea.
The task therefore tests whether a model:
- Understands why unmarked tools fail and neusis succeeds.
- Can map abstract neusis into concrete geometric steps.
- Can fill in many omitted ordinary compass-and-straightedge details.
- Can implement the process in code (for example an animation) that is correct and checkable.
Difficulty, verifiability, and chain length make it a strong evaluation sample.
3. How to design the evaluation prompt
A high-quality prompt states both the hard core and the full constraints. For this task, layers look like:
Goal: Build an HTML page that animates the full process of constructing a regular heptagon with a marked ruler.Starting construction:1. Take a square.2. Draw the perpendicular bisector of the base.Core neusis:3. With the lower-right vertex E as center and the diagonal as radius, draw circle E.4. Keep the ruler against the lower-left vertex D; slide the first mark on circle E until the second mark meets the base’s perpendicular bisector.Completion:5. Fill in every omitted ordinary compass-and-straightedge step.6. The compass should leave only necessary marks (points and arcs), not extra traces.7. Add export of the animation as a GIF.
The prompt gives the neusis constraints (steps 3–4) but omits many intermediate steps (step 5), so the model must complete the chain. A weak model may skip the constraints and fake a figure that only looks right.
4. Scoring: correctness of the result and fidelity of the process
Models often differ not on “whether a heptagon appears” but on whether they follow the path described in the prompt. Score two dimensions.
Result correctness. Is it a true regular heptagon? Check in code: seven vertices on a circle, consecutive central angles (2\pi/7):
// Check whether seven vertices form a regular heptagonfunction isRegularHeptagon(points, center, tol = 1e-6) {if (points.length !== 7) return false;// 1. Distances to center should be approximately equalconst radii = points.map(p => Math.hypot(p.x - center.x, p.y - center.y));const r0 = radii[0];if (!radii.every(r => Math.abs(r - r0) < tol)) return false;// 2. After sorting by angle, adjacent central angles should be 2π/7const angles = points.map(p => Math.atan2(p.y - center.y, p.x - center.x)).sort((a, b) => a - b);const step = 2 * Math.PI / 7;for (let i = 0; i < angles.length; i++) {const next = angles[(i + 1) % angles.length] + (i === angles.length - 1 ? 2 * Math.PI : 0);if (Math.abs((next - angles[i]) - step) > tol) return false;}return true;}
Process fidelity. Did the model start from a square, draw the circle at the lower-right vertex, and complete neusis with the marked ruler? This is harder to fake.
A typical failure mode: the model draws a regular heptagon but the method diverges from the prompt—extra auxiliary steps that were never specified, bypassing neusis. It got the answer and missed the process. Weak models tend to be result-oriented and dodge the hard constraint; strong models follow the given path and fill in every omitted compass step.
5. Common failure modes on long reasoning tasks
From this kind of evaluation, recurring issues include:
- Fake implementations that skip the core constraint: other means produce a correct-looking figure while ignoring the required method.
- Reasoning without delivery: lots of internal thought that never becomes a stable, runnable artifact (“thinks a lot, outputs little”).
- Broken step chaining: individual operations are understood, but compass steps are not sequenced; links are missing or out of order.
- Visualization disconnected from math: the animation is smooth, but coordinates are wrong, so the figure is not the intended one.
These often pass a “it runs” bar and only show up under strict process checks.
6. Practical notes for engineering such evaluations
To make hard evaluations reusable:
- Automate judging: encode result checks in scripts for consistency and reproducibility.
- Score process and result separately: record “correct figure” and “method matches the prompt” so a good result cannot hide a bad process.
- Separate generation quality from generation experience: fluency, observability of thinking, and latency are not the same as correctness. Speed–quality trade-offs depend on the use case.
- Build a difficulty ladder: from beginner prompts to constructions like the marked-ruler heptagon, so you can map the full capability distribution.
7. Conclusion
Evaluating LLMs is less about finding a hard puzzle and more about designing puzzles that cannot be gamed and whose correctness can be checked. The marked-ruler heptagon works because it is mathematically non-trivial, process-constrained, and result-verifiable. Capability is not only whether the drawing “looks like” a heptagon, but whether the path is the one required. When scoring weights both result and process, you can tell who completed the construction chain and who only lucked into the right shape.
