Geometric Construction Tasks for Evaluating Reasoning Fidelity in Code-Generating Models
This article explains how geometric construction tasks can test whether code-generating models truly follow procedural instructions rather than producing superficially correct outputs. It covers the mathematical background of regular heptagon construction, five evaluation dimensions, common failure modes, an automated verification script, and practical recommendations for testing and model selection.
1. Why Use Geometric Construction Tasks for Evaluation
Many model evaluations rely on fixed question banks or general Q&A, making it difficult to determine whether a model truly understands a task’s execution logic. Geometric construction tasks have two advantages. First, every step has clear mathematical constraints, and the result can be measured. Second, the task describes a sequence of operations, so the model must generate results step by step rather than relying on memorized answers.
Consider the task “construct a regular heptagon with a marked ruler.” The model must complete omitted compass-and-straightedge steps based on an operation description and generate a visualized construction process. This type of task simultaneously tests geometric knowledge, instruction understanding, code implementation, and animation expression, making it suitable as a comprehensive stress test.
2. Mathematical Background of the Task
A regular heptagon cannot be constructed using only a straightedge and compass. The Gauss-Wantzel theorem states that a regular n-gon is constructible with straightedge and compass if and only if all odd prime factors of n are distinct Fermat primes. Since 7 does not satisfy this condition, the task introduces a marked ruler, or “marked straightedge.”
A marked straightedge allows a neusis construction: a line passes through a known point, and the length between two marks on the line segment equals a given value. This operation exceeds the capability of ordinary straightedge-and-compass construction. Archimedes and Apollonius both used it to solve problems such as angle trisection.
The steps given in the task include: take a square, construct the perpendicular bisector of the bottom side, draw a circle centered at a corner point with the diagonal as radius, and then slide the marked ruler around a fixed point so that the first mark lies on the circle and the second mark passes through the perpendicular bisector. The subsequent steps are conventional straightedge-and-compass operations, such as determining circle centers, constructing angle bisectors, and connecting vertices.
The order of these steps cannot be changed arbitrarily. Each step provides geometric conditions for the next step, and omitting any step will invalidate the construction.
3. Designing Evaluation Dimensions
Checking only whether the final figure looks like a regular heptagon cannot reflect the model’s true capability. The following five dimensions should be scored separately.
1. Result correctness: Whether the final vertices form a regular heptagon, and whether side-length and central-angle errors are within tolerance.
2. Process fidelity: Whether the generated construction steps match the task description. If the model directly computes vertex coordinates analytically rather than performing the geometric operations specified in the task, it should be judged unfaithful even if the figure is correct.
3. Step completeness: Whether all omitted steps required by the task appear and whether their order is reasonable.
4. Trace control: The task requires that the compass’s movement traces be retained only in necessary parts. This dimension tests the model’s understanding of drawing semantics and also checks the animation’s rendering logic.
5. Delivery format: Whether the HTML page runs correctly, whether the animation is smooth, and whether the animated-export function can generate valid files.
4. Common Failure Modes
Common failures in similar tasks include the following.
First, superficial correctness with a deviating path. A model may first compute the coordinates of the regular heptagon’s vertices, draw the figure, and then add a note on the page saying “marked-ruler construction.” The result may appear to satisfy the requirements, but the actual method is unrelated to the task.
Second, unauthorized addition of steps that do not exist. To make intermediate steps close, the model may insert auxiliary lines or parameters not mentioned in the task so that the process can proceed.
Third, incorrect traces. All arcs and auxiliary lines are retained, or key traces are omitted, so the animation cannot show the sequential relationships in the construction.
Fourth, incomplete output. The generation process contains only reasoning, the final code appears late, or the code is truncated.
Fifth, excessive time consumption. Long generation time directly affects the interactive experience and can also cause timeout failures. Evaluation should record time to first output, time to complete output, and failure rate rather than only looking at the final result.
5. Example Automated Verification
For result correctness, a script can check the vertex coordinates exported by the model. The following function checks the distance from each vertex to the center, adjacent side lengths, and central angles to rule out misjudgments such as star polygons.
import mathdef check_regular_polygon(vertices, n, tol=1e-6):"""Check whether ordered vertices form a convex regular n-gon."""if len(vertices) != n:return False, "vertex count mismatch"cx = sum(x for x, _ in vertices) / ncy = sum(y for _, y in vertices) / nradii = [math.hypot(x - cx, y - cy) for x, y in vertices]r0 = radii[0]if any(abs(r - r0) > tol * max(1.0, r0) for r in radii):return False, "distances from vertices to center are not equal"sides = [math.dist(vertices[i], vertices[(i + 1) % n]) for i in range(n)]s0 = sides[0]if any(abs(s - s0) > tol * max(1.0, s0) for s in sides):return False, "side lengths are not equal"angles = [math.atan2(y - cy, x - cx) for x, y in vertices]steps = [(angles[(i + 1) % n] - angles[i]) % (2 * math.pi) for i in range(n)]if any(abs(s - 2 * math.pi / n) > 1e-4 for s in steps):return False, "central angle is not equal to 2π/n"return True, "passed"
This script can only check the result; it cannot determine whether the process is faithful. Process verification requires manual code review, or recording the geometric operation type at each step in the page and comparing it item by item with the task’s step list.
6. Engineering Practice Recommendations
Layer prompt design. Split the task into three layers: “background explanation,” “operation steps,” and “delivery requirements.” Validate the model’s understanding layer by layer to make it easier to locate failure points.
Control variables. Use the same input format and the same scoring criteria for the same task, and sample multiple times. A single result may fluctuate significantly and should not be used as the basis for a conclusion.
Record complete metrics. For each test, record elapsed time, output length, whether the output was interrupted, and whether there were cases with reasoning only and no result, forming a comparable data table.
Retain manual review. Detail errors in geometric animations are often difficult to spot with the naked eye, so key results should be checked step by step manually.
Do not draw conclusions from a single task. One geometry task can only reflect a model’s performance on a specific task. When selecting models, enterprises should combine multiple task types such as code generation, data processing, and document understanding for a comprehensive judgment.
7. Conclusion
The value of geometric construction tasks is that they distinguish “answered correctly” from “did it correctly according to the rules.” For applications that require reliable execution processes, such as engineering computation, teaching demonstrations, and automated document generation, process fidelity is often more important than the final result. Including these tasks in an evaluation system can help developers discover shortcut behavior earlier and provide a basis for subsequent prompt optimization and system design.
