Evaluating Lightweight LLMs: A Practical Guide to Geometric Construction Tasks
A practical guide to evaluating lightweight large language models on geometric construction tasks, using the neusis construction of a regular heptagon as the benchmark task. It explains why geometric drawing exposes reasoning depth better than single-page animations, why the heptagon problem cannot be bypassed, and how to build a weighted scoring rubric backed by automated geometric assertions. It also reports empirical differences in behavior between agent-based lightweight models and single-shot flagship models, interprets why small models can perform well here, and offers engineering recommendations on routing, process auditing, and evaluation pipelines.
1. The Evaluation Dilemma: Moving from “Toy Tests” to “Threshold Tests”
Over the past year, asking a model to generate a single-page animation has become the default exam question for almost every model comparison. Its strengths are clear: zero environment dependencies, intuitive visualization, and results you can judge at a glance. But as model capabilities have broadly improved, the discriminating power of these tasks is fading fast. The vast majority of models can now deliver something that runs, displays, and looks reasonably polished, so the differences are down to fine details like color schemes, spacing, and easing curves. In other words, these tests measure a model’s average aesthetic sense and coding fluency, not its depth of reasoning. The community’s judgment that “these tests are essentially toys” is a natural response to that loss of discriminating power.
Yet abandoning visualization tasks entirely is equally mistaken. Geometric construction tasks, in fact, have a rare property: the cost of describing them is extremely low, but the execution threshold is extremely high. You can state the requirement in a single sentence, but completing it correctly demands precise mathematical reasoning, long-chain state management, and strict self-verification. This makes such tasks a natural probe for reasoning ability.
The capabilities an evaluation should actually measure can be grouped into four areas:
- Instruction following under stacked constraints: whether the model can satisfy every requirement without omitting, rewriting, or inventing any;
- State consistency across long chains: whether step 20 still stays faithful to the geometric objects defined in step 1;
- Self-verification and correction: whether the model can recognize and adjust when it hits numerical degeneracy or unsolvable intersections;
- Correct mapping from semantics to code: whether a mathematical description is accurately translated into coordinates, transformations, and animation timing.
2. Why “Neusis Construction of a Regular Heptagon” Is an Excellent Probe
2.1 The Mathematical Background Makes It Un-bypassable
The constructibility of compass-and-straightedge figures follows a strict criterion (the Gauss–Wantzel theorem): a regular n-gon is constructible if and only if n is the product of a power of 2 and some number of distinct Fermat primes. Since 7 is not a Fermat prime, a regular heptagon is not constructible in the pure compass-and-straightedge framework. This is not “hard” — it is “impossible.”
If we allow the use of a neusis (a sliding ruler with markings), the capability boundary expands immediately: a neusis constraint is equivalent to solving a cubic equation, and the construction of the regular heptagon happens to lie on a cubic irreducible extension, which makes it constructible. This means any output that claims to be “complete” can be strictly judged — it either genuinely followed the neusis route, or it secretly switched to some other construction.
2.2 The Task Description Is Itself a Constraint Trap
The problem specifies a complete path: start from a square and construct the perpendicular bisector of the base edge; draw a circle centered on the bottom-right vertex with the diagonal as its radius; slide the neusis tight against the bottom-left vertex so that the first graduation always stays on the circle, until the second graduation passes through the perpendicular bisector of the base; finally, use the intersection of the perpendicular bisectors to locate the center and complete the remaining construction. Two engineering constraints are attached: the compass should leave only the necessary traces, and the result must support export as an animated image.
Hidden here is an extremely tempting shortcut: directly divide the circle into seven equal parts of 360°/7 each. The result is correct, the code is concise, and the visuals are perfect — but it completely departs from the construction chain the problem specified. This “result-aligned, process-deviated” failure mode is the most subtle and most evaluation-worthy part of long-chain reasoning tasks, precisely because visual acceptance alone can almost never catch it.
3. Evaluation Design: Experimental Group, Control Group, and Scorecard
3.1 The Two Groups and Their Variable Control
The experimental group is a class of lightweight flagship models, accessed through a coding tool with full tool capabilities (in agent form) that can read and write files, launch a local preview, take screenshots for self-checking, and iterate across multiple rounds. The control group is a class of flagship conversational models, generating the entire codebase in a single round through a purely conversational interface.
It must be emphasized that the interaction form is itself a variable. The agent environment permits a closed loop of “generate — run — observe — fix,” while a single-round conversation offers only one output opportunity. Therefore, a rigorous conclusion should be stated as the performance difference of the “model + tool environment” combination, and cannot simply be attributed to parameter scale. Conflating this layer in an evaluation report is a common source of distorted comparison results.
3.2 A Five-Dimension Scorecard
A weighted rubric is preferable to an overall impression of “looks pretty good”:
| Dimension | Weight | How to Judge |
|---|---|---|
| Instruction following | 30% | Compare each stated step against each implemented step; check for invented steps or replaced core constraints |
| Geometric correctness | 25% | Vertex count, concyclicity, edge-length consistency, central-angle error — verified with numerical assertions |
| Process interpretability | 15% | Whether the animation presents step by step, whether auxiliary-line traces are reasonable, whether it can be reviewed stage by stage |
| Engineering completeness | 15% | Whether it runs, scales to the viewport, and whether animated-image export actually works |
| Delivery efficiency | 15% | End-to-end time, token consumption, and whether manual rework is required |
3.3 Making “Right vs. Wrong” Automatically Judgeable
The biggest risk in visual evaluation is reliance on the naked eye. Translating the geometric conclusions into assertions turns the rubric into tests that can run in CI:
import mathdef check_heptagon(points, tol=1e-3):"""Validate the output vertex coordinates geometrically, replacing manual visual inspection."""n = len(points)assert n == 7, f"Vertex count mismatch, got {n}"cx = sum(p[0] for p in points) / ncy = sum(p[1] for p in points) / n# 1) Concyclicity: all vertices should be approximately equidistant from the centroidradii = [math.hypot(x - cx, y - cy) for x, y in points]assert max(radii) - min(radii) < tol, "Vertices are not concyclic"# 2) Equilateral: adjacent vertex distances should be consistentdef d(a, b):return math.hypot(a[0] - b[0], a[1] - b[1])edges = [d(points[i], points[(i + 1) % n]) for i in range(n)]assert max(edges) - min(edges) < tol, "Edge lengths are inconsistent"# 3) Central angle: each interval should be 2π/7angs = sorted(math.atan2(y - cy, x - cx) for x, y in points)gaps = [angs[i + 1] - angs[i] for i in range(n - 1)]gaps.append(2 * math.pi - (angs[-1] - angs[0]))assert all(abs(g - 2 * math.pi / 7) < tol for g in gaps), "Central angle deviation"return True
These assertions can only prove that “the result is correct.” Whether “the process is correct” still requires manual review of the source code and animation timing. Scoring the two separately is exactly where the evaluation value of this task lies.
4. Empirical Observations: Three Typical Behaviors
Observation 1: Process Divergence in Long-Chain Tasks
After a relatively long generation in the conversational interface, the control group delivered a highly complete page: the regular heptagon had the correct shape, vertex labels were clear, the animation was smooth, and at first glance there was almost nothing to fault. But line-by-line inspection of the implementation logic reveals that the construction chain had been rewritten — it introduced auxiliary steps the problem never asked for, took an easier-to-compute alternative path to bypass the neusis constraint, and replaced “finding the intersection by sliding the neusis” with an equivalent numerical distribution. The figure is right, but the derivation is broken.
Such a result would score highly in a purely visual review, yet it exposes precisely the model’s weak spot in inheriting long instructions: when the computational cost of the strict path is high, the model tends to self-compensate with an alternative path that “produces the right answer,” without proactively declaring the deviation.
Observation 2: The Cost of Strictly Executing Instructions
The experimental group advanced strictly along the given path: first construct the square and the perpendicular bisector of the base, then draw the circle centered on the specified vertex with the diagonal as its radius, then abstract the neusis into a movable line constrained by two points and search numerically and iteratively for the position that satisfies the neusis condition, and only then use the intersection of the perpendicular bisectors to determine the circumscribed-circle center, fill in the remaining constructions, and wire up the animation and animated-image export. Intermediate objects, auxiliary lines, and trace-erasure logic were all preserved as the problem required, and the entire process could be reviewed stage by stage.
The cost was an end-to-end time on the order of half an hour — clearly longer than the control group. This is a classic efficiency-vs.-fidelity tradeoff: strict constraint execution significantly increases the number of reasoning steps and self-verification passes.
Observation 3: Engineering Experience and Capability Must Be Evaluated Separately
The lightweight model side also revealed two experience-layer problems: the thinking process was returned as fragmented, discontinuous increments, giving a “laggy” impression; and there were long periods where only thinking was output and no formal answer followed, leaving users unable to distinguish “still reasoning” from “already failed.”
These two points are service-layer defects in streaming output and progress feedback, and should not be equated with model-capability shortfalls. But in production they are hard risks all the same: batch jobs may be mistakenly killed by timeouts, and interactive scenarios may be judged as unresponsive by users due to the lack of a heartbeat.
5. Capability Interpretation: Why Lightweight Models Can Reach This Level
Equating small parameter size directly with weak capability is an intuition that has already become outdated. Results like these are usually supported by three factors:
- Optimization of the mix of reasoning-trajectory data. Once long-chain reasoning, tool calls, and intermediate state checks are incorporated into training data, the model learns “how to advance step by step under multiple constraints,” not merely “how to write good-looking code.”
- Gains from reinforcement learning aimed at instruction following. Satisfying multiple constraints simultaneously is a sparse but clear reward signal, and targeted optimization of long chains of thought shows particularly marked gains on this class of tasks.
- Task-decomposition ability brought by the tool form. The agent environment breaks a one-shot hard problem into verifiable small steps of “read file — run preview — inspect screenshot — fix code,” turning an unsolvable long chain into a solvable loop of short chains.
At the same time, the conclusion must be bounded reasonably. Performing well on constraint-intensive geometry and algorithmic reasoning does not mean the model can replace flagship models on every dimension. Long-context retrieval, ultra-low-latency real-time scenarios, and long-document understanding that requires deep domain knowledge still need to be evaluated case by case. The value of an evaluation conclusion lies in drawing clear boundaries, not in issuing a blanket verdict of “who is stronger.”
6. Engineering Recommendations
- Layered routing. Classify tasks by “constraint density × verifiability”: route low-constraint, latency-sensitive tasks to lightweight models, and route high-constraint tasks requiring strict auditing to stronger models or add multi-sample voting.
- Prioritize process traces. Require the model to output intermediate objects and reasoning trajectories rather than only delivering final files. Black-box output that cannot be audited carries unacceptable risk in geometry- and finance-computation scenarios.
- Staged timeouts and degradation. Set a no-progress timeout for long-reasoning tasks (for example, abort and retry after several minutes with no new token), and add a heartbeat signal at the service layer to avoid misjudging “still working” as a failure.
- Replace visual acceptance with assertions. Script the geometric and engineering checks from the scorecard and fold them into CI, so that regression evaluation across model-version upgrades can run automatically.
- Build a small, high-quality evaluation set. Ten to twenty problems that have “low description cost and high execution threshold,” covering three failure modes — instruction following, mathematical correctness, and engineering completeness — often provide more guidance for model selection than chasing public leaderboards.
- Combine general cloud capabilities into a closed loop. On a cloud model-serving platform such as Baidu AI Cloud, orchestrate models of different sizes on demand and pair them with general-purpose capabilities like logging, monitoring, and alerting, so that evaluation results settle into an observable routing policy rather than a one-off manual conclusion.
7. Conclusion
The evaluation value of geometric construction tasks does not lie in whether the picture looks good, but in the fact that they provide an un-bypassable factual criterion: either the model genuinely completed the construction chain under the constraints, or it traded an alternative path for a correct appearance. This practical test shows that lightweight models are already quite solid at strict instruction inheritance and long-chain reasoning, with their shortfalls concentrated more in service-layer streaming output and progress feedback — whereas traditional flagship models are, on the contrary, more prone to the subtle “result right, process deviated” drift on the same kind of task.
The practical takeaway for developers is clear: do not base model selection on a single-round visual impression, but on an evaluation process that is assertable, reproducible, and auditable. Only when “process correctness” and “result correctness” are scored separately can the contributions of model capability, tool form, and engineering strategy be truly distinguished — and this is the first cornerstone of building stable AI applications.
