Evaluating Lightweight Flagship LLMs: What a Geometric Construction Problem Reveals
This article examines how to evaluate "lightweight flagship" (flash-class) LLMs that trade peak capability for lower latency and cost. It argues that geometric construction problems—specifically constructing a regular heptagon with a neusis (marked-ruler) construction—make strong probes because they combine high complexity, verifiability, and strict instruction constraints. Building on this problem, the article explains the underlying mathematics, how to design effective prompts, and why a "correct result" is not the same as a "correct process." It also covers observing reasoning traces and the latency tradeoff, a general front-end approach to geometric animation and GIF export, and practical steps for building a reproducible evaluation framework. Developers will learn to test models with constrained, verifiable, shortcut-resistant tasks rather than relying on seemingly correct outputs.
1. Background: Where Lightweight Flagship Models Fit, and Why Evaluation Is Hard
Over the past couple of years, the industry has seen a class of models positioned as “lightweight flagships” (flash-class). Rather than pushing parameter counts to the extreme, they aim to balance inference latency, cost per call, and overall capability: compared with top-tier flagships, they respond faster and cost less, while still trying to hold their own on complex tasks.
This positioning creates a practical dilemma: how do you prove that “lightweight” hasn’t come at the expense of core capability?
Traditional evaluation tends to fall into two traps:
- Results-only judging: looking solely at whether the final answer “looks right,” while ignoring whether the reasoning process is sound;
- Toy tests: long relying on small tasks such as drawing web pages, but the discriminating power of such tasks is fading, making it hard to separate models by their true reasoning depth.
To genuinely measure a model’s substance, you need a probe task that simultaneously satisfies three properties: high complexity, verifiability, and strong instruction constraints. Geometric construction problems happen to have all three.
2. Why Geometric Construction Makes a Good Probe
A high-quality evaluation task should ensure that “the result can be judged objectively, yet the process is hard to fake.” Geometric construction problems offer exactly this:
- Strong process constraints: the problem typically specifies explicit construction steps and drawing tools; deviating from those steps should be marked as failing even if the correct figure is drawn;
- Verifiability: whether the final figure meets the goal (such as the side length or angles of a regular heptagon) can be strictly checked through coordinates and geometric calculation;
- Resistance to shortcuts: it is easy for a model to “bypass the rules and just assemble the answer,” which precisely exposes whether it is truly reasoning or gaming the task.
So the focus of evaluation isn’t “can it draw the figure,” but “did it follow the given method strictly.”
3. Mathematical Background: Why a Regular Heptagon Needs a Neusis Construction
To appreciate the value of this problem, you first need to understand its mathematical difficulty.
With classical straightedge and compass (an unmarked ruler plus a compass), the constructible regular n-gons are strictly constrained: n must be a power of 2 multiplied by distinct Fermat primes (3, 5, 17, 257, 65537). Since 7 is not a Fermat prime—algebraically, cos(2π/7) is a root of an irreducible cubic equation, of degree 3 rather than a power of 2—the regular heptagon cannot be constructed with straightedge and compass alone.
Once you introduce the neusis construction (using a marked ruler with two marks), the capability boundary is broken: neusis is equivalent to solving a certain class of cubic equations, enabling constructions that straightedge and compass cannot achieve, such as angle trisection and constructing a regular heptagon.
The approach described in the problem belongs to this category: using a square and its diagonal circle, fitting the two-marked ruler against a specified vertex so that one mark slides along the circle while the other lands exactly on a perpendicular bisector—this “fit–slide–align” motion is the heart of the neusis construction.
Conceptual sketch (not exact coordinates):1. Draw square ABCD and the perpendicular bisector l of the base2. With the bottom-right corner E as center and the diagonal as radius, draw circle C3. Hold the two-marked ruler against the bottom-left corner D:- The first mark always stays on circle C- Slide until the second mark falls exactly on the perpendicular bisector l4. From the resulting key point → complete the remaining compass-and-straightedge steps → obtain the regular heptagon
Precisely because the difficulty is real, it acts as a touchstone for models: do they follow the instructions, or take shortcuts?
4. Evaluation Design: Building an Effective Test Prompt
To reproduce this kind of evaluation, prompt design should follow a few principles:
- Give explicit steps: lay out the construction workflow clearly so you can later judge whether the model “followed the steps”;
- Require filling in implicit steps: say only “complete the omitted compass-and-straightedge part” to test whether the model can autonomously fill in the intermediate constructions;
- Constrain construction marks: require that “circles leave traces only where necessary,” which actually tests the model’s understanding of the geometric process rather than random line-drawing;
- Require an interactive artifact: e.g., “add a feature to export a GIF,” which upgrades a static result into dynamic verification and raises difficulty and discriminating power together.
Such a problem looks like “geometry drawing” but actually assesses four capabilities at once: mathematical understanding, instruction following, process planning, and engineering implementation.
5. Core Evaluation Dimension: Correct Result vs. Correct Process
What this problem reveals most clearly is the divide between two kinds of models.
One kind of model takes a long time (on the order of tens of minutes), its reasoning output appears in “stuttering,” segmented bursts, and it may even “think without producing output”—yet its final artifact strictly follows the neusis route described in the problem and completes the required compass-and-straightedge steps.
Another kind of model may produce a “seemingly compliant” result in a short time—a regular heptagon does appear on screen—but its implementation has nothing to do with the problem: it adds steps that don’t exist in the problem and simply “assembles” the final figure.
This exposes a key conclusion:
Correct result ≠ correct process.
The latter is essentially “achieving the goal by bypassing the constraints.” In real business scenarios, this kind of shortcutting may mean the model is unreliable in situations that demand strict compliance (such as workflow tasks or traceable computation). Evaluation must therefore record both the final answer and the reasoning trace, and treat “did it follow the given steps” as an independent scoring item.
6. Observing the Reasoning Process and the Latency Tradeoff
For models with reasoning ability, evaluation should also pay attention to the observability of their thinking process:
- Characteristics of the reasoning stream: high-quality models often produce segmented, non-uniform reasoning output on complex tasks—this is a normal reasoning rhythm and shouldn’t be judged a defect simply as “stuttering”;
- “Thinking without output”: this may stem from the reasoning phase consuming a large reasoning budget before the model enters the answering phase; the evaluation framework needs to handle stream collection and timeouts properly;
- Latency is a real cost: one selling point of a lightweight flagship is speed, but on hard tasks, “fast” and “accurate” may not be achievable together. Evaluation should quantify total time, reasoning share, and answering share together, rather than looking only at final quality.
It is advisable to time each metric separately in the evaluation script, forming a three-dimensional profile of “quality–latency–compliance”:
metrics = {"task_correct": False, # Whether the result is correct (geometrically verifiable)"process_compliant": False, # Whether the given steps were strictly followed (manual/rule-based)"total_latency_s": 0.0, # End-to-end latency"reasoning_ratio": 0.0, # Share of time spent in the reasoning phase"answer_only_flag": False, # Whether "reasoning only, no output" occurred}
7. General Implementation of Geometric Animation and GIF Export
The “animation + GIF export” required by the problem is essentially a pure front-end engineering problem that doesn’t depend on any proprietary platform. The core is to describe geometric primitives with Canvas or SVG and implement the “drawing process” animation via frame-by-frame progression.
Basic loop for frame-by-frame animation:
function animate(drawFrame, duration = 2000) {const start = performance.now();function tick(now) {const t = Math.min((now - start) / duration, 1); // progress from 0 → 1drawFrame(t); // draw the current frame at this progressif (t < 1) requestAnimationFrame(tick); // continue if not finished}requestAnimationFrame(tick);}
Key primitive drawing (circles and marks as examples):
function drawCircle(ctx, cx, cy, r, t) {ctx.beginPath();// Draw only the necessary arc segment, not the full circle, honoring the "minimize construction marks" requirementconst end = Math.PI * 2 * t;ctx.arc(cx, cy, r, 0, end);ctx.stroke();}
The idea behind GIF export is also general: as the animation progresses, render each frame to an offscreen Canvas, read its pixel data, hand it to a GIF encoder to encode frame by frame, and finally package and download. Three implementation points matter:
- Fix the frame rate and frame count (e.g., 20 frames/second) to keep the GIF size manageable;
- Unify canvas dimensions and the color palette to avoid flicker and size jumps;
- Do the encoding in the browser, with no server involvement, making it easy to embed in the evaluation artifact page.
This implementation is model-agnostic—the model’s value lies in whether it can correctly understand the geometric constraints and generate code like this, not in which platform the code runs on.
8. Practical Advice for Building a Reproducible Evaluation Framework
To keep this kind of evaluation sustainable and comparable, it is recommended to:
- Fix the problem and scoring criteria: split “correct result” and “compliant process” into two independent scores, and codify the judging rules;
- Preserve the complete trace: save the reasoning log, generated code, and rendered artifact together, to make it easy to review afterward whether shortcuts were taken;
- Run multiple rounds: run the same problem several times to observe stability and avoid being misled by a single accidental outcome when selecting a model;
- Unify the environment: compare horizontally under the same parameters and the same prompt to reduce irrelevant variables;
- Quantify cost: record time and consumption together, making the advantages and costs of “lightweight” transparent.
9. Conclusion and Takeaways for Model Selection
Using a neusis-constructed regular heptagon as a probe, we can draw conclusions more valuable than a “benchmark score”:
- Lightweight flagship models can indeed reach flagship-level performance on complex reasoning, but often at the cost of longer reasoning time;
- What truly separates models isn’t “can it draw the correct figure,” but “did it strictly follow the given method”—compliance is a stricter standard than result correctness, and one closer to real business needs;
- Evaluators should move up from “looking at results” to “looking at the process,” folding latency, reasoning share, and instruction-following into a unified profile.
Practical advice for developers: don’t be fooled by a single demo or a “seemingly correct” result. Use tasks with strong constraints, verifiability, and resistance to shortcutting to test a model, so you can make a reliable judgment before selection and deployment.
