How to Evaluate Lightweight Flagship LLMs: Lessons from a Regular Heptagon Construction Problem

    In the past two years, large language model product lineups have almost universally converged on a layered structure: a full flagship that pushes the ceiling of capability, and a smaller variant that trims redundancy to optimize for low latency and low cost. The trouble is that buyers routinely treat latency and throughput as proxies for quality, even though they are nearly orthogonal to whether a task is actually done right. This article argues that a better test for lightweight models is one with long step chains, objectively verifiable results, and distinctive failure modes—and it uses a regular heptagon compass-and-straightedge construction problem (with a neusis/marked-ruler constraint) as a worked example. It covers task design as a verifiable specification, the metrics worth recording, three failure modes (hard failure, false success, and process degradation), how to automate the judge with geometric and step-level assertions, and three practical lessons for model selection.

    1. Where Lightweight Flagship Models Fit: Cheap, Fast, but Trustworthy

    Over the past two years, LLM product lineups have almost universally converged on the same layered structure: a full flagship that pursues the ceiling of capability, and a smaller variant that trims some redundancy and focuses on low latency and low cost. The latter usually carries suffixes like “lightweight,” “fast,” or “Flash-class,” and typically serves high-frequency calls, real-time interaction, batch structured extraction, and completion or rewriting embedded in IDEs and command-line tools.

    The problem lies precisely in this positioning. The cost advantage of lightweight versions is structural, so they must naturally sacrifice some capability. The most common mistake buyers make is treating two easily observable metrics—time to first byte and output throughput—as proxies for quality. These two metrics are indeed easy to measure and strongly correlated with user experience, but they are almost orthogonal to “whether the task can be done correctly.” A model can stably produce a mathematically completely wrong yet beautifully formatted answer at over a hundred tokens per second.

    Therefore, the evaluation question for lightweight models should be rewritten as: under the assumption that it is necessarily weaker, is it still reliable enough to be placed in a production pipeline? This demands that evaluation tasks have three properties: a long chain of steps, results that can be objectively judged, and distinctive failure modes. The geometry construction problem introduced below satisfies all three at once.

    2. Why “Toy Tasks” Hold the Best Filters

    In developer communities, “build a page with HTML” was once the classic model warm-up question. It became popular because the barrier to entry is low and the result is intuitive—quality is obvious at a glance. But this kind of task has seriously depreciated today: the density of related code in training corpora is extremely high, so models only need to reproduce it in a retrieval-like fashion to produce a beautiful result. At the same time, it lacks a verifiable definition of correctness—whether a page looks good is subjective, and whether functionality matches the spec is often not carefully checked.

    An ideal evaluation task should be designed in reverse: it looks unremarkable, it is unambiguous to judge, and it is full of pitfalls to execute. The regular heptagon construction problem is exactly such a sample.

    Mathematically, the regular heptagon is a classic example of a non-constructible figure with compass and straightedge. According to the Gauss–Wantzel theorem, a regular n-gon is constructible with compass and straightedge if and only if n is a product of a power of 2 and some number of distinct Fermat primes, and 7 is a prime outside the Fermat primes (3, 5, 17, 257, 65537), so it fails the condition. In other words, when only an unmarked straightedge and a compass are allowed, the regular heptagon is mathematically non-constructible.

    For this very reason, the model must correctly understand the purpose of introducing the “marked ruler” (a ruler with a fixed mark, i.e., the neusis, or “sliding ruler” construction known since the Hellenistic period) in the problem statement: the extra mark length provides one constraint that exceeds the power of compass and straightedge, thereby making the problem solvable. If the model ignores this premise and forces the construction with compass and straightedge, it is already mathematically wrong. If it introduces steps, auxiliary lines, or extra constraints never given in the problem to “make up” the result, it is already wrong in terms of instruction following. These two error paths are exactly the two most common degradation patterns of lightweight models, and the discriminating power of the problem comes from precisely this.

    3. Evaluation Task Design: Writing the Problem as a Verifiable Spec

    The prompt of the original test revolves around a clear construction procedure: take a square and construct the perpendicular bisector of its base; with the bottom-right vertex as center and the diagonal as radius, draw a circle; then, keeping the marked ruler flush against the bottom-left vertex, slide it so that the first mark travels along that circle until the second mark passes through the perpendicular bisector of the base; then complete the usual steps of finding the center via perpendicular bisectors; and finally, fill in all the omitted compass-and-straightedge steps, keep traces only where necessary, and provide an animated GIF export.

    A qualified task prompt should contain five categories of elements, and missing any one introduces ambiguity noise:

    • Goal definition: what to draw and what the byproducts are (the regular heptagon plus a playable process animation).
    • Construction premises: which tools are allowed (marked ruler + compass) and what the mark length is (the length of the diagonal).
    • Step-by-step constraints: how the input points, centers, radii, and intersection points of each step are produced, in an order that cannot be shuffled.
    • Visualization constraints: which traces must be visible and which must not appear (only necessary traces retained).
    • Delivery format: a single file that runs offline, can export an animation, and is parameterized (number of sides and radius adjustable).

    Among these, the requirement that “traces are retained only where necessary” is the most easily underestimated, yet it is highly discriminating: it forces the model into an engineering problem of “state management + lifecycle control” rather than pure graphics drawing. A implementation that can only draw static figures will lose points on this requirement no matter how beautiful it is.

    4. Execution Environment and Observation Metrics

    Evaluation should not happen in a chat box, but in a real intelligent coding tool or command-line agent, because only in this environment can production-relevant behaviors such as tool calls, file reads and writes, and self-verification loops be observed. To ensure comparability, it is recommended to fix the following conditions: the same prompt, the same context length, the same maximum thinking budget, the same runtime (with Node/Python runtime versions pinned), and the same export method.

    The fields to record are roughly as follows:

    Dimension Observation point Notes
    End-to-end latency Total task time, time to first token The main selling point of lightweight models; must be quantified, not described by feel
    Thinking process shape Whether output is continuously streamed, whether there are long “think-only, no output” stretches Stuttering output is perceived as freezing in interactive products
    Mathematical correctness Whether the central angle equals 2π/7, whether the radius is consistent, closure error Hard metric, judgeable by script
    Step alignment Whether each step in the prompt can be found in the code Hard metric; needs manual mapping plus semi-automated cross-checking
    Visualization constraints Whether trace retention matches the “only where necessary” requirement Semi-subjective; can be scored on a range
    Engineering usability Whether it runs offline, is parameterized, and handles abnormal inputs robustly Determines whether it can go straight into production

    5. Three Failure Modes: Hard Failure, False Success, and Process Degradation

    In real testing, the most valuable finding is not “who got it right,” but the structure of the failures.

    The first type is hard failure: the model cannot complete the construction, or gives a geometrically obviously wrong result (the regular heptagon is drawn as an approximate regular polygon, or there is a visible gap at the closure point). This kind of failure is easy to identify and easiest to catch in evaluation; it is within the expected behavior of lightweight models.

    The second type is false success, and this is the most dangerous. The artifact appears to fully satisfy the requirements: the page renders a neat heptagon, the animation plays, and even the screenshot is hard to distinguish from the correct result. But if you compare the construction process point by point, you find that the model’s implementation does not match the path described in the problem—it quietly adds steps that do not exist in the problem statement, such as drawing an extra arc, introducing an undefined auxiliary point, or reverse-solving angles with an approximation algorithm, thereby bypassing the core neusis constraint and achieving the surface result by swapping out the goal.

    The harm of false success far exceeds that of hard failure. A hard failure surfaces in integration tests, while a false success flows into production carrying a “verified” label. To reliably catch it in evaluation, the only reliable method is to break the spec into assertions:

    1. // Geometric assertion of the neusis constraint: the key to distinguishing "real construction" from "faking the answer"
    2. function assertNeusis(D, P, Q, E, radiusE, mark, axisX) {
    3. const eps = 1e-6;
    4. assert(Math.abs(dist(P, E) - radiusE) < eps, 'P is not on circle E');
    5. assert(Math.abs(dist(P, Q) - mark) < eps, 'mark length not preserved');
    6. assert(areCollinear(D, P, Q), 'D, P, Q are not collinear');
    7. assert(Math.abs(Q.x - axisX) < eps, 'Q is not on the perpendicular bisector of the base');
    8. }

    As long as these four assertions fail to hold on any frame of the animation, it means the construction did not follow the path prescribed by the problem, and it should be judged as failing regardless of how neat the final figure is.

    The third type is process degradation, where the metrics look normal but the delivery experience is degraded. Typical manifestations are a discontinuous thinking process presented as choppy, stuttering output, or even outputting only the reasoning without producing a final answer. Such problems can be tolerated in batch-processing scenarios, but in real-time interaction scenarios they directly constitute a failure, so they must be recorded as an independent metric and cannot be masked by “the final delivery is correct.”

    6. Automating the Judge: Making Evaluation Sustainable

    A one-off manual comparison cannot support the pace of model iteration. The right approach is to write assertions as automated tests, export the artifacts to a machine-readable intermediate structure, and then verify them.

    Geometric correctness can be verified directly from the exported vertex sequence:

    1. import math
    2. from math import pi, atan2, hypot
    3. def check_heptagon(vertices, n=7, tol=1e-6):
    4. cx = sum(p[0] for p in vertices) / n
    5. cy = sum(p[1] for p in vertices) / n
    6. radii = [hypot(x - cx, y - cy) for x, y in vertices]
    7. assert max(radii) - min(radii) < tol, "inconsistent circumradius, not a regular polygon"
    8. angles = sorted(atan2(y - cy, x - cx) % (2 * pi) for x, y in vertices)
    9. gaps = [(angles[(i + 1) % n] - angles[i]) % (2 * pi) for i in range(n)]
    10. target = 2 * pi / n
    11. assert all(abs(g - target) < tol for g in gaps), "central angle is not 2π/7"

    The trace constraint can be indirectly verified by tagging drawing primitives, avoiding subjective judgment:

    1. // Keep visible traces only for necessary construction steps; other computations leave no mark
    2. function trace(point, reason) {
    3. if (!NECESSARY.has(reason)) return;
    4. trail.push({ x: point.x, y: point.y, reason, t: performance.now() });
    5. }

    On top of this judge, mapping each prompt step to an “existence check” (whether the corresponding function, constant, or comment appears in the implementation) yields a regression scorecard: geometric correctness, step alignment, visualization constraints, and engineering usability each carry a weight, and the model runs through it once per iteration so that score changes can be quantified. If evaluation is triggered frequently, it can be wired into a continuous integration pipeline: use a container platform for isolated execution environments, a logging service to collect the thinking process and artifacts, monitoring and alerting to set thresholds on score and latency regressions, and object storage to archive each artifact snapshot for retrospective comparison.

    7. Putting Selection into Practice: Three Lessons

    First, do not draw conclusions from a single task. One geometry problem can only test one class of capability (long-chain spatial reasoning plus strict instruction following). An effective evaluation set should be layered: math and geometry construction, state machines and concurrency control, large-scale refactoring and dependency analysis, structured output and format constraints, and information retrieval in long contexts. Keeping several machine-judgeable problems at each layer is what forms a credible profile.

    Second, treat cost–latency–quality as a triangle to balance, not a single-point optimum. The value of the lightweight version lies not in being smarter than the flagship, but in usable quality per unit cost. It is advisable to run a small-percentage canary on real traffic and route by business tier: high-value, long-chain tasks go to the full flagship; high-frequency short tasks and batch extraction go to the lightweight version. Use unified instrumentation to track task success rate, retry rate, and human takeover rate.

    Third, write engineering constraints into the prompt. In the task above, what often makes the real difference is usually not mathematical ability, but engineering constraints like “retain traces only where necessary,” “export an animated GIF,” and “run offline as a single file.” In production, the constraint-handling ability over prompts, tool calls, and output formats usually has a greater impact on final usability than a difference of a few percentage points in baseline reasoning ability.

    8. Conclusion

    Whether a lightweight flagship can actually be used cannot be answered by output speed or leaderboard scores. A problem that requires constructing a regular heptagon with a marked ruler is valuable because it forces the two most hidden defects into the open: pretending something non-constructible is constructible, and arbitrarily adding nonexistent steps in order to produce a result. The former is a capability boundary; the latter is a trust problem—and the latter is more fatal.

    The advice to engineering teams is straightforward: build a small task set with machine assertions for your core scenarios, measure “whether it strictly followed the spec” and “whether it gave the correct answer” separately, incorporate the observability of the thinking process into the metrics, and then run the model’s iterations as regressions across the dual axes of score and latency. Only a lightweight model selected this way can be confidently placed in a production pipeline.