Drawing a Regular Heptagon with a Marked Ruler: Lightweight vs. Frontier Model Comparison

    How do lightweight LLMs handle tasks that demand rigorous, step-by-step reasoning? This article uses "constructing a regular heptagon with a two-mark ruler" as a benchmark to compare a lightweight flagship model against a frontier model. The test evaluates not only whether the generated HTML produces a correct heptagon, but whether the model actually follows the specified construction procedure. We cover task specifications, acceptance criteria, reproduction steps, single-run results, and a practical checklist for model selection.

    Introduction: Why This Task as a Selection Reference

    Lightweight models are cheap to deploy and fast to respond, and more companies are putting them into production. Whether they can handle tasks requiring rigorous reasoning determines which scenarios they can cover. This article uses “constructing a regular heptagon with a two-mark ruler” as the test task, because it requires the model to understand geometric principles, follow specified steps, and express the process as runnable code. Typical code-generation problems only check whether the output runs; this task additionally checks whether the model followed the given procedure.

    The target audience is technical leads and developers who need to choose between models. The scope is limited to one geometric construction task and a single generation run, and does not represent overall model capability across other tasks.

    Task Specification and Parameters

    The original problem statement did not provide enough numerical parameters for direct reproduction. The table below lists the test specification. Parameters marked “fixed by the test designer” must be calculated according to the chosen construction method and written into the prompt before formal testing. This article does not provide unverified values.

    Parameter Value Notes
    Circumradius R 1 The heptagon is inscribed in the unit circle centered at the origin
    Heptagon side length s 2·sin(π/7), ≈ 0.867767 Used for acceptance checks
    Central angle between adjacent vertices 2π/7, ≈ 51.4286° Used for acceptance checks
    Distance d between the two ruler marks Fixed by the test designer Must match the chosen construction method and be included in the prompt
    Auxiliary circle radius Defined by the construction method The prompt must state how the center coordinates and radius are computed
    Acceptance tolerance ε 1×10⁻³ Maximum deviation of side lengths and central angles

    The acceptance procedure is as follows. After running the model-generated HTML, read the seven output vertex coordinates and compute the distances between adjacent vertices and the central angles. The figure is judged correct only if the maximum deviation of all seven sides from the theoretical values is below ε. A correct figure does not imply a correct procedure, so the construction steps must be checked separately.

    Prompt, Environment, and Versions

    The following information must be completed and verified before publication.

    • Full prompt text: [to be attached]
    • Model versions, API, and call parameters (temperature, max output tokens): [to be determined]
    • Test date and platform: [to be determined]
    • Number of retries and failure handling: [to be determined]
    • The two models are labeled A and B. If vendor information is restricted by confidentiality agreements, the reason must be stated in the article.

    Reproduction Steps

    1. Fix the parameters from Section 2 and send the prompt together with the parameter table to the model.
    2. Run each model at least 5 times, saving the complete output, elapsed time, and HTML file for each run.
    3. Open the HTML in a browser, record whether the page throws errors, and export a GIF or animation frames.
    4. Run the vertex acceptance script to compute deviations in side lengths and central angles.
    5. Review the code to check whether the construction includes the logic for “sliding the marked ruler” and “checking whether the mark line passes through the perpendicular bisector.” If the code places points directly at fixed angles or assembles the figure through extra auxiliary lines, record it as a procedure violation.
    6. Have two reviewers score independently; disagreements are resolved by a third reviewer.

    Single-Run Results

    The results below come from a single test, with each model run exactly once and no repetitions.

    Dimension Model A (lightweight flagship) Model B (frontier)
    Instruction following Completed the sliding step and subsequent construction as specified Figure correct, but construction procedure inconsistent with the prompt
    Reasoning depth Long silent phase during reasoning Fast generation, no visible logical derivation
    Code quality Page runs; interactive logic matches the steps Page runs; complete structure
    Time (single run) ~30–40 minutes ~20 minutes
    Reproducibility Not verified Not verified

    Model A took roughly 1.5–2× as long as Model B. Whether this difference comes from a longer reasoning budget or additional verification steps cannot be confirmed from this test.

    Failure Case: Where Model B Deviated

    Model B’s page does draw a heptagon. The problem lies in how. The code contains none of the logic required by the prompt—no sliding of the mark line along the auxiliary circle, and no check of whether the second mark line passes through the perpendicular bisector of the base. The figure is produced by an approximate method or by extra auxiliary lines.

    The judgment is based on two checks. First, whether the code contains iterative solving or numerical判定 for the sliding position. Second, whether the vertices are derived from the construction steps or written directly from angle formulas. If the latter holds, the output should be judged as a procedure violation even if it passes vertex acceptance. This method applies to other models as well.

    How Model A Was Validated

    Model A’s output passed vertex acceptance, and the code contains logic corresponding to the construction steps. However, this test did not repeat the run, so it cannot be concluded that the model produces the same result every time. The judgment of “fully following the prompt” applies only to this single output. Repeated runs are needed to determine stability.

    Boundaries of the Conclusion

    This test only shows that, under the conditions of one geometric task and a single generation, Model A’s output satisfied the step constraints while Model B’s output did not implement the specified procedure. The difference cannot be generalized as an overall capability gap, nor does it prove that lightweight models are equally reliable on other complex tasks.

    Model Selection Checklist

    When choosing a model, organizations can work through the following questions.

    Check item Question to answer
    Task complexity Does the task require multi-step reasoning, or is it mostly pattern matching?
    Latency requirements Can users tolerate responses taking tens of minutes, or is sub-second needed?
    Cost Does the per-call cost multiplied by the expected call volume fit the budget?
    Reproducibility Is the output stable across repeated runs with the same input?
    Cost of failure Will an incorrect output cause business loss or compliance risk?
    Verification means Are there automated scripts or human review to judge output correctness?

    Conclusion and Recommendations

    First, run repeated tests on your own business tasks. Run each candidate model at least 5 times, record the pass rate, latency, and failure types, and only then decide whether to adopt it.

    Second, broaden the task sample. Beyond geometric construction, cover code generation, mathematical derivation, and long-text understanding, so that conclusions from a single task are not overgeneralized.

    Third, choose models by scenario. For latency-sensitive, simple tasks, prioritize fast, low-cost models. For offline, complex tasks requiring high-fidelity logic, a slower model is acceptable—but pair it with automated verification to ensure output reliability.