Evaluating Code Reasoning in Lightweight Flagship Models with a Regular Heptagon Neusis Animation
This article presents a reproducible benchmark for evaluating code-reasoning capability in lightweight flagship models. Using a dynamic neusis (two-mark ruler) construction of a regular heptagon as the test case, it explains why a task with a single correct answer can still expose models that only produce plausible-looking output. It covers the underlying geometry, a low-ambiguity evaluation prompt, a four-dimensional scoring rubric, an evidence-chain approach to catching false positives, and a minimal evaluation pipeline with logging and monitoring. It closes with selection guidance, metric definitions, and applicability boundaries for practitioners choosing a model for multi-step reasoning, code generation, and frontend visualization.
Conclusion first: judge process evidence before the final output
Here is the selection conclusion up front. If a business task involves multi-step reasoning, code generation, frontend visualization, or long chains of tool calls, then choosing a lightweight flagship model cannot rely only on “whether it finally generates a runnable page.” You have to check whether the model strictly follows instructions, whether it preserves an auditable process, and whether it can produce stable output under strict constraints.
A regular heptagon two-mark (neusis) animation is well suited to this kind of capability discrimination. It has a definite correct answer, but it is hard to fake by matching against a memorized template. The rendered images may look very similar, yet the construction process can completely miss the point.
This evaluation method fits code generation, geometric animation, visualization verification, and multi-step instruction-following scenarios. It does not fit pure text summarization, subjective aesthetic design, or open-ended tasks with no visual verification. For those non-applicable cases, you should switch to other evaluation methods; otherwise you will map engineering metrics onto the wrong capability dimension.
What is a lightweight flagship model, and why evaluate code reasoning?
A lightweight flagship model usually refers to a model that balances cost, latency, context length, and reasoning ability. It is neither the cheapest small model nor the most expensive top-tier model; the goal is to provide “good enough” complex-processing capability across most business tasks.
The focus of code-reasoning evaluation is not whether a model can write a simple piece of JavaScript, but whether it can reason stably when multiple constraints coexist: whether it understands the geometric relationships correctly, follows the step order, produces runnable code structure, keeps animation timing reasonable, and yields output that can be re-checked.
Why can a neusis animation distinguish false positives? Because it simultaneously demands a mathematical model, geometric constraints, code implementation, and visualization verification. A model that relies only on pattern matching to generate something that “looks like a regular heptagon” will easily give itself away in the steps, parameters, and process evidence.
Why does a regular heptagon animation filter out false-positive models?
Whether a regular polygon is constructible with compass and straightedge depends on whether the number of sides is a product of Fermat primes multiplied by a power of 2. The number of sides of a regular heptagon, 7, is not a Fermat prime, so it is a classic figure that cannot be constructed with compass and straightedge. Completing it requires a stronger tool: a two-mark ruler, that is, a straightedge with two marks at a fixed distance.
For reproducibility, it is recommended to fix the following normalized geometric parameters. During actual rendering, multiply all lengths by 100.
- Build a square with side length 1. Lower-left corner A=(0,0), lower-right B=(1,0), upper-right C=(1,1), upper-left D=(0,1).
- Draw the perpendicular bisector of the base edge L: x=0.5.
- With B as center and radius equal to the diagonal length √2, draw circle Γ: (x-1)^2+y^2=2.
- The two-mark ruler edge passes through A, and the two marks are separated by d=1. Slide the ruler so that:
- one mark P lands on circle Γ;
- the other mark Q lands on the line x=0.5;
- Q lies between A and P;
- the unique valid pose with 0°<θ<90° is taken, where θ is the angle between the ruler and the positive horizontal direction.
Under these parameters, the solution is θ≈51.428571°, that is, 360°/7. The corresponding positions are approximately:
- Q≈(0.5,0.6268)
- P≈(1.1239,1.4088)
The angle θ satisfies the relation 8cos³θ+4cos²θ-4cosθ-1=0. This is exactly the irreducible cubic relation associated with the central angle 2π/7 of a regular heptagon. In other words, the neusis step is not decoration; it actually does the work of solving for the key angle.
Next, determine the final heptagon:
- With B as center and radius R=√2.
- Using the horizontal direction as the reference, take θ as the central-angle step.
- Generate seven vertices: V_k = B + R(cos(π+kθ), sin(π+kθ)), k=0,1,…,6.
- Connect V0 through V6 in order and close the shape to obtain a regular heptagon.
It should be noted that the above is one reproducible illustrative construction path, not the only one. If a model uses a different neusis construction, it must also state in the page, logs, or exported files that the path is non-unique, and provide the fixed parameters, the angle-solving process, and the closure error. During evaluation, you cannot accept only “the final figure looks right”; you must also accept that “the process parameters can be re-checked.”
How to design low-ambiguity, reproducible evaluation instructions?
Evaluation fairness depends first on instruction clarity. It is recommended to write the prompt as an executable, acceptance-testable, artifact-producing engineering task rather than letting the model improvise. Below is a reproducible evaluation instruction skeleton:
Build with a single HTML file: an animated demonstration of the entire process of constructing a regular heptagon with a two-mark ruler.No external dependencies, no CDN, no network requests, no external images or fonts.Fixed initial geometry:1. Build a square with side S=100, A=(0,0), B=(100,0), C=(100,100), D=(0,100).2. Show the perpendicular bisector of the base edge x=50.3. With B as center and radius 100√2, draw circle Γ.Animation steps and durations:1. 0-2 s: show the square, the base perpendicular bisector, and circle Γ.2. 2-6 s: show the ruler edge passing through A, with the two marks separated by d=100.Slide the ruler so that mark P is on circle Γ, mark Q is on x=50, and Q lies between A and P.Stop at the unique pose with 0°<θ<90°.3. 6-9 s: record θ=atan2(P.y, P.x) and, centered at B, generate the first ray.4. 9-12 s: using θ as the step, rotate successively to generate V0 through V6.5. 12-14 s: close the heptagon and highlight the seven vertices.Compass-mark rules:1. The initial circle, perpendicular bisector, and final heptagon may be retained.2. Temporary trial circles and trial rays must fade out after their corresponding step ends.3. No unrelated marks that would interfere with the final judgment may remain.Export acceptance:1. The page must provide an "Export GIF" button.2. Clicking it must generate a GIF of at least 6 seconds, 720p, 10 fps.3. If export fails, the page must display an error code and failure reason.Verifiable fields:Expose theta_deg, vertices, step_log, console_errors to window.__EVAL__.Acceptance conditions:1. Single-file HTML, no external dependencies.2. θ error no more than 0.05 degrees.3. Vertex radius relative error no more than 0.5%.4. Heptagon closure error no more than 0.5 pixels.
The more precise the instruction, the more it strips away the uncertainty introduced by improvisation. The evaluation focus is no longer “whether the model happened to generate a runnable page” but “whether the model truly understood and executed each constraint.”
A four-dimensional scoring rubric: from “does it work” to “is the process trustworthy”
For this kind of complex generation task, it is recommended to use four-dimensional scoring rather than a single pass/fail judgment. Each dimension can be scored from 0 to 5 and then multiplied by a weight.
| Dimension | Weight | 0-5 scoring guidance | Evidence fields | Suggested passing line |
|---|---|---|---|---|
| Instruction following | 40% | 0 = completely off-topic; 5 = completes all specified steps exactly, with no extra steps | prompt_version, step_log, unauthorized_steps, tool_calls | score ≥ 4.5 and unauthorized_steps = 0 |
| Geometric correctness | 30% | 0 = wrong figure; 5 = angle, radius, and closure error all within thresholds | theta_deg, radius_error, closure_error, vertex_count | θ error ≤ 0.05°, radius error ≤ 0.5%, closure error ≤ 0.5px |
| Code engineering quality | 15% | 0 = cannot run; 5 = single file, no external dependencies, clear structure, zero errors | file_count, external_requests, console_errors, run_status | single file, external requests = 0, console_errors = 0 |
| Output stability | 15% | 0 = timeout or interruption; 5 = complete output within acceptable time | duration_ms, timeout, retry_count, output_complete | complete output, P95 duration ≤ 180 s, timeout = 0 |
The role of this scoring rubric is to turn “I think it did a good job” into “it has evidence to support it.” In particular, instruction-following should be treated as the core dimension, because it is the key to distinguishing true understanding from superficial compliance.
Watch for false compliance: replace visual judgment with an evidence chain
The most dangerous failure in practice is not a model that cannot produce a result, but a model that produces a result that appears to meet the requirements. For example, an anonymous comparison model M2, version 2026.06-preview, generated an animation whose shape was close to a regular heptagon in one evaluation. If you look only at the final GIF, you might conclude it already passes.
But after a deeper check, the evidence chain shows that it did not execute the two-mark constraint at all. Below is a sample of the evidence fields for a false-compliance case:
| Evidence field | Value |
|---|---|
| run_id | 20260722-0915-case01 |
| model | Anonymous comparison model M2 |
| model_version | 2026.06-preview |
| prompt_version | heptagon-v3 |
| duration_ms | 83421 |
| render_status | success |
| gif_status | success |
| console_errors | 0 |
| step_log_count | 3 |
| unauthorized_steps | 2 |
| theta_deg | 58.0 |
| expected_theta_deg | 51.4286 |
| closure_error_px | 18.0 |
The error-step list can be recorded as:
- It did not execute the two-mark sliding instructed as “mark Q on the perpendicular bisector, mark P on the circle.”
- It added an “automatically compute the regular heptagon vertices” step on its own, bypassing the geometric constraint.
- The animation did not retain the necessary angle record, so the origin of θ cannot be verified.
- The final closure relied on approximate manual adjustment, and the closure error exceeded the threshold.
Visual comparison should also be part of the evidence:
- Screenshot comparison: fake_step3.png vs. reference_step3.png.
- GIF comparison: fake.gif vs. reference.gif.
- Comparison focus: whether the two-mark sliding exists from second 3 to second 6.
- Log comparison: whether key fields such as neusis_angle_search, mark_points, and line_circle_intersection exist.
- Conclusion output: check.json records pass=false, fail_reason=missing_neusis_constraint.
The harm of this kind of false positive is that it is hard to spot with the naked eye or a quick preview. Only by checking the construction logic, log fields, and parameter errors step by step can you avoid being misled by surface results.
How to turn the evaluation into a minimal executable pipeline?
Evaluation needs a reproducible, replayable runtime. Without environment isolation and log persistence, conclusions are hard to reuse. It is recommended to build a minimal evaluation pipeline with five stages: generation, rendering, verification, archiving, and monitoring.
A minimal directory can look like this:
heptagon-eval/prompt/heptagon-v3.txtrender/render.jscheck.jsexport_gif.shruns/{run_id}/model.htmlcheck.jsonscreenshot.pngoutput.gifrun.log
The generation stage is responsible for calling the model and obtaining the HTML. It is recommended to place the rendering stage in an isolated container to avoid malicious scripts, infinite loops, or resource exhaustion affecting the evaluation machine. Container rendering can tighten permissions as much as possible:
docker run --rm \--network none \--cpus 1 \--memory 1g \--pids-limit 256 \--cap-drop ALL \--security-opt no-new-privileges \--read-only \--tmpfs /tmp:rw,noexec,nosuid,size=256m \-v "$PWD/runs:/runs:rw" \heptagon-eval:0.1 \node /app/render.js --input /runs/model.html --out /runs
If Docker is not available locally, you can also run on the local environment:
node render.js --input runs/model.html --out runsffmpeg -framerate 10 -i runs/frame_%03d.png runs/output.gif
Open-source alternatives can include:
- Browser automation: Playwright or Puppeteer.
- GIF generation: FFmpeg.
- Local object storage: MinIO.
- Logging and metrics: Prometheus, Grafana, Loki, or ELK.
- Regression testing: pytest, Jest, or Node.js test scripts.
If you use cloud services, you are not locked to a single cloud vendor. Baidu AI Cloud, other public clouds, or a self-built cluster can all take on the roles of containers, object storage, logging, and monitoring. The key is to standardize the evaluation pipeline rather than depend on a specific product name.
Log fields are recommended to be unified as JSON Lines, including at least:
{"run_id": "20260722-0915-case01","stage": "render","model": "M2","prompt_version": "heptagon-v3","duration_ms": 83421,"timeout": false,"console_errors": 0,"external_requests": 0,"theta_deg": 51.4286,"closure_error_px": 0.12,"step_log_count": 7,"unauthorized_steps": 0,"artifact_html": "runs/20260722-0915-case01/model.html","artifact_gif": "runs/20260722-0915-case01/output.gif"}
Artifact naming is recommended to follow a unified rule:
{run_id}_{model_id}_{prompt_version}_{timestamp}.html{run_id}_{model_id}_{prompt_version}_{timestamp}.gif{run_id}_{model_id}_{prompt_version}_{timestamp}.check.json
Monitoring metrics are recommended to include at least:
| Metric | Meaning |
|---|---|
| pass_rate | Fraction of runs passing all acceptance conditions |
| render_success_rate | Fraction of pages that render successfully |
| gif_success_rate | Fraction of successful GIF exports |
| timeout_rate | Fraction of tasks that time out |
| console_error_rate | Fraction of runs with page console errors |
| p50_duration / p95_duration | Latency percentiles for generation or rendering |
| cost_per_pass | Average cost per successful pass |
| unauthorized_step_rate | Fraction of runs containing unauthorized steps |
Security must also be covered. Containers should use least privilege, with no unnecessary mounts or network access; logs that contain user input or secrets should be redacted; HTML, GIF, and logs in object storage should have lifecycle policies to avoid unbounded growth; and monitoring alerts should cover cost, failure rate, and abnormally long tasks.
Selection guidance: applicability boundaries, metric definitions, and risk notes
Recommendations for decision-makers can be summarized in three points:
- Judge capability by process, not just the final image. Focus on whether the model executes the two-mark steps as specified, whether unauthorized steps appear, and whether replayable logs are retained.
- Replace subjective impressions with quantifiable metrics. It is recommended to focus on latency, cost, success rate, instruction-following rate, and stability.
- Accumulate evaluation assets with a standardized pipeline. Evaluation scripts, prompts, acceptance thresholds, log formats, and artifact naming should all be version-controlled for easy regression testing.
Metric definitions can be given as:
| Metric | Definition |
|---|---|
| Latency | First-token latency and full-response latency, each measured at P50 and P95 |
| Cost | Cost per 1000 runs and cost per successful pass |
| Success rate | pass@1, pass@3, instruction-following rate, geometric pass rate |
| Stability | Pass rate over repeated runs, timeout rate, render failure rate |
| Process fidelity | step_log completeness, number of unauthorized steps, proportion of auditable logs |
Applicability boundaries are also important:
- Suitable: code generation, frontend animation, multi-step reasoning, visualization verification, tool-call chains, and tasks that require strict instruction-following.
- Not suitable: pure text summarization, subjective aesthetic design, open-ended creative writing, and simple Q&A with no visual verification.
- It is not suitable to use this evaluation to directly infer a model’s overall capability across all scenarios; it is better suited as a specialized filter for complex code-reasoning and process-following capability.
Ultimately, the value of a lightweight flagship model lies not in “whether it can quickly produce an answer that looks complete” but in “whether it can stably complete complex tasks under strict constraints.” The significance of the regular heptagon neusis animation evaluation is precisely to turn this question from a subjective feeling into a quantifiable, replayable, and comparable engineering practice.
