Benchmarking Lightweight Large Language Models on Geometric Drawing: A Practical Two-Measurement Ruler Method
This article presents a practical, reproducible benchmark for evaluating whether lightweight large language models can perform reliable structured reasoning. Instead of relying on parameter counts, vendor names, or subjective impressions, the author argues for executable and verifiable tasks. Using the two-measurement ruler construction of a regular heptagon as a case study, the article walks through benchmark design, environment and model-version pinning, prompt templates, a reference scoring script, tiered grading, failure-mode troubleshooting, model-selection guidance, and compliance requirements. Developers will learn how to separate fluent output from reliable reasoning, how to record the metrics that matter (parameter size, latency, hardware, cost, context length, temperature, seed, and task quality), and how to reach defensible engineering decisions about lightweight versus larger models.
Conclusion First
Whether “lightweight equals weak reasoning” cannot be judged by parameter count, vendor name, or a single subjective impression alone. A more reliable evaluation approach should record, at the same time, parameter scale, inference latency, hardware conditions, unit cost, context length, temperature and random seed, and task-completion quality. For large model application developers, it is not advisable to keep using overly broad tests like “generate a webpage,” because a model may produce a page that looks complete but cannot actually run or contains logical errors. Such tests make it hard to distinguish “fluent expression” from “reliable reasoning.”
A more effective approach is to use an executable, verifiable, reproducible geometric construction benchmark. This article uses the “two-measurement ruler construction of a regular heptagon” as an example to show how to design the task, how to pin the environment, how to write the prompt, how to run the code, how to score in tiers, and how to identify failure modes.
The conclusion up front: if a lightweight model can stably output a runnable page in a fixed environment and provide reasonable geometric constraints, approximate calculations, and rendering results, it shows strong potential for structured reasoning. Conversely, if it can only generate natural-language descriptions or surface-level code, that cannot be taken as proof that its reasoning is reliable.
1. Benchmark Design: Two-Measurement Ruler Construction of a Regular Heptagon
1.1 Why a Regular Heptagon
A regular heptagon cannot be completed in a single pass with a traditional unmarked straightedge and compass. It usually requires approximate construction, numerical calculation, or a more complex constraint-solving process. It distinguishes models’ reasoning depth better than drawing a triangle, square, or circle, because the model must not only “draw it” but also handle angles, side lengths, closure conditions, coordinate transformations, and error control.
The common problem with “generate a webpage” tests is that the model may produce a structurally complete HTML page without actually solving the geometric problem. For example, the page may contain a canvas, but the vertex coordinates are generated arbitrarily; it may display the words “regular heptagon,” but the figure does not close; or the code may depend on external resources and fail to run in an offline environment. For this reason, the benchmark upgrades the task from “generate a page” to “generate a verifiable geometric construction system.”
1.2 Task Definition
The benchmark task can be defined as follows:
Output a complete HTML file that does not depend on external network resources and, when opened in a browser, demonstrates the two-measurement ruler construction of a regular heptagon. Here, “two-measurement ruler” is defined in this benchmark as: allow the model to perform two ruler readings, and based on these two readings determine key points, radii, centers, or polygon vertices. The page must include the following capabilities:
- Use SVG or Canvas to draw a regular heptagon.
- Display the position, intersection points, center, radius, or auxiliary lines of the two ruler measurements.
- Automatically write the polygon vertices into
window.__polygon_pointsafter the page loads. - Not depend on external images, fonts, or scripts.
- Include a “regenerate” button that recomputes and re-renders when clicked.
- If the model cannot complete an exact construction, it may provide an approximate construction, but it must explain the approximation strategy within the page.
1.3 Scoring Criteria
This benchmark does not rely on manual subjective scoring; it uses programmatic scoring as much as possible. The scoring dimensions are:
| Dimension | Suggested Weight | Full-Score Condition | Common Deductions |
|---|---|---|---|
| Geometric correctness | 30 | 7 vertices, closed at both ends, interior angles close to the theoretical regular heptagon value | Duplicate vertices, missing closing edge, obvious angular deviation |
| Reasonableness of the two-measurement process | 20 | The position, intersection, or radius relationships of the two readings are explainable | Only concepts written, no coordinates or auxiliary lines |
| Runnability | 20 | No script errors on browser open, page renders | Blank page, errors, external dependencies |
| Reproducibility | 15 | Clear code structure, adjustable parameters, no hidden dependencies | Hard-coded and unverifiable, chaotic logic |
| Robustness | 15 | Still works after window resizing or clicking reset | Deformation after scaling, broken reset, event errors |
For a regular heptagon, each interior angle is theoretically about 128.57 degrees. Initial recommended thresholds can be set as: relative error of adjacent vertex side lengths no more than 2%, interior-angle error no more than 3 degrees, and 0 page script errors. These thresholds are not fixed industry standards; developers can adjust them to their business scenarios, but they must be written clearly in the evaluation report in advance.
1.4 Discrimination Evidence and Boundary Conditions
The evidence for discrimination does not come from fabricated scores, but from the task design itself: the model must simultaneously complete geometric reasoning, numerical approximation, front-end code generation, and verifiable output. If the model can only generate a text description, the difficulty is low; if it can generate a runnable page and correctly expose the vertex data, the discrimination is higher.
Boundary conditions must be stated in advance, otherwise they affect scoring stability:
- “Two-measurement ruler” should be clearly defined as two measurements or two constraint readings, not a vague historical anecdote.
- Using trigonometric approximations, a fixed viewport, or preset coordinates is allowed, as long as the result is verifiable.
- The model is not required to complete a rigorous mathematical proof.
- If the model refuses to execute or outputs an explanation without outputting HTML, it should be recorded as not completed.
- All evaluation inputs must be desensitized; including user privacy or sensitive business information is prohibited.
2. Reproduction Process: Environment, Model Version, and Prompts
2.1 Environment Versions
Before reproducing the evaluation, the runtime environment should be pinned. At minimum, record the following:
- Python 3.10.14
- Node.js 18.20.4
- Playwright 1.43.0
- Chromium 125.0.6422.60
- Pillow 10.3.0
- Operating system and CPU/GPU model
- CUDA, driver, or inference framework version, if using GPU inference
- Dependency lock files, such as
requirements.txtandpackage-lock.json
The above is an example environment; in practice, pinned versions should be used. Reproducibility requirements cover not only the model version but also the browser version, font rendering, script execution environment, and file paths.
2.2 Recording the Model Version
Each evaluation must record the model identity, at least including:
model_id: model name or pathrevision: model revision number, commit hash, or snapshot versiontokenizer_revision: tokenizer version, if applicablequantization: whether quantized and the quantization methoddtype: for example fp16, bf16, int8max_new_tokens: maximum generation lengthtemperature: recommended to set to 0 or a fixed low valueseed: random seedhardware: inference hardware information
Example record format:
model_id = your-model/model-namerevision = fill in the actual revision number or snapshot versiontokenizer_revision = fill in the actual versiontemperature = 0seed = 42max_new_tokens = 1200dtype = int8hardware = CPU or GPU model
Do not fill in unconfirmed model version numbers. If the version information is incomplete, mark it as “not reproducible.”
2.3 Prompt Template
The prompt must clearly require the model to output a complete HTML file, not explanatory text. An example is as follows:
You are a geometry reasoning and front-end rendering assistant. Output a complete HTML file that does not depend on external network resources.Task: Use the two-measurement ruler to construct a regular heptagon.Definition of "two-measurement ruler": allow two ruler readings, and based on these two readings determine key points, the center, the radius, or polygon vertices.Requirements:1. Use SVG or Canvas to draw the regular heptagon.2. Display in the page the position, intersection, center, and radius of the two ruler measurements.3. After the page loads, automatically write the polygon vertices into window.__polygon_points.4. If the figure is an approximate construction, explain the approximation method in the page.5. The page must include a "regenerate" button.6. Do not reference external images, fonts, scripts, or network APIs.7. Output only the HTML code, no additional explanation.
The more specific the prompt, the more stable the evaluation. If the model output contains explanations, Markdown code blocks, or extra text, HTML should be extracted uniformly in a post-processing step to avoid scoring differences.
2.4 Reference Code
The following is a reproducible reference script. In practice, replace call_model with your own model-calling logic.
import osimport jsonfrom playwright.sync_api import sync_playwrightMODEL_ID = os.getenv("MODEL_ID", "your-model/model-name")REVISION = os.getenv("REVISION", "fill in the actual revision")PROMPT = r"""You are a geometry reasoning and front-end rendering assistant. Output a complete HTML file that does not depend on external network resources.Task: Use the two-measurement ruler to construct a regular heptagon.Definition of "two-measurement ruler": allow two ruler readings, and based on these two readings determine key points, the center, the radius, or polygon vertices.Requirements:1. Use SVG or Canvas to draw the regular heptagon.2. Display in the page the position, intersection, center, and radius of the two ruler measurements.3. After the page loads, automatically write the polygon vertices into window.__polygon_points.4. If the figure is an approximate construction, explain the approximation method in the page.5. The page must include a "regenerate" button.6. Do not reference external images, fonts, scripts, or network APIs.7. Output only the HTML code, no additional explanation."""def call_model(prompt: str) -> str:"""Replace with the real model call.The return value must be a complete HTML string."""return os.environ.get("MODEL_OUTPUT", "<!doctype html><html></html>")def save_html(html: str) -> None:with open("artifact.html", "w", encoding="utf-8") as f:f.write(html)def run_browser_check():with sync_playwright() as p:browser = p.chromium.launch()page = browser.new_page(viewport={"width": 1200, "height": 800})page.goto("file://" + os.path.abspath("artifact.html"))page.wait_for_timeout(800)points = page.evaluate("() => window.__polygon_points || []")errors = page.evaluate("() => window.__errors || []")browser.close()return points, errorsdef polygon_metrics(points):if len(points) < 3:return {"points": len(points), "valid": False}edges = []for i in range(len(points)):x1, y1 = points[i]x2, y2 = points[(i + 1) % len(points)]edges.append(((x2 - x1) ** 2 + (y2 - y1) ** 2) ** 0.5)return {"points": len(points),"edges": edges,"closed": True}def main():html = call_model(PROMPT)save_html(html)points, errors = run_browser_check()metrics = polygon_metrics(points)result = {"model_id": MODEL_ID,"revision": REVISION,"script_errors": errors,"metrics": metrics,"reproducible": bool(MODEL_ID and REVISION)}with open("result.json", "w", encoding="utf-8") as f:json.dump(result, f, ensure_ascii=False, indent=2)print(result)if __name__ == "__main__":main()
This script accomplishes the minimal closed loop: generate the page, open the page, read the vertices, and record the results. To evaluate the regular heptagon more rigorously, you can further add interior-angle calculation, side-length consistency calculation, center consistency calculation, and screenshot comparison. For engineering implementation, it is recommended to save at least prompt.txt, artifact.html, result.json, run logs, and browser screenshots, to make reproduction and auditing easier.
3. Tiered Scoring: Making Results Comparable
To make results comparable across different models, it is recommended to use tiered scoring rather than a single total score.
| Grade | Suggested Total Score | Judgment | Applicable Conclusion |
|---|---|---|---|
| A | 90–100 | Geometry, running, and process are all complete | Can be a candidate model advancing to the next round |
| B | 75–89 | Runs, but has minor errors or insufficient explanation | Can be kept, requires manual review |
| C | 60–74 | Page can be generated, but geometry or process has obvious problems | Use with caution |
| D | 0–59 | Cannot run, missing vertices, or serious geometric errors | Not recommended for complex reasoning scenarios |
The key to tiered scoring is fixing thresholds in advance. For example:
- If the number of script errors is greater than 0, judge it directly as D.
- If
window.__polygon_pointsis empty, judge it as D. - If the vertex count is not equal to 7, judge it as C or D.
- If the figure is not closed at both ends, judge it as D.
- If 7 vertices can be generated but the angle deviation exceeds 5 degrees, judge it as C.
- If the two-measurement ruler information is visible and consistent with the figure logic, the “process reasonableness” score can be increased.
If used for model selection, it is not advisable to look only at the total score. Developers should focus on the failure type: whether it is a geometric error, an engineering error, a prompt-understanding error, or a lack of model capability. Different error types have different impacts on business decisions.
4. How to Choose a Lightweight Model: Latency, Cost, and Reasoning Quality
The answer to “does lightweight mean weak reasoning” depends on the evaluation approach. Fewer parameters do not necessarily mean weak reasoning, and more parameters do not necessarily mean strong reasoning. Whether a model can complete tasks under fixed hardware, fixed latency, and fixed cost is the core of engineering selection.
It is recommended to build a comparison table like the following. All values must come from your own measured environment and should not directly cite unreproduced data:
| Evaluation Dimension | Lightweight Model Focus | Mid/Large Model Comparison | Recommended Recording Method |
|---|---|---|---|
| Parameter scale | Whether suitable for edge deployment | Whether suitable for server-side batch tasks | Record total parameters, active parameters, context length |
| Latency | First-token latency, full-output latency | Throughput under concurrency | Record P50, P95, average elapsed time |
| Hardware conditions | CPU, VRAM, memory | GPU model, parallelism | Record device model and resource usage |
| Cost | Cost per request, deployment cost | Scaling cost, maintenance cost | Record token cost, machine cost |
| Reasoning quality | Whether it can complete the geometric drawing task | Whether it can handle complex multi-step reasoning | Use the same benchmark and the same scoring script |
| Stability | Impact of temperature and random seed | Whether it degrades under long context | Run repeatedly multiple times and measure variation |
When choosing a model, you can follow these principles:
- If the business is mainly template generation, summarization, classification, and simple code completion, a lightweight model may be sufficient.
- If the business involves geometry, mathematics, complex rules, tool calling, and long-chain reasoning, priority should be given to verifying the model’s stability on structured tasks.
- If latency and cost are sensitive, benchmarking should be used to determine whether a lightweight model can be downgraded for use.
- If the model only performs fluently on open tasks like “generate a webpage” but the actual output cannot run or cannot be verified, it should not be used as evidence of reasoning capability.
5. Failure Modes and Troubleshooting Checklist
5.1 Prompt Ambiguity
Common problem: the model does not know what the “two-measurement ruler” is, or misinterprets “two ruler readings” as two drawing steps.
Fix: clearly define the “two-measurement ruler” in the prompt, require the model to output the position, intersection, or radius of the two measurements, and write it into a verifiable variable.
5.2 Drawing Errors
Common problem: the model generates a canvas, but the coordinate calculation is wrong, causing the figure not to close or the side lengths to be inconsistent.
Fix: check whether window.__polygon_points contains 7 vertices, check whether the figure is closed at both ends, and calculate adjacent side lengths and angles. If the model uses an approximate construction, require it to declare the approximation method.
5.3 Unstable Scoring
Common problem: temperature changes or different random seeds cause result fluctuation.
Fix: set temperature=0 during evaluation, fix the seed, run multiple times, and compute the mean, minimum, and maximum.
5.4 Environment Differences
Common problem: it runs locally but cannot run in CI or an evaluation container.
Fix: pin the browser version, fonts, viewport size, and file paths; use a headless browser for uniform execution. Write all dependencies into lock files.
5.5 Output Format Contamination
Common problem: the model returns Markdown code blocks, explanatory text, or extra descriptions.
Fix: extract the HTML content in a post-processing step, or enforce “output only HTML code” in the prompt. If contamination still occurs, record it as a format failure.
5.6 External Dependency Leakage
Common problem: the page references external scripts, fonts, or images, causing it to fail in offline environments and potentially introducing security risks.
Fix: disable network requests; check whether the HTML contains http://, https://, external CDNs, or remote APIs.
5.7 Non-Reproducible Results
Common problem: the model version, environment version, prompt, or random seed is not recorded.
Fix: uniformly archive all evaluation artifacts, including input, output, environment, logs, and scoring results. Results without complete records should not be used as a basis for selection.
6. FAQ: How to Do It, How to Choose Models, and Common Mistakes
6.1 How Do I Run This Evaluation?
The minimal steps are: prepare a fixed environment, record the model version, send the unified prompt, generate the HTML, run it in a headless browser, read window.__polygon_points, calculate geometric metrics, and output the scoring result. It is recommended to save complete logs and page files for each evaluation.
6.2 How Do I Choose a Model?
Do not look only at parameter count. Compare latency, cost, hardware conditions, reasoning quality, and stability. For lightweight models, focus on whether they can complete verifiable tasks under low-resource conditions; for mid/large models, focus on their performance in complex tasks and long contexts.
6.3 What Are the Common Mistakes?
Common mistakes include: unclear prompts, figures not closing, wrong angles, missing verifiable interfaces, external dependencies, excessive randomness, not recording the model version, scoring without troubleshooting, and relying only on subjective judgment.
7. Risk Warnings and Compliance Requirements
Evaluation data must be desensitized; using real user data, privacy information, trade secrets, or unpublished business data is prohibited. Evaluation inputs should be reviewed to avoid including personal identity information, accounts, phone numbers, ID numbers, addresses, or other sensitive content.
For licensing compliance, confirm the usage licenses of the tested model, datasets, browser components, codebases, and graphic assets. If open-source models or third-party tools are used, record the license type and avoid using restricted content for commercial publication.
The reproducibility requirement should include at least: a fixed environment, fixed model version, fixed prompt, fixed scoring script, fixed output format, and retained run logs. If any of these is missing, mark it as a “reference result” rather than a “reproducible conclusion.”
Ultimately, whether a lightweight large model has weak reasoning should not be judged by slogans. Developers should make verifiable engineering trade-offs through clear benchmarks, executable code, tiered scoring, and complete risk records.
