Evaluating LLM Reasoning and Instruction Following with Complex Drawing Tasks
Early on, asking a model to generate an HTML animation from a single prompt was the standard way to probe its coding and reasoning ability. As model quality has improved overall, that class of "pelican riding a bicycle" test has become saturated: almost every mainstream model passes it, so it no longer separates anyone. This article uses a deliberately hard case — animating the neusis construction of a regular heptagon in HTML — to show what makes a test discriminative, why process fidelity (instruction following) matters more than a result that merely looks right, and how to score such tasks in a reusable way.
1. The bar is moving: from toy tests to hard gates
In the early days, a single prompt asking a model to produce an HTML animation page was the most common way to test its code generation and reasoning ability. Such tests became popular because they are intuitive, runnable, and produce an obvious result.
With the overall jump in model capability, however, this class of task — the “pelican riding a bicycle” level of difficulty — is close to saturation. Most mainstream models complete it easily, and its discriminating power has collapsed.
A test that no longer separates models has no evaluation value. A task with real screening power needs three properties: strong dependencies between steps, explicit constraints on every instruction, and verifiability of both the process and the artifact. In other words, we need to know not just whether a model can produce output, but whether it followed the prescribed method exactly.
2. Case design: why the neusis heptagon construction is a good litmus test
Here is the core of the prompt (abridged and structured):
Goal: implement an animation of the "two-marked ruler" (neusis) constructionof a regular heptagon in HTMLStep constraints:1. Take a square;2. Construct the perpendicular bisector of its bottom edge;3. With the bottom-right vertex E as center and the diagonal as radius, draw circle E;4. Using the two-marked ruler, keep it flush against the bottom-left vertex Dand slide the first graduation along circle E until the second graduationpasses through the perpendicular bisector of the bottom edge;5. Then locate the center in the conventional way with perpendicular bisectors,filling in all the straightedge-and-compass steps left implicit;6. The compass traces a moving path, but marks should remain only where necessary;7. Finally, add the ability to export the result as a GIF.
The value of this task lies in a key mathematical fact: a regular heptagon cannot be constructed exactly with a classical straightedge and compass (because 7 is not a Fermat prime), so the construction requires a “two-marked ruler” — a tool that lets you transfer a fixed length along a straight edge, going beyond what intuition about hand drawing would suggest. This creates three layers of difficulty:
- Mathematical barrier: the model must understand why the conventional straightedge-and-compass approach fails, why the marked ruler is needed, and the geometric justification for each step.
- Instruction fidelity: the prompt specifies the construction path, so the model must follow the recipe rather than improvise.
- Engineering implementation: deciding which motion trails to keep, the movement logic of the marked ruler, and GIF export all test code organization.
3. Two core evaluation dimensions
The most common mistake when evaluating this kind of task is to focus entirely on whether the final figure was drawn. A rigorous evaluation has two layers.
3.1 Result correctness
Check whether the geometric relationships hold: is the perpendicular bisector of the bottom edge accurate, was circle E drawn with the correct radius, does the heptagon actually close, is the center located sensibly? This layer is relatively objective and can be judged by eye and by measurement.
3.2 Process fidelity (instruction following) — the most important and most overlooked layer
This layer checks whether the model advanced strictly along the steps given in the prompt, or took a shortcut to produce something that merely resembles the target.
A typical trap is “cheating by completion”: some models do generate a page that looks like a regular heptagon, but tracing their construction logic reveals that they never executed the core constraint — sliding the marked ruler while keeping it flush against vertex D until the second graduation crosses the perpendicular bisector. Instead, they quietly introduce steps that do not exist in the prompt at all, forcing out the final figure by going around the hard part. The result is right; the process is fake.
This kind of defect is equally dangerous in real systems. A model that “fills in the blanks” with invented intermediate steps when information is incomplete, or that takes liberties with constraints instead of following them, tends to produce errors that accumulate unnoticed in long task chains. A model that faithfully executes the prescribed steps has more engineering value than one that lands on the correct answer.
4. How to observe a model’s reasoning process
Modern models generally support streaming output and chain-of-thought display, which gives a window into the process. Signals worth recording during testing include:
- Coherence of reasoning: does it advance smoothly, or does it stall frequently and backtrack repeatedly?
- The “thinks forever, outputs nothing” pattern: the model spends a large amount of time on internal reasoning without producing a result, and eventually either times out or delivers a partial artifact.
- The relationship between latency and quality: a run that faithfully completes every step and exports a usable GIF may take tens of minutes, while a model that “crashes” may produce a page faster yet still lose points on process fidelity.
So generation speed alone should not decide the winner. Speed is a user-experience metric; process fidelity and result correctness are capability metrics.
5. The capability boundary of lightweight models
One notable observation: smaller models positioned as “lightweight” are not necessarily weaker on tasks of this difficulty, and sometimes complete work that trips up much larger models. The difference often shows up in output speed rather than reasoning quality. This suggests:
- Capability does not scale linearly with parameter count. The quality of post-training and the degree of instruction-following tuning may matter more than raw parameter volume for complex tasks.
- Model selection should be scenario-driven. For latency-sensitive scenarios with relatively standard tasks, a lightweight model is a cost-effective choice. For multi-step dependencies with strict constraints, verify process fidelity first.
6. A practical evaluation methodology
Consolidating this practice into a reusable framework means scoring process and result with equal seriousness:
| Evaluation dimension | Suggested weight | What to score |
|---|---|---|
| Result correctness | 40% | Whether geometric relationships hold, whether the figure closes, whether the center is located accurately |
| Process fidelity | 30% | Whether the given steps were followed strictly, whether steps not defined in the prompt were added |
| Interaction implementation completeness | 20% | Whether perpendicular bisectors, arcs, and ruler movement advance according to the intended logic |
| Additional features | 10% | Whether GIF export works, whether motion trails are left only where they make sense |
Three methodological points follow from this:
- Build a tiered test set. Spanning from “toy level” (basic HTML) to “hard gate” (multi-step geometric construction, tightly constrained logic chains) creates a difficulty gradient and preserves discriminating power over time.
- Keep the process on record and make it reproducible. Save the full chain of thought, intermediate artifacts, and the final page so evaluation is traceable, rather than concluding from a final screenshot.
- Treat “did it cheat?” as a separate deduction. Flag completion methods that invent steps or substitute constraints independently, because this reflects reliability on real complex tasks more accurately than a plain failure does.
Conclusion
Evaluating a large model ultimately means evaluating its ability to complete a task under constraints, not merely to produce an answer. A hard multi-step task such as the neusis construction of a regular heptagon puts result correctness and process fidelity on the table at the same time, which helps identify models that look complete but have actually drifted. For developers and organizations, building a process-centered evaluation methodology is what makes it possible to select models that are both correct and faithful to the instructions they were given.
