How to Evaluate Lightweight Models on Complex Multi-Step Tasks

    In generative AI product lines, "lite" or "lightweight" models are usually defined by speed—smaller parameter counts, faster responses, and lower cost per call—which leads many teams to assume they must underperform full-size versions on complex work. This guide argues that speed only measures how quickly a model answers, not whether it actually completes the task as instructed. Using a constructive geometry example—building a regular heptagon with a neusis (marked-ruler) construction rendered in HTML—the article breaks down how to design evaluation items that test complex instruction-following, how to score both result correctness and process fidelity, and how to make the evaluation repeatable, traceable, and integrable into a CI/CD pipeline. It also covers common failure modes and practical advice for model selection and iteration.

    1. The Problem: Capability Gaps Hidden Behind a “Speed” Label

    In generative model product lines, a “lite” version usually means a smaller parameter count, faster responses, and a lower cost per call. It targets latency- and cost-sensitive scenarios such as high-concurrency Q&A, code completion, and content summarization. Because of this, people tend to define lightweight models by “fast” and assume they must be inferior to the full version on complex tasks.

    Real evaluations, however, often show a contrast. A lightweight flagship may come close to the flagship version on isolated single-point capabilities yet struggle noticeably on multi-step tasks. Other models produce output quickly with a timely first-token response but frequently drift from requirements on complex instructions. Speed only measures how fast an answer arrives; it says nothing about whether the answer actually completes the task as required.

    More concerning, many benchmark questions have been reused and openly discussed so often that a model may have encountered similar item types during training, which makes the scores a poor reflection of true generalization. Community tests such as “build a webpage,” while they present visual output directly, mostly assess the look of the page and place little constraint on the logical rigor of the task itself.

    So how do you design an evaluation method that both checks complex instruction-following and scores automatically? That question is unavoidable both for developers choosing a model and for teams iterating on one. This article uses a constructive geometry task as an example to break down the design principles, scoring dimensions, and engineering practices behind such an evaluation.

    2. Evaluation Design: Making Both the “Result” and the “Process” Verifiable

    1. Why Looking at the Result Isn’t Enough

    A common failure mode in complex tasks is this: the final result looks correct, but the path to it does not match the requirements at all. For example, when asked to construct a figure by a specified method, a model may skip the key steps and instead introduce methods of its own—methods not present in the prompt—to arrive at the answer.

    Such errors are hard to spot from a screenshot alone. The figure is indeed drawn, and a reviewer might even consider the task complete. But from an instruction-following standpoint, it has already deviated. For tasks that require strict compliance (such as data processing that must follow a fixed workflow, or operations executed by prescribed steps), this deviation is more dangerous than simply “failing to draw,” because it wears the guise of correctness.

    2. Four Traits of a Good Evaluation Item

    • Unique conclusion: The answer or geometric relationship can be judged precisely and does not depend on subjective aesthetics.
    • Rigid steps: The prompt clearly specifies the implementation path, and you cannot freely switch routes midway.
    • Long-range dependency: Each later step depends on the output of the previous one, so getting a local step right by chance cannot guarantee overall correctness.
    • Classifiable failures: It can distinguish among capability gaps, step substitution, and defective artifacts.

    Only items that satisfy all four can separate “whether the result is correct” from “whether the process is faithful” into two independent dimensions.

    3. Example: Constructing a Regular Heptagon with a Neusis (Marked-Ruler) Construction

    Mathematical background

    By Gauss’s findings, a regular heptagon cannot be constructed with compass and straightedge alone. But it can be constructed using a marked ruler (also called a “neusis” ruler) through an insertion construction. This is because such constructions can handle the cubic relationships that compass-and-straightedge tools cannot express, and constructing the regular heptagon happens to depend on exactly this relationship.

    The prompt

    Write an HTML page that constructs a regular heptagon using a neusis (marked-ruler) construction.

    1. Take a square ABCD, where A and B are the two endpoints of the bottom edge and C and D are the two endpoints of the top edge. Construct the perpendicular bisector of AB, and draw a circle centered at B with radius equal to the square’s diagonal.
    2. Using the neusis ruler: the ruler body always passes through vertex A, and the distance between two marks on the ruler is fixed and equal to the square’s diagonal. Slide the first mark along circle B until the second mark lands exactly on the perpendicular bisector of AB.
    3. Then, in the conventional way: construct a perpendicular bisector to locate the center, draw a circle with an appropriate radius, and step off chords equal to the known side length successively on the circle to obtain the vertices of the regular heptagon.
    4. Complete all omitted compass-and-straightedge steps.
    5. Animation requirement: during compass construction, show the trajectory of the pen tip, but leave traces only where necessary.
    6. Add a feature that can export the animation as a GIF.

    Why this item works

    • Verifiable result: The coordinates of each point and the equations of each circle and line can be written out precisely, and the geometric relationships can be verified by a program.
    • Rigid process: It specifies “the ruler passes through a fixed point, the spacing between the two marks is fixed, the first mark is on the circle, and the second mark is on the line (the perpendicular bisector).” Any detour is recorded as a deviation.
    • Long-range dependency: Locating the center and stepping off chords afterward all build on the key length obtained from the insertion construction. If the key length is wrong, the entire chain follows it into error.
    • Classifiable failures: Wrong geometry, non-conforming steps, missing animation trails, and a broken export feature each map to a distinct problem.

    3. Scoring Dimensions: Replace “Does It Look Right” with “Is It Correct”

    Dimension What it mainly examines Typical way to judge
    Result correctness Whether the insertion conditions are satisfied exactly, whether the vertices are evenly distributed, and whether the side lengths are equal Analytic computation and comparison against a tolerance
    Process fidelity Whether the construction follows the specified method and whether extra steps were introduced Compare the step record against execution logs
    Completeness Whether omitted steps are completed and whether elements such as perpendicular bisectors, arcs, and vertices are all present Inspect graphical objects and annotations
    Interaction and visuals Whether the animation is continuous and whether traces appear only where necessary Headless-browser screenshots or screen-recording comparison
    Extra features Whether GIF export works and whether the frame content is correct Trigger the export and parse the file
    Stability and efficiency Whether errors occur, plus first-token response, generation time, and whether retries were needed Collect call logs and timing data

    Result correctness and process fidelity are the core dimensions and should carry significantly more weight than visual performance. On one hand, what is being evaluated is instruction-following, not art design; on the other hand, a high weight on visual items easily lets “looks right” outweigh “does it right.”

    The specific weights for each dimension should be adjusted to the business scenario. If the page is mainly for human review, you can raise the weight of interaction and visuals; if the output enters an automated pipeline, then parseable structure and stable interfaces should be treated as hard gates.

    4. Engineering the Evaluation: Making It Repeatable and Traceable

    1. A Fixed Execution Environment

    • Pin the runtime into a unified image, fixing the browser version, rendering parameters, and time zone.
    • Hard-code the random seed so that differences between repeated runs of the same model come only from the model itself.
    • Use a unified input/output format so that how the prompt is written or the output spec does not affect results.

    2. Collecting Observable Data

    • Record the output cadence of the thinking process. Continuous output and silence during thinking are two different patterns, and both affect the real experience.
    • If the messages a model returns include visible reasoning text, keep it together with the call-chain logs.
    • Cases of “outputting only internal thoughts with no visible artifact for a long time” should be counted as a stability defect, not a capability defect.

    3. Logging and Replay

    • Save each run’s input, all messages, artifact files, screenshots, and timing.
    • Write all material into object storage for archiving, indexed by model and item tags, to make horizontal comparison and historical review easier.

    4. Reusable Automatic Validation

    Validation scripts can be split into a “geometry check” and a “structure check.” Sample code:

    1. import json
    2. import re
    3. import math
    4. from urllib.parse import urlparse
    5. TOL = 1e-3 # Geometry tolerance; adjust to the precision required by the prompt
    6. def check_insertion(page):
    7. """Geometry check: verify that the insertion construction exactly satisfies the prompt's conditions."""
    8. ctx = page.evaluate("""() => window.__geometry__ || null""")
    9. if not ctx:
    10. return False, "No verifiable geometry data exposed"
    11. # The fixed spacing between the two marks on the ruler should equal the square's diagonal
    12. seg = math.hypot(ctx["tick2"][0] - ctx["tick1"][0],
    13. ctx["tick2"][1] - ctx["tick1"][1])
    14. if abs(seg - ctx["diagonal"]) > TOL:
    15. return False, "Spacing between the two marks does not match the diagonal"
    16. # Whether the ruler body always passes through the fixed point
    17. for name, p in (("tick1", ctx["tick1"]), ("tick2", ctx["tick2"])):
    18. if not on_line(p, ctx["pin"], ctx["dir"], TOL):
    19. return False, f"{name} is not on the line through the fixed point"
    20. if dist(ctx["tick1"], ctx["circle_center"]) - ctx["circle_r"] > TOL:
    21. return False, "First mark does not lie on the circle"
    22. if not on_bisector(ctx["tick2"], ctx["a"], ctx["b"], TOL):
    23. return False, "Second mark does not lie on the perpendicular bisector of the bottom edge"
    24. return True, "Insertion conditions satisfied"
    25. def check_heptagon(page):
    26. """Result check: verify vertex distribution and side-length consistency."""
    27. pts = page.evaluate("() => window.__geometry__.vertices || []")
    28. if len(pts) != 7:
    29. return False, f"Vertex count is {len(pts)}, expected 7"
    30. center = (sum(p[0] for p in pts) / 7, sum(p[1] for p in pts) / 7)
    31. radii = [math.dist(p, center) for p in pts]
    32. if max(radii) - min(radii) > TOL:
    33. return False, "Distance from vertices to center is inconsistent"
    34. sides = [math.dist(pts[i], pts[(i + 1) % 7]) for i in range(7)]
    35. if max(sides) - min(sides) > TOL:
    36. return False, "Side lengths are inconsistent"
    37. return True, "Regular heptagon geometric relationships hold"
    38. def check_structure(page):
    39. """Structure check: verify extra requirements such as trails and the export feature."""
    40. issues = []
    41. if not page.locator("[data-role='arc-trace']").count():
    42. issues.append("Missing arc trail element")
    43. if page.locator("[data-role='stray-mark']").count() > 0:
    44. issues.append("Unnecessary stray marks present")
    45. btn = page.get_by_role("button", name=re.compile("export|GIF"))
    46. if btn.count() == 0:
    47. issues.append("Missing export entry")
    48. else:
    49. btn.first.click()
    50. suffix = page.evaluate("() => window.__last_export__?.name || ''")
    51. if urlparse(suffix).path.split('.')[-1].lower() != "gif":
    52. issues.append("Exported file is not in GIF format")
    53. return (not issues), issues

    on_line, on_bisector, dist, and similar are basic geometry helper functions that can be implemented in whatever stack you use. Once this validation is running, it can automatically re-run after every model update, turning the argument over whether a model is “good enough” into comparable numbers.

    5. Integrating with CI/CD

    • Trigger a full regression on every model-version or prompt-template update.
    • Compare changes in core dimensions such as result correctness and process fidelity, rather than only looking at visual screenshots.
    • Feed metrics such as latency and retry counts into monitoring and alerting, and notify promptly on abnormal fluctuation.
    • If the pipeline is deployed on Baidu AI Cloud, you can use a container platform to host the unified execution environment, use a logging service to aggregate run data, and combine it with monitoring and alerting to build a metrics dashboard.

    5. Common Failure Modes and How to Read Them

    Evaluations should focus on identifying the following kinds of problems:

    • Method substitution: Bypassing the specified steps and introducing methods not in the prompt to force a result. This is a serious defect in process fidelity.
    • Geometric approximation: Replacing exact construction with approximate values or simplified formulas. It looks fine on the drawing but fails numerical validation.
    • Defective artifacts: Missing trails, a broken export feature, or missing key annotations.
    • Long-range instability: A discontinuous cadence during thinking, or no visible output for a long time. This is related to both reasoning-chain length and scheduling strategy and should be tracked separately.
    • Format non-compliance: Inconsistent output structure that downstream scripts cannot parse reliably.

    When reading results, it is advisable to report “capability problems” and “artifact problems” separately. The former point to the model itself; the latter point to the integration approach (prompting, output parsing, tool wrapping), and their optimization paths are entirely different. Discussing them together distorts the conclusion.

    6. Guidance for Selection and Iteration

    1. Use a question set instead of a single item: A single question’s conclusion is highly random, so a baseline should be a set of items covering different failure modes.
    2. Put the process at the center: Prioritize whether the model executes step by step under constraints, rather than only whether the page looks good.
    3. Keep speed metrics separate: Present response speed and generation time alongside capability metrics so users can weigh them per scenario, rather than letting speed substitute for a capability judgment.
    4. Fix the baseline when iterating: Once the items and scoring method are set, keep their version fixed so that each update changes only the object under test and produces trustworthy comparisons.
    5. Treat failures as assets: Every case that “looks right but is actually wrong” should be preserved as a new test case, gradually covering more hidden defects.

    7. Conclusion

    The value of a lightweight model should not be defined by response speed alone. Whether it can complete multi-step tasks under explicit constraints and advance strictly along a specified path is what determines whether it can enter a production workflow. By bringing the process into the evaluation, handing validation to a program, and preserving failures as test items, developers and enterprises can make better-grounded judgments in selection and iteration, instead of letting the single label of “speed” drive their decisions.