Lightweight Flagship Model Evaluation: Constraint Following Is Harder Than Results

    When evaluating generative models, tasks that merely "look finished" tend to produce inflated scores. This article argues that the real gap between lightweight and heavyweight models shows up not in whether a target figure can be drawn, but in whether it can be drawn by the prescribed path under the given constraints. Using an abstracted geometric construction task, it explains the core tension between a correct result and genuine capability, shows how to build high-discrimination, auditable evaluation tasks, and documents three typical failure modes seen in controlled comparisons: path substitution, silent stalling, and approximate passing. It then provides a quantifiable scoring rubric, a minimal automated-validation approach, and engineering recommendations—tiered routing, stall detection, task decomposition, artifact governance, and throughput-centered capacity planning—for turning evaluation results into architecture decisions, followed by a selection checklist.

    1. The Core Tension: A Correct Result Does Not Mean the Capability Is Met

    Generative models tend to score inflated on tasks that merely “look done.” When a task asks the model to complete a construction following specified steps, the final image may indeed show the target figure—but if the model skips a required step or adds an operation that was never in the original task, the output is unacceptable from an engineering standpoint. It does not demonstrate that the model can advance step by step under given constraints.

    This kind of drift can be summarized as outcome fitting: the model approaches the visible end state rather than moving along the prescribed path. In real production systems, this is equivalent to letting a system bypass the API and write to the database directly—the goal is superficially met, but the controllability and auditability of the process are broken.

    Therefore, the goal of high-quality evaluation design should be auditability, not just renderability. The real gap between lightweight and heavyweight models often does not show up in “whether it can be drawn,” but in “whether it can be drawn in the prescribed way.”

    2. How to Build High-Discrimination Evaluation Tasks

    An effective evaluation task should satisfy four conditions at once:

    1. Unique end state, constrained path: the final answer is unique, which makes it easy to judge right or wrong; the implementation means are restricted, which makes it easy to detect substitutions.
    2. Statically checkable intermediate steps: each step produces a geometry object with a clear definition that a program can verify.
    3. Results cannot be gamed by approximation: visual “closeness” is not enough to pass; the result must hold up to mathematical precision.
    4. Inherently includes tool use and formatted output: the same task requires reasoning, state management, and file generation.

    Take a typical geometric construction task as an example (described in the abstract):

    In a web page, demonstrate how to construct a regular heptagon using a graduated ruler (a marked straightedge). Start from a square and construct the perpendicular bisector of the base; draw a circle centered at one vertex with a radius equal to the diagonal length; then keep one end of the graduated ruler always touching a vertex, sliding the first graduation along the circle until the second graduation lands on the perpendicular bisector of the base; then use conventional construction methods to determine the circumcenter and the remaining vertices. Fill in all omitted construction steps; the construction process should show a dynamic trajectory, but auxiliary marks should be kept only where necessary; finally, provide a feature that exports the process as an animated image.

    This task’s difficulty is severely underestimated; it contains at least four layers of challenge:

    • Discretizing continuous constraints: “one end of the ruler touches a vertex, the first graduation lies on the circle, the second graduation lies on the perpendicular bisector” is essentially a system of simultaneous equations. The model must derive the location of the unique solution itself rather than eyeballing an approximate value.
    • State machine management: the animation needs to maintain states such as current step, generated points, drawn arcs, and whether it is complete, and drive step-by-step progression.
    • Layered trace rendering: auxiliary arcs and the final figure must be kept separate, or the image will degrade into a clump of noise.
    • Additional engineering ability: off-screen redrawing, frame-by-frame capture, encoding into an animated image, and triggering a download—this is a typical “capability spillover” checkpoint.

    3. Three Typical Failure Modes Exposed in Controlled Comparisons

    In real side-by-side comparisons, models at different tiers exhibit clearly different failure patterns, which are worth recording as evaluation conclusions in their own right.

    Type A: Path substitution (correct result, wrong process). Some more capable models did produce the target figure, but the implementation path did not match the task statement at all—they added steps that were not in the statement and forced their way to the end state. This output is easily judged as passing in a quick human check; only by comparing the generated code against the actual answer sequence line by line can the deviation be identified. This is the most dangerous kind of failure, because it fools both human reviewers and automatic scoring based on the final screenshot.

    Type B: Silent stalling (reasoning without producing). During long-chain reasoning, the model keeps emitting reasoning content but never enters the artifact-generation phase, eventually timing out or returning an empty result. It shows up as choppy, non-smooth output and long stretches with no effective increment on the streaming channel. This reminds us that evaluation records must include the effective output ratio, not just the final answer.

    Type C: Approximate passing (visually close but mathematically wrong). There are systematic deviations in figure proportions and intersection-point positions; these are hard to notice at the thumbnail level, but once you quantify the coordinate error, they fail.

    Notably, lightweight-tier models may perform counterintuitively better than heavier tiers on this kind of task: they are more willing to strictly follow the order of steps given in the statement and do not proactively optimize the path. Strict constraint following is itself a high-value capability trait, especially in process-driven business scenarios.

    4. Quantifiable Evaluation Dimensions and a Scoring Rubric

    To avoid subjective judgment, the evaluation needs to be broken into computable metrics. A suggested scoring structure is:

    1. {
    2. "constraint_following": {
    3. "required_steps_present": 0.0,
    4. "extra_steps_invented": 0,
    5. "order_violations": 0
    6. },
    7. "math_accuracy": {
    8. "max_coordinate_error": 0.0,
    9. "collinearity_check": false,
    10. "distance_check": false
    11. },
    12. "artifact_quality": {
    13. "runs_without_error": false,
    14. "animation_correct": false,
    15. "export_available": false,
    16. "trace_layering": false
    17. },
    18. "runtime": {
    19. "wall_clock_seconds": 0,
    20. "effective_output_ratio": 0.0,
    21. "silent_stall_detected": false
    22. }
    23. }

    Here, extra_steps_invented and order_violations are the key items that separate tiers, while effective_output_ratio (effective-artifact tokens / total output tokens) is a practical signal for detecting silent stalling.

    5. A Minimal Implementation Approach for Automated Validation

    Manual review does not scale, so programmatic validation should be introduced as early as possible. The core idea is to abstract the geometric operations in the generated artifact into a parseable event sequence, then assert them against analytic-geometry conditions.

    1. import math
    2. TOL = 1e-6
    3. def dist(p, q):
    4. return math.hypot(p[0] - q[0], p[1] - q[1])
    5. def on_circle(p, center, radius):
    6. return abs(dist(p, center) - radius) < 1e-4
    7. def on_line(p, a, b):
    8. # Cross product to check collinearity
    9. cross = (b[0]-a[0])*(p[1]-a[1]) - (b[1]-a[1])*(p[0]-a[0])
    10. return abs(cross) < 1e-4
    11. def audit(events, spec):
    12. findings = {"missing": [], "invented": [], "imprecise": []}
    13. produced = [e.name for e in events]
    14. for step in spec.required_steps:
    15. if step not in produced:
    16. findings["missing"].append(step)
    17. for name in produced:
    18. if name not in spec.allowed_ops:
    19. findings["invented"].append(name)
    20. # Key constraint: the sliding endpoint must lie on both the circle and the perpendicular bisector
    21. if not (on_circle(spec.neusis_point, spec.center_e, spec.radius_e)
    22. and on_line(spec.neusis_point, spec.mid_a, spec.mid_b)):
    23. findings["imprecise"].append("neusis_point")
    24. return findings

    Supporting engineering measures:

    • Headless rendering plus screenshot baseline comparison: render the generated page in a headless browser and compare it against a reference baseline for structural similarity, to catch cases that pass at thumbnail size but break in the details.
    • Interactive export verification: the export feature must pass a click–download–decode-frame-count closed loop; it cannot be marked as passing just because an export function appears in the code.
    • Multiple sampling: sample the same task at least three times and record the pass rate and variance; a single data point has no statistical meaning.

    6. Engineering Recommendations for Long-Chain Tasks

    Evaluation conclusions should translate into architecture decisions. The following points are generally applicable:

    1. Tiered routing rather than one-size-fits-all replacement. Layer tasks by path freedom: route structural generation and process-following tasks to lightweight-tier models with strong constraint following and better cost, and leave open-ended design and hard fallback cases to heavyweight models. The routing basis comes from a pass-rate matrix on an offline evaluation set.
    2. Stall detection and timeout policy. For reasoning-heavy long tasks, a watchdog threshold for “no effective incremental output” must be set; once triggered, abort and retry or degrade, to avoid wasting connections and compute on invalid work.
    3. Task decomposition and self-check loops. Split deriving the constraint solution and drawing-and-exporting into two stages, and persist and verify the intermediate artifacts before continuing—this can significantly reduce the end-to-end failure rate and makes problems easier to locate.
    4. Context and artifact governance. Generated pages, coordinate data, logs, and version numbers should be archived together in object storage, combined with log services and monitoring alerts, so that problems can be replayed.
    5. Throughput-centered capacity planning. A lightweight-tier model’s per-call latency may be higher, but its concurrency capacity per unit cost is usually stronger. Load testing should focus on throughput, queueing latency, and failure rate under concurrency, rather than a single-call stopwatch reading.

    7. Selection Checklist

    • Does the evaluation task have the properties of a constrained path plus static verifiability?
    • Have you counted the number of extra steps the model invented on its own?
    • Have you distinguished mathematically precise from visually approximate?
    • Have you recorded the effective output ratio and the number of silent stalls?
    • Have you performed multiple samplings on the same task and reported the pass rate?
    • Have you distilled the evaluation conclusions into reusable routing rules and a regression test set?

    Conclusion: The value of a lightweight flagship model cannot be judged only by how it looks on generation tasks. When the evaluation standard shifts from “does the result look good” to “can it hold to the constraints,” the ranking of model tiers may reshuffle. Building an evaluation system centered on constraint following, mathematical precision, and artifact verifiability is the prerequisite for rational selection.