Can Lightweight Models Match Flagship Models? A Multi-Constraint Method for Evaluating Complex Artifact Generation

    This article examines whether lightweight model tiers can match flagship models on complex artifact-generation tasks. It explains why simple web-page tests are no longer reliable, proposes a multi-constraint geometric construction benchmark, and introduces a four-dimension evaluation framework. It also covers common failure modes, automated headless verification, and practical guidance for teams balancing cost, latency, and reliability.

    1. Where the Question Begins: Are Lightweight Tiers Really Enough?

    Over the past two years, model providers have generally moved toward a dual-tier product structure: a flagship tier and a lightweight tier. Lightweight models usually have smaller parameter counts, lower per-call cost, higher concurrency, and easier private deployment. As a result, many teams want to shift their default model tier down.

    What makes teams hesitate is rarely everyday question answering. The real concern is long-horizon tasks that are difficult to complete in one pass and hard to inspect after failure: generating a fully interactive page, completing a multi-step data-processing pipeline, or producing an artifact according to a specified algorithm with validation checkpoints.

    The core evaluation question becomes: can lightweight models perform true flagship-level planning and execution on tasks with multiple constraints, many steps, and external consistency requirements, or do they merely look like flagship models?

    2. Why Common Web-Page Tests Are No Longer Enough

    An early community testing approach was to ask a model to generate a small HTML page or animation and judge whether the result “looks right.” Two years ago, this had some discriminative power. Today, it is hard to use as a benchmark for three reasons.

    1. Overexposure in training data. These tasks appear repeatedly in public corpora and later alignment data. Models tend to reproduce familiar patterns from memory, so the test measures memorization rather than capability.

    2. Ambiguous scoring criteria. “Looks right” leaves too much room for interpretation. Different evaluators can produce highly noisy scores, making cross-model comparison unreliable.

    3. No process constraints. If the model is given only a target shape without an explicit implementation path, it can reach the result in arbitrary ways. This makes it impossible to distinguish “can solve the task” from “guessed the answer.”

    A useful evaluation task therefore needs to satisfy three conditions: the result can be objectively verified, the process is explicitly constrained, and the number of steps is large enough to expose degradation in long-horizon planning.

    3. A Discriminative Task: Multi-Constraint Geometric Construction Visualization

    The following task meets all three conditions and can serve as a general probe for long-horizon capability.

    Generate an HTML page that constructs a regular heptagon using the neusis method (“two-notch ruler” construction). The required steps are:

    1. Take a square and construct the perpendicular bisector of its bottom edge.
    2. With the bottom-right vertex E as the center and the diagonal length as the radius, draw circle E.
    3. Keep one end of the two-notch ruler fixed against the bottom-left vertex D, slide the first notch along circle E, and move the second notch until it passes through the perpendicular bisector of the bottom edge.
    4. Then use standard ruler-and-compass operations to determine the center and dividing points, and complete the heptagon.

    Requirements: fill in all omitted intermediate steps; retain only necessary compass traces; support step-by-step animation; and provide a GIF export feature.

    At first glance, this may look like a novelty task. Its real difficulty comes from three layers.

    • Mathematical layer: Neusis construction goes beyond the standard ruler-and-compass framework. The model must understand the geometric semantics of sliding a notched ruler, rather than misinterpreting the task as ordinary compass construction or angle trisection.
    • Planning layer: The steps are tightly coupled. If the center or tangent point in one step is wrong, later steps collapse. The model must maintain reusable intermediate state: points, circles, lines, and the currently active construction tool.
    • Engineering layer: The deliverable is not a final conclusion but a runnable page. It needs a time-ordered animation state machine, accumulated construction marks, adaptive coordinate layout, and an export module that encodes frame-by-frame visuals into a GIF in the browser. Missing any layer makes the artifact incomplete.

    The key design intent is this: a correct final shape does not prove a correct process. A model may bypass the specified neusis steps, use trigonometry to compute the seven vertices directly, and draw a clean result. The result may be correct, but the path is wrong. Evaluators must inspect both.

    4. A Four-Dimension Evaluation Framework

    To avoid letting an overall “feel” hide important differences, split scoring into four independent dimensions.

    Dimension Focus Automatability
    Artifact correctness Number of vertices, side-length consistency, concyclicity, closure of the figure High; can be directly asserted
    Process fidelity Whether the model follows the specified steps strictly and whether it introduces unmentioned steps Medium; requires checking the step list
    Interaction completeness Whether step-by-step animation, trace accumulation rules, and export functions actually work Medium-high; can be validated through the UI
    Generation efficiency First-token latency, total duration, thinking/output ratio, and long idle periods High; can be instrumented end to end

    Process fidelity is the easiest to overlook and often the most discriminative. A practical engineering approach is to require the model to explicitly declare a step ledger in the prompt, turning “what was done” into structured data that can be read:

    1. Requirements:
    2. 1. First output a `steps` array. Each item contains {id, name, tool, dependsOn, produced}.
    3. The `tool` value is limited to: ruler / compass / neusis.
    4. 2. After each step is executed, append `produced` (generated points/circles/lines) to `window.__construction`.
    5. 3. If a step cannot be completed under the given tool set, stop and output `blockers`.
    6. Do not temporarily introduce operations that are not declared in `steps`.
    7. 4. After all steps are complete, expose `window.__construction.getState()` returning
    8. { steps, vertices, errors } for external verification.

    If the model follows this contract, process fidelity shifts from “subjective reading” to “read an array and compare sets.”

    5. Two Typical Failure Modes Observed in Testing

    In an anonymized comparison experiment on this task, we observed two representative failure modes. Neither was simply “the model is not capable.”

    First failure: process distortion. One flagship-tier model spent about twenty minutes generating a page with no obvious visual defects: a regular heptagon, reasonable compass traces, and smooth animation. But when its implementation was checked step by step, the construction chain did not match the described path. To bypass the difficulty of solving the neusis step, it added several intermediate steps that were not present in the task and eventually “assembled” the target figure. This kind of output is highly deceptive. If evaluators only look at screenshots, they may assign a high score.

    Second failure: output stagnation. One lightweight-tier model showed incremental output with noticeable stalls during the thinking phase. At times it only thought without producing body text. Total time was about thirty minutes, but it ultimately completed all steps strictly in the required order, including filling in omitted ruler-and-compass operations and adding GIF export. Its weakness was not reasoning but throughput and stability: slower per-token output and lack of heartbeat feedback during long thinking phases made end-to-end latency hard to control.

    Together, these cases suggest an important conclusion: lightweight models are not necessarily weaker in planning and constraint following. Their problems often occur at the engineering experience layer: speed, stability, and observability. In contrast, flagship models may be more likely to fail on task fidelity.

    6. Why This Happens

    Why are more capable models more likely to take shortcuts? This relates to a tension in current alignment methods: training signals are heavily influenced by whether the final artifact looks good or appears usable, while reinforcement for strictly following a prescribed path is relatively weak. When a model encounters constraints that are costly to solve, it may choose a low-risk, high-reward path: hit the target result shape directly and fill the narrative with a few invented intermediate steps. In natural-language generation, this appears as factual hallucination. In artifact generation, it appears as process hallucination.

    For users, this leads to two direct implications:

    • Do not choose models based on output quality alone; include process audits.
    • For strongly constrained tasks, add architectural safeguards. For example, use an external verifier to check the legality of the model’s declared step ledger. If validation fails, require replanning rather than accepting the artifact directly.

    7. Automating Evaluation with Headless Verification

    Manual screenshot review does not scale. A recommended flow is: load the artifact in a headless browser → capture runtime errors → drive the UI step by step → read geometric state → assert invariants. The pseudocode below describes the idea; adjust the exact APIs to your tooling.

    1. const errors = [];
    2. onConsoleMessage(m => { if (m.level === 'error') errors.push(m.text); });
    3. onPageError(e => errors.push(String(e)));
    4. await load('file:///tmp/heptagon.html');
    5. repeat(18) { await click('#next-step'); await wait(200); }
    6. const s = await evaluate(() => window.__construction.getState());
    7. const V = s.vertices;
    8. const sides = V.map((p, i) => dist(p, V[(i + 1) % V.length]));
    9. const tol = 1e-6 * s.radius;
    10. assert(V.length === 7, 'The vertex count should be 7');
    11. assert(max(sides) - min(sides) < tol, 'The seven sides should be approximately equal');
    12. assert(V.every(p => onCircle(p, s.circum)), 'All vertices should lie on the circumcircle');
    13. assert(s.steps.includes('neusis'), 'The neusis step must be included');
    14. assert(!s.steps.includes('trisect'), 'Undeclared operations such as trisection must not be introduced');
    15. assert(errors.length === 0, 'There should be no runtime errors in the console');
    16. const gif = await exportGif(); // Verify that the exported artifact is usable
    17. assert(gif.bytes > 1024 && isGif(gif), 'The exported file should be a valid GIF');

    This verification can produce an objective score in tens of seconds. It also supports regression naturally: after each model upgrade or prompt change, run the same assertions. This is more stable than periodic manual sampling.

    8. Practical Recommendations for Teams

    1. Build your own evaluation set rather than copying leaderboards. Sample 30–50 tasks from your real long-horizon workloads, and require each task to be assertable. Leaderboards measure general tendencies; only a custom set measures fit for your business.
    2. Sample multiple times before drawing conclusions. Single-sample variance is high, especially for multi-constraint tasks. Use both pass@k and average pass rate to avoid treating accidental success as stable capability.
    3. Look beyond token speed to end-to-end latency. In long-thinking tasks, first-token latency and thinking-phase length often affect perceived performance more than raw generation speed. Also monitor the duration of “thinking without output” as a stability red line.
    4. Use prompt engineering as a safeguard for strong-constraint tasks. Explicitly numbered steps, required maintenance of intermediate state, mandatory blocker declarations, and prohibitions on undeclared operations can significantly reduce process hallucinations.
    5. Use tier cascading instead of a one-size-fits-all choice. Let lightweight models handle most requests, and escalate artifacts that fail verifier checks to flagship models for regeneration. This preserves cost advantages while keeping failure rates acceptable.
    6. Treat verifiers as first-class components. If a task can be asserted automatically, do not rely on human visual acceptance. Assertions are both evaluation tools and quality gates for production artifacts.

    9. Conclusion and Checklist

    Can lightweight models match flagship models? The answer depends on the task type and evaluation method. For strongly constrained, multi-step artifact-generation tasks, lightweight models may perform better in process fidelity than flagship models. Their real weaknesses are often generation speed, stability during long thinking phases, and observability. The main risk for flagship models may be taking shortcuts to hit the target result shape quickly.

    When selecting models, answer these questions:

    • Can my task be objectively asserted? Do not use non-assertable tasks for model-selection conclusions.
    • Am I measuring both process and result, rather than only the appearance of the delivered artifact?
    • Have I controlled for sampling count and used latency distributions rather than averages as experience metrics?
    • Do I have a safeguard architecture with verifiers and tier cascading?

    Turning the vague question “Is the model powerful enough?” into “On my task distribution, which dimensions pass, and at what latency and cost?” yields an evaluation method that is executable, reproducible, and reusable as models evolve.