Testing Multimodal LLMs for Design Generation: A Practical Benchmarking Method

    A practical methodology for benchmarking how multimodal large language models actually design and generate interfaces — not just whether they can read an image. The article defines design generation as a four-layer capability (perception, comprehension, planning and execution, and self-reflection), shows how to control variables for a fair cross-model comparison, proposes three task types (original visuals, reference reproduction, and state consistency), and provides a scorecard split into hard, automated metrics and soft, human-reviewed ones. It also covers why the visual self-inspection loop is the real differentiator, how to account for cost and latency correctly, how to engineer the evaluation pipeline, and which biases to guard against.

    1. Defining the problem: “reading an image” and “doing design” are not the same thing

    Most multimodal evaluations stop at “recognize text, describe the scene” — essentially single-step perception tasks: feed in one image, output one description, and a human can judge correctness at a glance. But design generation is a long-chain task that must span at least four layers of capability:

    1. Perception: Can it read the block layout, type hierarchy, spacing rhythm, and color tendencies in a reference image?
    2. Comprehension: Can it judge information priority — what deserves emphasis, what should yield, and whether the brand feel should lean restrained or bold?
    3. Planning and execution: Can it translate that judgment into a structured interface description and generate runnable code?
    4. Self-reflection: Can it render the page itself, review the screenshot, and spot and fix issues such as overflow, truncation, console errors, and broken interactions?

    The fourth layer is what actually decides success. The acceptance bar for a design task isn’t “does the code compile” but “does the page hold up, and do the interactions close the loop, across both desktop and mobile viewports.” So a fair design-capability evaluation must include a rendering environment and screenshot review, and it must score repair ability separately.

    2. Evaluation design: control variables before comparing

    The most common reason cross-model comparisons fail is that the variables aren’t controlled, so the conclusions aren’t trustworthy. You need to fix the following during experiment design:

    • Uniform task inputs: the same prompt version, the same asset pack (images, datasets, audio, etc.), and the same acceptance checklist;
    • Uniform runtime: the same version of the design-generation and rendering pipeline, the same headless browser version, and the same viewport set (e.g., desktop 1440×900, mobile 390×844);
    • Uniform tool access: give every model the same screenshot, console-log reading, and file read/write capabilities, and forbid any single approach from enjoying privileged “internal review”;
    • Uniform repair budget: allow only one repair round per task, and that repair must take the automated check report as its input;
    • Uniform accounting: record first-pass latency/cost and post-repair total latency/cost separately, and don’t mix them.

    Only when these are nailed down can the final differences be attributed to the models themselves, rather than to a well-written prompt, generous assets, or a larger number of screenshots.

    3. Task design: three task types, three capability profiles

    A single task can’t distinguish model capabilities, so it’s better to design a task matrix around what you want to probe. It can be abstracted into three types:

    Task A: Original visuals and data storytelling
    Given a time-series dataset, a world map base layer, and several real-world assets, build a scroll-driven, exhibition-style page: year changes drive curve updates, and clicking a region switches the base map and risk indicators. This probes original visuals, information hierarchy, and data storytelling. With no reference image, a model can’t score by imitation.

    Task B: Reference-image reproduction of a product page
    Given desktop and mobile reference images, a geographic dataset, and several locations with accompanying audio, implement map point selection, playback, track switching, favoriting, collection sync, and state restoration after refresh. This probes reference-image comprehension accuracy and interaction completeness — close to real product requirements.

    Task C: Responsiveness and state consistency
    On top of B, add boundary scenarios such as narrow screens, extremely long text, empty data, and rapid consecutive clicks. This probes layout robustness and state-machine correctness.

    4. The scorecard: breaking “looks good” into verifiable items

    Scoring should have two layers: hard metrics and soft metrics. Hard metrics can be judged automatically and serve as entry gates; soft metrics reflect aesthetics and product sense and need human or multimodal review.

    Dimension Specific checks Judgment method
    Layout robustness Horizontal overflow, elements compressed to near-zero size, text truncation, control overlap Automated assertion
    Visual assets Image/font/audio load failures, leftover placeholders Automated assertion
    Interaction closure Whether clicks, track switching, favoriting, and filtering respond, and whether state is correct Script-driven
    State consistency Whether selected items, playback progress, and collections are restored after refresh Script-driven
    Responsiveness Whether the first screen holds up in both viewports, and whether key information is left exposed Automated + manual
    Console health Uncaught exceptions, resource 404s, warning count Automated statistics
    Visual & originality Hierarchy, whitespace, color scheme, and structural consistency with the reference image Human/review scoring
    Cost efficiency First-pass cost, total cost, total latency, repair rounds Instrumentation statistics

    Schematic check logic (pseudocode, tool-agnostic):

    1. def run_checks(page, viewport, state_script):
    2. return {
    3. "console_errors": page.console_errors(),
    4. "overflow_x": page.scroll_width() > viewport.width + 1,
    5. "collapsed_nodes": find_nodes(lambda r: r.height <= 3 or r.width <= 3),
    6. "text_truncated": detect_truncation(page),
    7. "assets_broken": page.broken_assets(),
    8. "state_restored": (page.reload(), page.read_state() == state_script.expected)[1],
    9. }

    Pipeline main flow:

    1. task = load_task(task_id) # prompt + assets + acceptance checklist
    2. artifact = generate(task, budget={"repair_rounds": 1})
    3. report = {
    4. "desktop": run_checks(artifact, viewport=(1440, 900), state_script=task.state),
    5. "mobile": run_checks(artifact, viewport=(390, 844), state_script=task.state),
    6. }
    7. if report.has_errors() and artifact.repair_rounds < 1:
    8. artifact = repair(artifact, feedback=report.as_screenshot_findings())
    9. report = re_run_checks(artifact) # must re-run, not just fix and submit
    10. score = rubric(hard=report, soft=visual_review(screenshots))
    11. archive(task, artifact, report, score, cost, latency)

    Two easily overlooked details: after a repair, the screenshots must re-run the same set of assertions, otherwise “fix and submit” can’t be verified; and the repair budget needs a cap — an uncapped approach falls into an infinite “screenshot → edit → re-screenshot” loop, which both makes costs incomparable and masks a weak first pass.

    5. Empirical observations: three capability profiles

    In this round of controlled-variable experiments, the differences among the three model types showed not in “whether they pass” but in the path they take to pass:

    • Steady type: passes most hard metrics on the first pass; the visual presentation leans magazine-like and high-impact, with high structural fidelity to the reference image and the lowest retry rate. Its strength is first-pass quality; its weakness is that the marginal gain during revision is limited.
    • Complete type: tends to output a full product structure rather than a single-page visual draft; it has the highest information density and the most complete interaction modules, but it depends most on screenshot review — the first pass often shows containers compressed to around 2px or collapsed blocks, which must be found via browser inspection and fixed before the result becomes usable.
    • Minimal type: the lowest unit cost and latency; under the test setup, total cost can be as low as roughly one-tenth of the highest; its first-pass completeness isn’t strong, but it passes acceptance after repair. This shows its capability leans more on “self-reflection + patching” than on getting it right in one shot.

    It’s worth stressing that these conclusions are highly sensitive to task distribution: swap in a different set of tasks or a different viewport set, and the ranking may flip. Their value lies in revealing the structure of capabilities, not in producing a permanent leaderboard.

    6. Key conclusion: the visual self-inspection loop is the watershed

    Failure modes are highly concentrated and fall into five categories: container size collapse, text overflow and truncation, control occlusion, asset load failures, and state not persisting. At the code level these are often just a few lines of styling or a state field, but to human eyes the “page is broken.” Whether a model can catch them on its own depends on whether it has the closed loop of “generate → render → review → locate → patch,” and whether it treats screenshots as real input rather than decoration.

    7. The right accounting for cost and latency

    When comparing cost across models, do at least four things: report first-pass and post-repair costs separately; fix the number of screenshots and viewports; state the billing method (per call, by context length, or another measure) and the context-reuse strategy; and record retry rounds. Otherwise “cheaper” often just means “took fewer screenshots and ran fewer assertions.”

    8. Engineering the evaluation pipeline

    Once an evaluation grows beyond a few dozen tasks, it must be engineered: version-manage tasks and assets, archive artifacts and screenshots to object storage for retrospection, host the headless browser on a container platform and scale by concurrency, and continuously track failure rate, repair rate, and unit cost through a logging service and monitoring alerts. On Baidu AI Cloud these are general product categories, independent of any specific model, and they can also be replaced by a self-built setup. The point of engineering is to make evaluation repeatable — the same task set can still produce comparable data three months later.

    9. Limitations and anti-cheating

    You should proactively acknowledge four kinds of bias: first, training data may already contain public task samples, so build a private task set; second, prompt leakage inflates apparent capability, so prompts should be held uniformly by the evaluator; third, aesthetic scoring has rater bias, so use multiple people cross-scoring against a fixed scale; fourth, a single run has randomness, so key conclusions should be drawn from distributions over multiple runs rather than from a single point.

    10. Implementation recommendations

    To adopt this method within a team, proceed in this order: first, split the task set into three layers (original visuals, reference reproduction, state consistency) and build the acceptance checklist; then, connect rendering and screenshot review, and write the automated assertions as entry gates; then, fold the repair rounds into the budget and score them separately; finally, put cost, latency, retry rate, and visual scores in one report for a four-dimensional comparison. Model selection shouldn’t look at the single highest score, but at whether the combination of first-pass pass rate and unit cost on your own task distribution matches your business cadence.