Fair Evaluation of Multimodal Models on Open Design Tasks: A Practical Guide

    This guide explains how to build a fair, reproducible evaluation process for multimodal models that generate runnable web pages from open design tasks. It covers why evaluation has shifted from image recognition to producing working, interactive output; the controlled-variable principles that make comparisons meaningful; three task categories (information architecture, interactive components, and responsive visuals); an end-to-end workflow from first-version generation through a single fix round and retest; how to interpret results beyond a simple pass/fail; and how to build a reproducible evaluation sandbox with automated visual regression, structured issue detection, cost tracking, and human review. It also lays out practical steps for turning a one-off experiment into a lasting evaluation system for model selection and engineering optimization.

    1. From “Seeing Images” to “Completing Designs”: What Changes in Multimodal Evaluation

    Multimodal capabilities are moving beyond single tasks like image recognition, text extraction, and scene description toward more complex, end-to-end task loops. For design tasks, a model must not only understand the elements, copy, layout, and visual style in an input image, but also turn that understanding into a runnable, browsable, and interactive page. In other words, the focus of evaluation has shifted from “can it recognize” to “can it turn understanding into an engineering deliverable.”

    These tasks share several clear characteristics:

    1. Incomplete inputs. Design requirements often arrive as a short brief, a few assets, or a reference image, so the model has to fill in the information hierarchy on its own.
    2. No single correct output. Pages have no absolute ground truth, but there are clear quality differences—for example, whether the structure is sound, the hierarchy is clear, and the responsive behavior is usable.
    3. Runnable results. The final deliverable is not a description but a renderable page, so it must be verified in a real browser environment.
    4. Problems surface through the full loop. Issues like overflow, truncation, misalignment, and broken interactions usually can’t be caught at generation time alone; they require screenshot regression and runtime checks.

    As a result, multimodal evaluation of design tasks cannot rely only on “does it look similar” or “does it look nice.” It needs a complete evaluation process that spans generation, rendering, detection, and repair.

    2. A Prerequisite for Fair Comparison: Controlling Variables Beats Chasing Speed

    In open design tasks, fair evaluation is not simply handing the same problem to different models—it is eliminating external interference as much as possible. Design tasks depend heavily on the environment. Even a change in browser viewport, asset dimensions, page cache, or screenshot timing can produce noticeably different results. Evaluation design must therefore prioritize reproducibility first.

    Common fairness principles include:

    1. Same task. All candidates use the same requirement description, the same set of constraints, and the same batch of assets.
    2. Same environment. Pages are rendered with the same browser environment, the same viewport sizes, the same screenshot conventions, and the same detection scripts.
    3. Same budget. Each task gets the same input size, output length, number of tool calls, and runtime limit, so resource differences don’t skew the results.
    4. Same fix rounds. A common setup is “first version + one fix round,” which lets you observe both a model’s one-shot generation ability and its ability to correct problems based on feedback.
    5. Same scoring standard. Combine automated metrics with human review rather than relying on subjective impressions alone.
    6. Full traceability. Every task’s page snapshot, screenshots, issue list, repair log, and consumption data are saved completely for later review.

    Without these controls, results easily become a comparison of coincidences rather than a comparison of capability. For example, a better first version in one run may reflect a clearer prompt rather than a stronger model; a lower cost in another run may simply mean shorter output, not higher generation quality.

    3. Task Set Design: Three Categories of Open Design Tasks That Cover Real Capability

    To bring evaluation closer to real engineering scenarios, you can design three typical task categories: information architecture, interactive components, and responsive visuals. Each examines a different capability—content organization, interaction logic, and visual consistency.

    3.1 Information Architecture and Hierarchy Design Tasks

    These tasks ask the model to design a clear information hierarchy based on content blocks, data fields, and page goals. For example, given assets for a data overview page, the model must make the first screen highlight the core metrics, present secondary information in layers, and avoid crowding or misalignment on mobile.

    Evaluation focuses on:

    • Whether the primary/secondary relationships on the page are clear;
    • Whether the hierarchy among titles, summaries, metrics, and action areas is reasonable;
    • Whether information density is too high or too low;
    • Whether desktop and mobile keep the same structure;
    • Whether the layout holds up in empty-data, low-data, and high-data states.

    These tasks do a good job of separating a model’s “understanding ability” from its “layout ability.” Some models can recognize the text in the assets but can’t turn that text into a sensible page structure; others can generate a complete framework but with an overly flat hierarchy that leaves no clear visual focus.

    3.2 Interactive Components and State Design Tasks

    These tasks require the model to do more than generate a static page—it must also handle interaction states. For example, design a form, filter area, modal, list card, or action panel that includes default, selected, disabled, error, and empty states.

    Evaluation focuses on:

    • Whether the control types match the task goal;
    • Whether click behavior is predictable;
    • Whether state transitions are clear;
    • Whether error messages are understandable;
    • Whether interaction feedback matches the visual style;
    • Whether there are problems such as unclickable elements, overlapping layers, or broken event handlers.

    Interactive tasks are closer to product prototypes than purely static pages. A model that only produces visual styling without handling state changes will struggle to pass a complete evaluation. In contrast, models that fill in interaction paths around the user task tend to fit engineering needs better.

    3.3 Responsive and Visual Consistency Tasks

    These tasks require the model to balance desktop and mobile behavior on the same page. The input can be a set of visual constraints—spacing rules, color scheme, corner style, type hierarchy, and component density—and the model must keep the style consistent across viewports.

    Evaluation focuses on:

    • Whether horizontal scrolling appears;
    • Whether text is truncated;
    • Whether elements overlap;
    • Whether images fail to load;
    • Whether mobile stays usable;
    • Whether desktop shows excessive whitespace or sparse information;
    • Whether the visual style is consistent across pages.

    Responsive issues are where design tasks most easily expose flaws. Many pages look complete on desktop but, once switched to mobile, run into blocked buttons, abnormal card widths, overflowing text, and distorted tables all at once. This is why browser regression testing is an indispensable part of open design task evaluation.

    4. Evaluation Workflow: A Closed Loop from First-Version Generation to One Fix Round

    A rigorous evaluation workflow should include these stages:

    1. Task delivery. Feed the model a unified design brief, asset package, constraints, and acceptance criteria. The task description should avoid ambiguity while leaving some openness, to test the model’s design judgment.
    2. First-version generation. This is the stage that best reflects a model’s raw understanding. At this point, you should not offer specific fix hints like “watch out for mobile” or “avoid overflow,” which would weaken the comparison.
    3. Static checks. Before the page runs, check whether the output is complete—for example, whether the page structure exists, whether major regions are missing, whether asset references are valid, and whether the copy has obvious errors.
    4. Browser rendering and screenshot regression. Deploy the page to a unified sandbox environment, capture screenshots at both desktop and mobile viewports, and automatically detect common problems such as horizontal overflow, text truncation, element overlap, and missing key regions.
    5. Issue list generation. Organize detection results into a structured issue list, for example:
    1. {
    2. "task": "responsive visual consistency",
    3. "desktop_overflow": false,
    4. "mobile_overflow": true,
    5. "truncated_text": ["title area", "button label"],
    6. "interaction_error": ["submit button unresponsive"],
    7. "screenshot_diff": true
    8. }
    1. One fix round. Feed the issue list back to the model and allow only one fix round. This setting is critical, because in real engineering teams don’t iterate infinitely—they expect the model or an engineer to converge quickly based on feedback.
    2. Post-fix retest. Rerun the page and check again whether the issues are actually resolved. Note that passing after a fix doesn’t mean the first version was weak, but it does reveal how well the model understands feedback and locates problems.
    3. Cost and latency tracking. Record each task’s output size, number of calls, tool interactions, rendering time, and total cost, to compare efficiency and return on investment.

    5. Interpreting Results: Don’t Look Only at Whether It Finally Passes

    In a controlled experiment like this, a common phenomenon is that several general-purpose multimodal models can all complete basic design tasks once a fix round is allowed. But that doesn’t mean they have the same capability. What really matters is the difference before and after the fix.

    You can interpret results across these dimensions:

    1. First-version stability. Some models produce a stable first version with a complete page skeleton, all major regions present, and no obvious mobile problems; others produce a complete page but with many layout or interaction issues that depend on the fix.
    2. Problem-discovery ability. Many problems in design tasks can’t be judged from the generated content alone—they must be checked against screenshots or runtime state. Being able to find issues like overflow, truncation, and misalignment from visual regression results and fix them precisely is an important sign of engineering capability.
    3. Fix efficiency. The focus is whether a single fix round truly resolves the core problems rather than introducing new ones. A high-quality fix has three traits: accurate localization, minimal changes, and a stable result.
    4. Cost and speed. When results are close, cost and speed directly affect how easily a solution can be adopted. If a task requires many tool calls, repeated screenshots, and long repair chains, it may pass but still be unsuitable for high-frequency production.
    5. Maintainability. Whether the output is clearly structured, semantically sound, and easy for humans to modify later also affects the real-world experience. Running successfully is only the baseline; being maintainable is where the engineering value lies.

    So a fair evaluation’s conclusion should not be a simple “who is better,” but “who is better suited to which type of task.” For example:

    • Scenarios that prioritize first-version stability should focus on page skeleton completeness and initial accuracy;
    • Complex interaction scenarios should focus on tool-loop capability and problem-fixing ability;
    • Batch generation scenarios should focus on cost, throughput, and stability;
    • High-fidelity showcase scenarios should focus on visual consistency, responsive quality, and screenshot regression performance.

    6. Engineering Implementation: Build a Reproducible Evaluation Sandbox

    To make results repeatable, enterprises are advised to build their design task evaluation platform as a standardized sandbox. The sandbox should have the following capabilities:

    1. Unified runtime environment. Each task runs in an isolated container to avoid cross-contamination between tasks. Environment variables, dependency versions, font resources, and viewport configuration should all be pinned.
    2. Asset version management. All input assets, prompts, constraint files, and screenshot baselines should be under version control. If a task changes, regenerate the evaluation baseline so historical results stay comparable.
    3. Automated visual regression. Capture desktop and mobile screenshots for each task and compare them against the baseline page. You can use pixel diffs, region diffs, or key-element diff checks, but output the results as an issue list rather than only a subjective score.
    4. Structured issue detection. Automated detection should cover common problems:

      • Whether horizontal scrolling exists;
      • Whether text is truncated;
      • Whether images fail to load;
      • Whether buttons or links are unclickable;
      • Whether modal layers behave abnormally;
      • Whether blank areas are too large;
      • Whether mobile controls are too small.
    5. Cost ledger. Each task should record input, output, tool calls, number of screenshots, fix rounds, and total elapsed time. Only by counting cost in the evaluation can you avoid the misjudgment of “great quality but can’t scale.”

    6. Human review supplement. Automated metrics are good at finding obvious problems; human review is good at judging aesthetics, information expression, and design intent. A combined approach of “automated detection + human spot checks + expert review” is recommended.

    When building such an evaluation platform in the cloud, you can combine general cloud capabilities to form a closed loop. On Baidu AI Cloud, for example, you can use object storage to save assets, screenshots, and evaluation reports, a container platform to run isolated sandboxes, a log service to record task call chains, and monitoring and alerting to watch for task failures, timeouts, and abnormal consumption—improving the operability of the evaluation workflow.

    7. Enterprise Adoption: From a One-Off Experiment to a Long-Term Evaluation System

    For enterprises, open design task evaluation should not be a one-off experiment but a foundational capability for model selection, product validation, and engineering optimization. Suggested steps:

    1. Build a small standard task set. Early on, don’t chase a large number of tasks; first cover the three core scenarios—information architecture, interactive components, and responsive visuals. Each task should have clear acceptance criteria.
    2. Define the evaluation scope. Distinguish “model capability evaluation” from “product effectiveness evaluation.” The former focuses on whether the model can complete the design loop; the latter focuses on whether the final page meets business goals. The two shouldn’t be mixed.
    3. Establish a fix-round strategy. For rapid prototyping, allow one fix round; for production, add multiple fix rounds, human review, and pre-launch regression.
    4. Build a continuous regression mechanism. When the model version, prompt templates, asset library, or runtime environment changes, rerun regression tasks so results don’t become stale.
    5. Build a selection matrix. Don’t look at a single dimension. Recommend scoring across multiple dimensions: first-version stability, fix success rate, responsive performance, interaction completeness, cost efficiency, and maintainability.

    8. Summary

    Multimodal evaluation of open design tasks is essentially turning the vague question “can the model do design” into an observable, reproducible, and comparable engineering problem. The key to fair evaluation is not simply comparing the visual effects of a page, but observing the model’s complete capability—from understanding requirements, generating pages, and discovering defects to completing the fix loop—under the same task, same assets, same environment, and same fix rules.

    As multimodal capabilities keep evolving, a truly valuable evaluation should not only answer “who generates better,” but “who is better suited to a real design workflow.” Only when first-version stability, problem-discovery ability, fix efficiency, responsive performance, and cost consumption are all included in the same evaluation system can enterprises accurately judge the deployment value of multimodal models in design scenarios.