Fair Evaluation of Open-Ended Design Tasks for Multimodal Models

    This article presents a reproducible framework for evaluating multimodal models on open-ended design tasks. It moves beyond image recognition to assess runnable front-end delivery, responsive behavior, interaction states, screenshot-based debugging, repair effectiveness, end-to-end latency, and total cost under controlled conditions. It also proposes task categories, evaluation metrics, experiment orchestration, and engineering guidelines for model selection.

    Multimodal models are increasingly being used for design tasks, but a common misconception is to treat text recognition and image description as the endpoint of their capabilities. Recognition belongs to the perception layer. Actual design work also requires understanding information hierarchy, spatial relationships, brand character, and interaction states; translating those judgments into a runnable page; and identifying overflow, truncation, lost focus, and interaction errors from desktop and mobile screenshots.

    For this reason, multimodal design capability should not be evaluated solely through image-question answering or one-shot generation. A fairer approach is to build an evaluation methodology around runnable delivery that is reproducible, measurable, and suitable for engineering decisions.

    This article does not recommend or benchmark any specific product. Instead, it abstracts open-ended design tasks into a general technical problem: under the same environment, assets, browser automation conditions, and repair budget, how can different vision-language models be compared in terms of first-draft quality, final pass rate, dependence on repair, end-to-end time, and total cost?

    1. Redefining the Problem: From “Seeing” to Runnable Delivery

    The acceptance target for an open-ended design task is not a textual description. It is a runnable page and its behavior across multiple viewport sizes.

    A model must typically complete four types of work:

    1. Perceive the source materials
      This includes text, images, colors, and structural elements.

    2. Plan hierarchy and spatial relationships
      The model must determine priorities, alignment, spacing, and content density.

    3. Generate a runnable front-end implementation
      The resulting layout, styling, and basic interactions must work as intended.

    4. Inspect and correct the result
      The model should use desktop and mobile screenshots to identify horizontal scrolling, truncated text, overlapping elements, blocked dialogs, missing form validation messages, and similar issues.

    An evaluation that stops at image recognition will overestimate a model’s practical delivery capability. A more useful design separates the process into two stages: first-draft generation and one-round repair.

    The first-draft stage measures how reliably a model can produce a usable result in one attempt. The repair stage measures how effectively it can use screenshots and defect feedback to complete a correction loop. These capabilities should be reported separately rather than combined into a single score.

    2. Controlled Conditions for Fair Comparison

    The foundation of a fair comparison is control over variables.

    First, all candidate models should use the same version of the open design environment, the same asset set, the same browser automation workflow, and the same viewport collection, such as desktop widescreen, tablet, and narrow mobile layouts.

    Second, each task should allow only one repair round. Repair input should include the defect list and desktop/mobile screenshots. Unlimited iteration can hide weaknesses in the first draft and make results difficult to interpret.

    Third, prompts, asset paths, build commands, screenshot timing, and scoring rules must remain fixed. Their version numbers should be recorded in the experiment logs.

    To reduce randomness, each task should be run multiple times, with both averages and distributions reported. If a model supports parameter controls, values such as temperature and maximum output length should be fixed, and the random seed should be recorded.

    Scoring can combine blind review with rule-based checks:

    • Layout overflow, horizontal scrolling, and text truncation can be detected through browser-side rules.
    • Information hierarchy, brand character, and visual consistency can be evaluated by multiple independent reviewers, with agreement-based results reported.

    Cost accounting must also use a consistent scope. Total cost should include at least:

    • Model invocation costs
    • Browser automation runtime
    • Page rendering and screenshot generation
    • Log storage
    • Required human review

    Elapsed time should be divided into five stages:

    1. First-draft generation
    2. Build and runtime startup
    3. Screenshot capture and defect detection
    4. One-round repair
    5. Final verification

    Only with consistent definitions can cost and speed be meaningfully compared.

    3. Three Task Categories and Acceptance Criteria

    To cover different capability dimensions, an evaluation suite can include three general task categories.

    Task A: Information-Dense Dashboard

    The page contains multiple heading levels, metric cards, a filter area, a table, and an empty state.

    The evaluation should focus not only on visual appeal, but also on:

    • Whether the information hierarchy is clear
    • Whether the table remains readable on narrow screens
    • Whether the filter area overflows
    • Whether empty and loading states are complete

    This task primarily evaluates the model’s understanding of information density, alignment, and component states.

    Task B: Brand-Oriented Responsive Landing Page

    The page includes a hero section, value propositions, calls to action, and mobile breakpoints.

    Key evaluation criteria include:

    • Brand consistency
    • Image cropping
    • Clickable button areas
    • Horizontal scrolling on mobile
    • Text truncation

    This task evaluates how well the model balances visual expression with responsive layout behavior.

    Task C: Interaction-Complete Form or Modal

    The page includes input validation, error messages, focus management, keyboard accessibility, and handling for mobile keyboard occlusion.

    Evaluation should cover:

    • Whether validation errors are recoverable
    • Whether focus moves appropriately
    • Whether all modal closing paths work
    • Whether mobile controls remain usable when the keyboard is visible

    This task examines the model’s coverage of interaction states and accessibility requirements.

    Each task should provide a clear acceptance checklist and allow one repair round. During repair, the model should receive only the defect list and screenshots, without manually edited code. This ensures that the comparison measures the model’s own closed-loop repair capability.

    4. Metrics and Data Collection

    The following metric framework can be used:

    Metric Definition Collection Method Purpose
    First-draft pass rate Percentage of tasks that satisfy the acceptance checklist without repair Rule-based checks and blind review Measures one-shot stability
    Final pass rate Percentage of tasks that pass after one repair round Re-test and blind review Measures final delivery capability
    Defect density Number of defects weighted by severity per task Defect classification and counting Compares issue volume and severity
    Repair effectiveness Repaired defects divided by first-draft defects Before-and-after comparison Measures screenshot-based correction capability
    End-to-end time Total time from generation through final verification Timestamped logs Measures engineering efficiency
    Total cost Combined model, automation, rendering, storage, and review costs Unified accounting scope Measures scalability
    Closed-loop dependence Degree to which browser screenshots are required to reach acceptance Difference between first-draft and final results Indicates the importance of validation
    Manual intervention rate Percentage of tasks requiring manual fixes or adjudication Manual intervention records Measures deployment controllability

    Cost and elapsed time must be measured in practice. They should not be inferred from parameter count or labels such as “vision model.” Differences can be substantial across task types, so the evaluation should preserve detailed records for every task, model, and round rather than reporting only overall averages.

    5. Experiment Orchestration Pseudocode

    The following product-independent pseudocode illustrates a reproducible evaluation workflow:

    1. config:
    2. tasks = [T1, T2, T3]
    3. candidates = [M1, M2, M3]
    4. viewports = [desktop, tablet, mobile]
    5. repair_budget = 1
    6. repeat = N
    7. for candidate in candidates:
    8. for task in tasks:
    9. for run in 1..repeat:
    10. first_page = candidate.generate(task.prompt, task.assets)
    11. first_defects = audit(first_page, viewports, task.checklist)
    12. if first_defects is not empty and repair_budget > 0:
    13. screenshots = capture(first_page, viewports)
    14. fixed_page = candidate.repair(first_page, first_defects, screenshots)
    15. else:
    16. fixed_page = first_page
    17. final_defects = audit(fixed_page, viewports, task.checklist)
    18. record(candidate, task, run,
    19. first_defects, final_defects,
    20. time_breakdown, cost_breakdown,
    21. repair_effective_rate, manual_intervention)

    Logs should include at least:

    • Task version
    • Asset hashes
    • Environment version
    • Viewport list
    • First-draft defects
    • Repair inputs
    • Final defects
    • Time spent at each stage
    • Cost accounting scope

    Experiment artifacts can be stored in object storage, while process logs can be sent to a logging service and metrics can be connected to monitoring and alerting infrastructure. For example, general-purpose services such as object storage, logging, and monitoring provided by Baidu AI Cloud can support reproducibility materials and metric collection. These services are infrastructure components for the evaluation workflow, not part of the model comparison itself.

    This setup makes experiments easier to reproduce and supports long-term comparison.

    6. Interpreting Results: Similar Final Outcomes Do Not Mean Equal Capability

    In controlled trial runs with one allowed repair round, it is common for final pass rates to converge even though the underlying processes differ significantly.

    Some models produce more reliable first drafts with fewer defects, reducing manual review and rework. Others may provide a more complete product-like implementation but depend more heavily on browser screenshots and feedback to close the loop. Lightweight vision models may offer advantages in total cost and end-to-end time, making them suitable for frequent trial runs and large-scale initial screening.

    Therefore, evaluation should not focus only on whether the final result passes.

    • First-draft stability determines one-shot efficiency.
    • Repair effectiveness reflects closed-loop correction capability.
    • Closed-loop dependence indicates how important the validation pipeline is.
    • Total cost and elapsed time determine whether the approach is practical at scale.

    Model selection should assign weights according to the use case:

    • High-volume generation should emphasize unit task cost and first-draft pass rate.
    • Brand-oriented pages should emphasize visual consistency and blind human review.
    • Interaction-heavy pages should emphasize browser-based validation and accessibility coverage.

    7. Engineering Recommendations

    1. Build a task benchmark library

    Fix the assets, prompts, acceptance checklists, and defect categories so that each evaluation uses a stable benchmark rather than a changing set of tasks.

    2. Report first-draft and final results separately

    A repair budget should not conceal weaknesses in the initial output.

    3. Treat browser-based validation as a first-class component

    Screenshot capture, defect detection, one-round repair, and final verification should form a standard pipeline.

    4. Display cost and quality together

    A cost dashboard should be shown alongside the quality dashboard. This helps avoid two common misjudgments: choosing a cheap approach that requires excessive rework, or choosing a high-quality approach whose cost cannot be controlled.

    5. Retain human review and blind evaluation

    Subjective dimensions such as brand character and information hierarchy require consistency checks through human reviewers.

    6. Record security and compliance boundaries

    Sensitive data should not be introduced into evaluation assets, and the scope of permitted data use should be documented.

    8. Conclusion

    Multimodal design evaluation should move beyond the question of whether a model can recognize an image. The more meaningful question is whether it can deliver a runnable page and complete a screenshot-driven repair loop.

    Fair comparison depends on a consistent environment, shared assets, a fixed repair budget, unified cost definitions, and reproducible logs. Only by evaluating first-draft stability, final pass rate, repair effectiveness, closed-loop dependence, elapsed time, and total cost together can engineering teams make model choices that match their operational constraints.