Multimodal Model Evaluation on Open-Ended Design Tasks: A Fair and Reproducible Practice

    This article examines how to evaluate multimodal models fairly on open-ended design tasks. It defines the capability gap from visual recognition to a runnable, inspectable, and repairable design closed loop; proposes controlling variables and using multiple metrics; describes three differentiated task scenarios; presents a six-step execution pipeline with configuration and pseudocode; and offers guidance for interpreting results and operationalizing evaluation through versioning, defect taxonomies, blind review, repair budgets, cost accounting, and compliance checks.

    In multimodal model evaluation, “can it recognize an image” is no longer the core question. Recognizing text, describing a scene, and judging colors only show that a model “sees.” Open-ended design tasks truly test whether a model can understand information hierarchy, spatial relationships, and brand character; whether it can turn judgments into a runnable frontend page; whether it can find overflow, truncation, misalignment, and interaction errors from desktop and mobile screenshots; and whether it can complete a closed loop within a limited number of repair rounds. This article focuses on open-ended design tasks and discusses a fairer, more reproducible method for evaluating multimodal capabilities, with recommendations for task design, metric definition, execution pipelines, and engineering implementation.

    1. Problem Definition: From “Seeing” to “Doing Design”

    An open-ended design task usually contains four continuous stages: visual understanding, information architecture, code implementation, and result verification. Visual understanding requires the model to read charts, maps, images, and text assets. Information architecture requires the model to establish a hierarchy among titles, descriptions, data, and action entry points. Code implementation requires the model to generate runnable page structures and styles. Result verification requires the model to locate defects based on screenshots or runtime results. Strength in any single stage is not enough to represent complete design capability.

    Therefore, evaluation cannot only check whether the final page “opens,” nor only whether the first screen “looks good.” A more reasonable approach is to break the task into observable stages: first-version generation quality, automated defect inspection, one-round repair effectiveness, final pass status, and cost and time. Only by comparing these stages in the same environment can we avoid mistaking tool differences, prompt differences, or human intervention differences for differences in model capability.

    2. Fair Evaluation Framework: Control Variables and Multiple Metrics

    The first principle of fair comparison is controlling variables. All evaluated models should use the same batch of assets, the same frontend scaffold version, the same design tokens, the same browser automation inspection permissions, the same repair budget, and the same review criteria. Prompts should use a unified template and retain only task-necessary information, avoiding additional implicit constraints for one model. Each task should allow a fixed number of repair rounds, for example one round, to observe the model’s ability to correct issues under defect feedback.

    The second principle is using multiple metrics. A single “pass rate” can easily hide differences: some models are stable in the first version but cost more; some are low-cost and fast but have higher first-version defect density; some can generate a more complete product but rely more on a browser-screenshot closed loop. At minimum, the following metrics should be recorded:

    • Functional pass: whether key interactions, data linkage, routing, and state switching work.
    • First-version stability: the proportion of tasks that pass without repair after initial generation.
    • Visual quality: information hierarchy, whitespace, type scale, color consistency, and brand-character alignment.
    • Responsive quality: whether overflow, truncation, occlusion, and horizontal scrolling appear at desktop and mobile breakpoints.
    • Accessibility: basic checks such as semantic structure, contrast, focus states, and alternative text.
    • Repair efficiency: the proportion of defects resolved in one repair round given a defect list.
    • Closed-loop dependency: whether the model must rely on screenshot diagnosis to find key issues.
    • Cost and latency: resource consumption and time across generation, repair, and inspection.
    • Reproducibility: result fluctuations and defect-type distribution across repeated runs of the same task.

    The third principle is blind review and archiving. Visual quality should be scored by multiple reviewers with model identities hidden, while original screenshots, defect lists, before-and-after repair differences, and run logs are retained. This reduces subjective bias and supports later reproduction and audit.

    3. Three Distinct Scenario Types: Covering Different Capability Combinations

    To avoid task homogenization, three differentiated scenario types should be designed. The first is data storytelling, such as a scrolling ocean memory exhibition: year changes drive a temperature curve, and clicking a sea region switches the map and risk panel. This task tests original visuals, data narrative, chart linkage, and state management. The second is brand showcase, such as a product launch page: the first screen should have a clear visual proposition, clear information hierarchy, a prominent call to action, and maintained rhythm on mobile. This task tests brand character, layout control, and responsive adaptation. The third is information-intensive work, such as an operations dashboard or a complex form: tables, filters, empty states, error states, and loading states should be complete, with clear interaction feedback. This task tests information density control, component consistency, and boundary-state handling.

    The three task types should not use the same scoring weights. Data storytelling should increase the weight of chart linkage and narrative structure; brand showcase should increase the weight of visual hierarchy and consistency; information-intensive tasks should increase the weight of functional completeness and accessibility. Weights should be fixed before evaluation to avoid result-oriented tuning.

    4. Execution Pipeline: Reproducible Evaluation Configuration

    The execution pipeline can be divided into six steps: preparation, first-version generation, automated inspection, one-round repair, retest, and archiving. In the preparation stage, assets, scaffold versions, and the browser environment are frozen. In the first-version generation stage, only the unified task description and assets are provided. In the automated inspection stage, screenshots are taken at multiple viewports and basic interactions are executed. In the repair stage, only a structured defect list is provided, without additional design suggestions. In the retest stage, checks are repeated and differences are recorded. In the archiving stage, reports are saved. An illustrative configuration is as follows:

    1. case_id: data_story_001
    2. inputs:
    3. data: ./data/annual_series.csv
    4. media: ./assets/
    5. constraints:
    6. viewports: [375, 768, 1440]
    7. must_have:
    8. - scroll_driven_chart
    9. - region_switch
    10. - risk_panel
    11. repair_budget: 1
    12. evaluation:
    13. functional: pass_fail
    14. visual: 1_to_5
    15. accessibility: basic_checks
    16. responsive: overflow_truncation_focus

    The pseudocode can be abstracted as:

    1. def run_case(case, model, env):
    2. env.reset(case)
    3. first = model.generate(case.prompt, case.assets)
    4. defects = inspect(first, viewports=case.viewports)
    5. repaired = model.repair(first, defects, budget=1) if defects else first
    6. final_defects = inspect(repaired, viewports=case.viewports)
    7. return report(first, repaired, defects, final_defects)

    Here, inspect should cover screenshot diffing, element out-of-bounds checks, text truncation, key interaction reachability, and basic semantic checks. The report must retain both the first version and the repaired version, avoiding the practice of showing only the final result while ignoring first-version stability.

    5. Interpreting Results: Pass Rate Is a Gate, Not a Ranking

    In an internal reproducible experiment, different models eventually all passed the tasks, but their processes differed clearly: some models were more stable in the first version with lower defect density; some had better cost and latency but slightly weaker first-version stability; some could generate a more complete product structure but relied more on a browser-screenshot closed loop to find and fix issues. These differences show that the final pass rate can only serve as a gate metric, not as a direct capability ranking.

    A more reasonable interpretation should combine first-version defect types, repair rounds, closed-loop dependency, and cost and latency. If a model is stable in the first version but consumes more resources, it is suitable for scenarios with high determinism requirements. If another model is low-cost and fast but requires denser automated inspection as a fallback, it is suitable for batch drafts and rapid exploration. Engineering teams do not need to pursue a single all-round model; instead, they should design multi-model routing based on task risk: high-value brand pages should prioritize first-version stability, internal prototypes should prioritize cost and latency, and screenshot checks and defect lists should be used uniformly as quality gates.

    6. Engineering Recommendations

    First, version the evaluation set. Assets, prompts, scaffolds, inspection scripts, and scoring sheets should all be placed under version control to ensure historical results are reproducible. Second, standardize defect classification. Encode defects such as overflow, truncation, occlusion, misalignment, interaction failure, and insufficient contrast to make it easier to count model weaknesses. Third, combine automated inspection with human blind review. Automated inspection handles objective defects, while human review handles brand character, narrative rhythm, and visual hierarchy. Fourth, set a repair budget. Unlimited repair can hide differences in first-version capability; fixed rounds are closer to real delivery constraints. Fifth, cost accounting should cover generation, repair, screenshot inspection, and human review, rather than comparing only single-call cost. Sixth, establish security and compliance checks to ensure asset authorization, data masking, and traceability of generated content.

    7. Conclusion

    Evaluation of multimodal models on open-ended design tasks should move from “can it recognize” to “can it complete a runnable, inspectable, and repairable design closed loop.” The key to fair comparison is not stacking promotional demos, but controlling variables, fixing budgets, recording multiple metrics, and archiving blind reviews. Pass rate is a gate; first-version stability, visual hierarchy, responsive quality, closed-loop dependency, and cost and latency together form a capability profile. Only by engineering the evaluation method can multimodal capability comparison move from subjective impressions to reproducible technical judgment, providing a reliable basis for practical model selection and quality governance.