Evaluating Multimodal Large Models for Open-Ended Design: Methods and Practices

    This article presents a practical evaluation framework for multimodal large models in open-ended design tasks. It explains why evaluation must move beyond basic image recognition to tasks that require information hierarchy, spatial reasoning, brand consistency, code generation, responsive behavior, screenshot-based visual regression, and repair within a limited feedback loop. The article covers three representative design tasks, standardized input and runtime controls, multidimensional scoring, process logging, automated screenshot checks, cost efficiency, result interpretation, and engineering integration.

    1. Background: From Image Recognition to Design Execution

    As multimodal model capabilities continue to improve, evaluation should shift from isolated recognition abilities to composite task completion. Recognizing text, describing an image, or judging image content generally shows that a model has basic visual perception. Open-ended design tasks impose higher requirements: the model must not only “see” and “understand,” but also “execute” and “verify.”

    A real page design task usually involves multiple layers:

    1. Information hierarchy judgment: identifying which elements are primary headings, supporting descriptions, buttons, labels, tables, or form fields.
    2. Spatial relationship reasoning: determining whether elements are aligned, stacked, placed in columns, or floating; whether spacing is reasonable; and whether the visual focus is clear.
    3. Brand tone consistency: checking whether colors, typography, corner radii, shadows, and image style are consistent with the provided assets and theme.
    4. Code generation capability: converting design judgments into runnable page structure, styles, and interaction logic.
    5. Visual regression capability: identifying overflow, truncation, misalignment, occlusion, and invisible buttons in desktop and mobile screenshots.
    6. Repair loop capability: when one repair round is allowed, determining whether the model can quickly locate and fix issues based on screenshot feedback.

    Therefore, open-ended design evaluation for multimodal large models should not only ask whether the model “can see the image.” It should ask:

    • Can the model generate a reasonable structure based on the provided assets?
    • Can it maintain a consistent experience across desktop and mobile?
    • Can it perform effective repair after screenshot feedback?
    • Can it complete the task reliably within a reasonable cost range?

    This type of evaluation is closer to real engineering scenarios and better reflects a model’s overall capability in page construction, prototype implementation, and automated verification.


    2. Evaluation Goals: A Reproducible, Comparable, and Explainable Design Capability System

    The difficulty of open-ended design tasks lies in their high degree of freedom. Unlike fixed visual question answering tasks, open-ended design usually does not have a single correct answer. Therefore, evaluation design must address three problems:

    1. Fair comparison: different models must complete the task under the same input, same assets, same tool environment, and same repair rules.
    2. Traceable process: evaluation should not only look at the final screenshot. It should also record the first output, repair process, failure reasons, and cost consumption.
    3. Explainable results: scoring should not rely only on subjective impression. It should be broken down into dimensions such as functionality, visual quality, responsiveness, interaction, and cost.

    Based on this, a unified evaluation loop can be established:

    1. Task specification
    2. → Asset preparation
    3. → First model generation
    4. → Automatic screenshots and structure checks
    5. → One repair round
    6. → Second screenshots and interaction validation
    7. → Multi-dimensional scoring
    8. → Cost aggregation
    9. → Result archiving

    The key to this loop is “controlling variables.” Only when the input, environment, and acceptance criteria are consistent does comparison between different models become meaningful.


    3. Task Design: Three Typical Open-Ended Design Tasks

    To simulate real design scenarios, it is better to design three complementary tasks rather than a single simple page. These tasks should cover different information structures, interaction complexity, and responsive requirements.

    Task 1: Campaign Landing Page

    Goal: Generate a marketing campaign landing page based on brand colors, a hero visual, campaign title, and button copy.

    Key focus areas:

    • Whether the above-the-fold information is prominent;
    • Whether the main button is clear and clickable;
    • Whether image cropping is reasonable;
    • Whether there is no horizontal scrolling on desktop and mobile;
    • Whether bottom content is blocked or truncated.

    This task mainly examines the model’s ability to prioritize marketing page information. Marketing pages usually have concentrated content and strong visual guidance, making them suitable for observing whether the model can correctly identify the hierarchy among “main title—selling points—call-to-action button.”

    Task 2: Product Detail and Purchase Conversion Page

    Goal: Generate a browsable product detail page based on product images, parameter descriptions, price information, review summaries, and purchase buttons.

    Key focus areas:

    • Whether the main product image is fully displayed;
    • Whether the parameter table is readable;
    • Whether the price module is prominent;
    • Whether the review section has a clear structure;
    • Whether the sticky mobile button covers body content;
    • Whether long text causes overflow or abnormal line breaks.

    This task places more emphasis on information density and structured presentation. The model must not only generate a visual page, but also handle complex elements such as tables, lists, labels, buttons, and long text.

    Task 3: Back-Office Data Overview Page

    Goal: Generate a clean data dashboard based on metric cards, trend charts, status labels, filters, and data lists.

    Key focus areas:

    • Whether metric cards are aligned;
    • Whether chart areas have enough whitespace;
    • Whether status label colors make semantic sense;
    • Whether filter controls are easy to understand;
    • Whether table column widths are balanced;
    • Whether empty states and loading states receive basic consideration.

    This task is more product-oriented. It examines whether the model can organize information within a limited space rather than simply stacking visual elements.


    4. Fairness Design: Unified Environment and Unified Acceptance Rules

    If open-ended design tasks lack unified constraints, the evaluation may easily become “whose output looks better” instead of “whose capability is stronger.” Therefore, the following rules should be clarified.

    1. Same Input Specification

    All models should use the same task description, the same assets, and the same constraints. The task description should specify:

    • The purpose of the page;
    • The target users;
    • Required content modules;
    • Prohibited placeholders;
    • Mobile and desktop adaptation requirements;
    • Whether external resources are allowed;
    • Whether a pure front-end implementation is required;
    • The submission format.

    2. Same Asset Library

    Assets should be as consistent as possible, including:

    • Brand hero visuals;
    • Product images;
    • Icon resources;
    • Copy text;
    • Sample data;
    • Color specifications.

    This prevents models from gaining or losing an advantage because of differences in assets.

    3. Same Runtime Environment

    Generated pages should be run and screenshotted in a unified environment. It is recommended to fix:

    • Desktop viewport size;
    • Mobile viewport size;
    • Browser automation waiting strategy;
    • Screenshot timing;
    • Font loading completion condition;
    • Image loading completion condition;
    • Network failure handling strategy.

    For example, the first-screen screenshot should be taken only after page resources have finished loading. This avoids misjudging model capability because rendering is incomplete.

    4. Same Repair Opportunity

    To observe a model’s closed-loop capability, one repair round may be allowed for each task. Repair must be based on screenshot feedback or automatic detection results, rather than manually rewriting the prompt sentence by sentence.

    Repair rules may include:

    • Only allowing the model to view the first screenshot;
    • Allowing the model to adjust styles, layout, and interactions;
    • Disallowing a complete redesign of the product direction;
    • Requiring new screenshots after repair;
    • Requiring the repair result to enter the scoring process again.

    This approach evaluates both first-pass capability and the model’s ability to make targeted corrections based on visual feedback.


    5. Scoring System: From Pass/Fail to Multi-Dimensional Scoring

    If evaluation only records “pass” or “fail,” too little information remains. It is recommended to use multi-dimensional scoring and break capability down into measurable indicators.

    1. Functional Correctness

    Evaluate whether the page meets basic functional requirements:

    • Are key buttons clickable?
    • Are forms editable?
    • Do links navigate correctly?
    • Is navigation interactive?
    • Do state switches work normally?
    • Are there blank areas on the page?

    2. Visual Fidelity

    Evaluate whether the output matches the task expectation:

    • Is the main visual prominent?
    • Is color usage reasonable?
    • Is font hierarchy clear?
    • Is spacing consistent?
    • Are images stretched or distorted?
    • Are elements aligned?
    • Are there obvious cropping or occlusion issues?

    3. Responsive Adaptation

    Open-ended design tasks should not be evaluated only on desktop. Both desktop and mobile should be checked:

    • Is there horizontal scrolling?
    • Is content truncated?
    • Are buttons too small?
    • Does text overlap?
    • Are tables hard to read?
    • Do images exceed their containers?

    4. Code Maintainability

    If the output includes code, evaluate whether its structure is clear:

    • Are tags semantic?
    • Are styles managed centrally?
    • Are there many repeated styles?
    • Are unreadable names used?
    • Is the code easy to modify later?
    • Is there obviously invalid code?

    5. Repair Effectiveness

    Repair capability should be scored separately. Key observations include:

    • Can the model accurately identify problems in the screenshot?
    • Does it fix problems without damaging the original design?
    • Can it handle mobile-specific issues?
    • Does it introduce new visual problems?
    • Does it complete one effective repair loop?

    6. Cost and Stability

    Cost should not be measured only by the price of a single output. A unified measurement standard should be established:

    • Input consumption;
    • Output consumption;
    • Number of calls;
    • Number of retries;
    • Total elapsed time;
    • Human waiting time;
    • Whether additional screenshot rounds are needed;
    • Whether additional repair rounds are needed.

    A relative cost score can then be formed to judge whether a model is usable in engineering scenarios.


    6. Process Records: Do Not Only Look at the Final Result

    For open-ended design tasks, process data is often more valuable than the final result. It is recommended to save the following records for each task:

    1. {
    2. "task_id": "landing_page_001",
    3. "model": "model_a",
    4. "first_generation": {
    5. "status": "success",
    6. "desktop_screenshot": "desktop_first.png",
    7. "mobile_screenshot": "mobile_first.png",
    8. "issues": ["mobile button overlap", "abnormal image aspect ratio"]
    9. },
    10. "repair": {
    11. "status": "success",
    12. "repair_reason": "Fix mobile layout based on screenshot feedback",
    13. "desktop_screenshot": "desktop_final.png",
    14. "mobile_screenshot": "mobile_final.png"
    15. },
    16. "score": {
    17. "function": 8,
    18. "visual": 7,
    19. "responsive": 9,
    20. "repair": 8,
    21. "cost": 9
    22. }
    23. }

    This allows teams to review common failure modes across models:

    • Is first-pass generation ability insufficient?
    • Is mobile adaptation awareness insufficient?
    • Is visual screenshot understanding insufficient?
    • Does repair damage the original structure?
    • Is the cost too high for the quality achieved?
    • Is the model fast but unstable in first-pass output?

    This process data helps teams determine whether a model is suitable for an automated design pipeline.


    7. Automated Screenshots and Visual Regression Checks

    Screenshot checking is a critical part of open-ended design evaluation. Many design issues may not be visible at the code level, but become obvious in the rendered result.

    Several types of automated checks can be designed.

    1. Page-Level Checks

    • Does the page have horizontal scrolling?
    • Is the page height abnormal?
    • Is there a large blank area?
    • Is there obvious occlusion?
    • Are key above-the-fold elements visible?

    2. Component-Level Checks

    • Are buttons within the visible area?
    • Do labels exceed their containers?
    • Do images maintain their aspect ratio?
    • Are table columns squeezed?
    • Does text overflow its card?

    3. Interaction-Level Checks

    • Do button clicks trigger feedback?
    • Are form inputs editable?
    • Can navigation switch correctly?
    • Can dialogs be closed?
    • Do hover states work properly?

    An example check logic can be represented as follows:

    1. def check_page(page):
    2. issues = []
    3. if page.has_horizontal_scroll():
    4. issues.append("desktop_horizontal_scroll")
    5. if not page.is_visible("main_action_button"):
    6. issues.append("main_button_not_visible")
    7. if page.is_overflowing("card_title"):
    8. issues.append("card_title_overflow")
    9. if page.mobile.has_horizontal_scroll():
    10. issues.append("mobile_horizontal_scroll")
    11. if page.mobile.is_overlap("cta_bar", "footer_content"):
    12. issues.append("mobile_cta_overlap")
    13. return issues

    Such rules can cover most basic problems in open-ended design tasks and reduce variation caused by subjective judgment.


    8. Cost Efficiency: Not Only “Can It Pass,” but Also “Is It Worth Using”

    In engineering scenarios, model usability depends not only on quality but also on cost and speed. A model may be able to produce high-quality pages, but if it is too expensive, too slow, or frequently retries, it may still be unsuitable for an automated design pipeline.

    Cost evaluation can be considered from the following dimensions.

    1. Relative Cost

    Because different models are called in different ways, evaluation does not need to force conversion into a unified currency amount. Relative cost can be used instead:

    • Compare against the cost of a baseline task;
    • Compare against the lowest-cost task;
    • Compare against the average cost of high-quality passing tasks.

    2. Elapsed Time

    Record total elapsed time, including:

    • First generation time;
    • Screenshot waiting time;
    • Repair generation time;
    • Final validation time.

    For batch design tasks, elapsed time directly affects pipeline throughput.

    3. Retry Rate

    If a model frequently fails or needs multiple repairs, even a low per-call cost may cause additional resource waste. It is recommended to record:

    • First-pass success rate;
    • Post-repair success rate;
    • Automatic detection failure rate;
    • Manual review failure rate.

    4. Cost Efficiency Ratio

    A simple model can be established:

    1. Cost efficiency = overall score / total resource consumption

    The overall score can include functionality, visual quality, responsiveness, and repair capability. This indicator is not an absolute standard, but it helps compare the balance between quality and cost across models.


    9. Interpreting Results: Different Models May Show Different Capability Profiles

    Under the same task, same assets, same tool environment, and same repair rules, different models may show different characteristics. These results should not be simplified as “who is better overall.” They should be analyzed from an engineering perspective.

    1. First-Pass Stability Profile

    Some models produce relatively stable first-pass outputs with clear basic layout and few serious misalignments or occlusions. These models are suitable for quickly generating prototypes and reducing later repair costs.

    2. Repair-Loop Profile

    Some models may not produce the most complete first version, but can make relatively accurate local repairs based on screenshot feedback. These models are suitable for automated screenshot inspection workflows where one or a limited number of repair rounds can improve result quality.

    3. Cost-First Profile

    Some models achieve good quality with lower resource consumption and faster response. These models are suitable for batch generation of initial drafts, followed by human refinement or refinement by higher-quality models.

    4. Product Completeness Profile

    Some models may be more inclined to generate a complete product structure, including more page states, interaction areas, or module layers. Such outputs are suitable for prototype demonstrations, but may also increase additional validation costs.

    These profiles do not have absolute advantages or disadvantages. The key is to match business goals:

    • If the goal is a fast draft, focus on first-pass quality;
    • If the goal is automated repair, focus on screenshot understanding and repair accuracy;
    • If the goal is batch production, focus on cost and stability;
    • If the goal is a high-fidelity prototype, focus on visual completeness and interaction details.

    10. Engineering Adoption: Integrating Evaluation into Continuous Regression

    When open-ended design tasks move from one-time experiments to long-term evaluation, they should be incorporated into a continuous regression process.

    1. Build a Task Set

    Common page types should be accumulated into a task set, such as:

    • Campaign landing pages;
    • Product detail pages;
    • Content reading pages;
    • Back-office dashboard pages;
    • Form submission pages;
    • Marketing topic pages.

    Each task type should have fixed acceptance criteria to prevent evaluation standards from drifting over time.

    2. Build an Automated Execution Pipeline

    A usable evaluation pipeline can include:

    1. Task library
    2. → Prompt templates
    3. → Model invocation
    4. → Page rendering
    5. → Automatic screenshots
    6. → DOM checks
    7. → Visual rule checks
    8. → Interaction script checks
    9. → Score aggregation
    10. → Report generation

    3. Build a Baseline

    Each evaluation should be compared with historical baselines, including:

    • First-pass success rate;
    • Post-repair success rate;
    • Average elapsed time;
    • Average cost;
    • Common error types;
    • Manual review scores.

    This helps determine whether model upgrades, prompt optimization, or environment changes bring real improvements.

    4. Introduce Human Sampling

    Automated checks can solve basic problems, but design aesthetics and brand expression still require human sampling. It is recommended to sample results at a certain ratio for human scoring and to preserve representative samples for later optimization of evaluation rules.


    11. Common Risks and Cautions

    Open-ended design evaluation also needs to avoid several common pitfalls.

    1. Only Looking at the Final Screenshot

    The final screenshot reflects the result but does not explain the reason for failure. The first version, repaired version, failure logs, and automatic check records must all be preserved.

    2. Using Too Narrow a Task Set

    A single task can easily cause evaluation bias. It is recommended to cover at least three different structures: marketing pages, detail pages, and back-office pages.

    3. Inconsistent Repair Rounds

    If different models have different repair opportunities, results cannot be directly compared. Repair rounds, repair evidence, and repair boundaries should be clearly defined.

    4. Ignoring Mobile

    Many models perform well on desktop but show button occlusion, text overflow, or layout squeezing on mobile. Mobile checks must be mandatory.

    5. Over-Focusing on a Single Pass Rate

    Pass rate is only a basic metric. It should be combined with cost, stability, visual quality, and repair capability for comprehensive judgment.

    6. Ignoring Maintainability

    If generated code is difficult to maintain, even a visually complete page may be hard to include in later iteration. Code structure and readability should therefore be part of evaluation.


    12. Conclusion: The Key to Open-Ended Design Evaluation Is a Complete Loop

    Open-ended design capability in multimodal large models is not simply image recognition. It is a combination of visual understanding, information organization, code generation, visual regression, and repair loops. To evaluate this capability fairly, the evaluation system must cover task design, environment control, automatic screenshots, repair rules, scoring, and cost recording.

    A mature evaluation method should at least answer the following questions:

    • Can the model generate a reasonable page structure based on assets?
    • Can the model maintain basic usability on both desktop and mobile?
    • Can the model identify layout problems from screenshots?
    • Can the model improve the result within one repair round?
    • Is the model feasible in terms of cost and speed?
    • Is the model suitable for automated design or prototype generation workflows?

    When these questions can be verified in a stable, repeatable, and quantitative way, open-ended design evaluation is no longer a subjective experience, but an engineering capability assessment mechanism that can be continuously optimized.