A Fair Comparison Methodology for Multimodal LLMs on Design Tasks

    This article presents an end-to-end methodology for comparing multimodal large language models on real design tasks. It explains why image recognition alone is insufficient for design work, describes a controlled experiment with unified tasks, toolchain, repair budget, and acceptance criteria, and evaluates models on cost, latency, first-pass stability, and tool dependency. It then summarizes three distinct capability profiles and offers practical guidance for engineering teams building design-oriented AI workflows.

    A Fair Comparison Methodology for Multimodal LLMs on Design Tasks

    1. Why “Can Recognize Images” Does Not Mean “Can Do Design”

    After multimodal large models added visual capabilities, the most common validation method has been to ask them to describe an image or recognize text. Such tests can prove that a model “can see,” but they cannot answer the question engineering teams really care about: can it complete a real design task?

    Design work is far more complex than image understanding. A qualified design deliverable requires a model to simultaneously understand the information hierarchy behind the copy and decide which content should be emphasized or downplayed; grasp spatial relationships and reasonably allocate whitespace, alignment, and grids; perceive brand character and maintain a consistent visual language across details such as color palette, font weight, and corner radius; and finally translate all these judgments into runnable page code. It must also be able to inspect rendered screenshots from desktop and mobile viewports, identify issues such as overflow, truncation, misalignment, and broken interactions, and fix them on its own.

    In other words, a design task is a comprehensive test of the full multimodal chain: perception, reasoning, generation, and self-checking. This article therefore designs an experiment: under the same constraints, three mainstream multimodal models each complete three design tasks, and their actual performance is observed.

    2. Experiment Design: How to Ensure “Fairness”

    The most common mistake in comparative experiments is inconsistent environments—Model A gets higher-resolution assets, Model B gets an extra repair round, and the conclusions lose meaning. This experiment严格控制 variables across four dimensions:

    1. Unified task set. The three design tasks cover typical scenarios, such as a marketing landing page, a data dashboard, and a mobile campaign page. Difficulty increases progressively, and each task includes clear copy assets and visual assets.

    2. Unified toolchain. All models use the same version of an open-source design framework, the same batch of image and font assets, and the same browser automation tool. The browser tool renders pages and captures screenshots in both desktop and mobile viewports for model self-checking.

    3. Unified repair budget. Each task allows one repair round: after receiving the first-version rendered screenshot, the model has one opportunity to analyze visual defects and modify the code. A single repair round both simulates the real-world “review—rework” rhythm and prevents unlimited iteration from masking differences in first-version quality.

    4. Unified acceptance criteria. Acceptance does not rely on subjective aesthetic scoring. It is based on checkable objective items: whether text is complete without truncation, whether the layout overflows its container, whether mobile adaptation works, whether interactive elements are clickable, and whether assets are referenced correctly.

    3. Evaluation Dimensions: Cost, Speed, First-Version Quality, and Loop Dependency

    In addition to pass rate, the experiment records four key metrics:

    • Cost: Calculated from actual input and output tokens consumed, reflecting the cost threshold for scaled use.
    • Latency: Total time from task assignment to final acceptance, affecting the experience of interactive design workflows.
    • First-version stability: The number of defects in the first generation before repair, measuring the model’s ability to “get it right the first time.”
    • Tool dependency: The degree to which a model relies on the browser loop of “screenshot—analysis—repair” to meet the standard, reflecting the maturity of its visual self-supervision.

    4. Results: Three Different Capability Profiles

    Under strictly identical conditions, all three models eventually passed acceptance, but their paths to success were very different, showing clear differentiated profiles.

    The first model had the most stable first-version quality. Its generated pages made the fewest mistakes in information hierarchy and spatial layout. For most tasks, the first-version screenshot was already close to the acceptance standard, and the repair round involved only minor adjustments. This means that in “one-shot delivery” scenarios, such as batch generation of static pages, it can significantly reduce manual intervention.

    The second model had the highest completeness but depended most on the closed loop. It tended to deliver a more fully featured product form—for example, components with interactive states and richer page structures—but its first version often had overflow or mobile adaptation issues and required browser screenshot feedback to complete repairs. It is suitable for workflows with a strong toolchain and multiple iterations.

    The third model’s economics and speed were surprising. The total cost of completing all three tasks was only about CNY 1.35, and its total time was the shortest. Its first-version quality was not the best, but its repair efficiency was high, converging after a single repair. It has clear advantages in cost-sensitive, throughput-oriented scenarios, such as batch exploration of design drafts.

    5. Implications for Engineering Practice

    The value of this experiment is not in selecting the “strongest model,” but in providing a reusable evaluation paradigm and several practical recommendations.

    First, evaluation should be end-to-end. Looking only at isolated capabilities, such as image-description accuracy, cannot predict performance on design tasks. Evaluation must combine task pass rate, repair rounds, and cost.

    Second, closed-loop tools are amplifiers. Equipping models with rendering and screenshot capabilities can turn visual weaknesses into automatically fixable problems. This kind of “environment compensation” is often more cost-effective than switching models.

    Third, choose by scenario rather than by ranking. First-version-stable models suit low-interaction batch production, loop-dependent models suit complex interactive products, and low-cost fast models suit high-frequency draft exploration. Enterprises can deploy inference services on general compute platforms and logging/monitoring systems, routing tasks to different models according to task characteristics to balance quality and cost.

    Fourth, objective acceptance criteria should come first. Only by breaking “looking good” into machine-checkable items such as overflow, truncation, adaptation, and interaction can multi-round model comparisons become repeatable scientific experiments rather than one-off subjective evaluations.

    As the visual capabilities of multimodal models continue to evolve, this fair comparison method—“same tasks, same assets, same tools, same acceptance”—can become a routine benchmark for teams and provide a long-term, credible basis for technical decisions in design intelligence.