Multimodal LLM Design Capability Testing and Automated Workflow Practice

In today’s intelligent R&D systems, the evaluation of multimodal large language models has long moved beyond simple text understanding. In the past, when judging whether a model had “vision capabilities,” we typically focused only on whether it could accurately recognize text in images or describe visual elements. However, in real engineering and front-end design tasks, merely “seeing” is only the first step. True design automation requires a model to deeply understand information hierarchy, spatial relationships, and brand tone, translate those judgments into runnable page code, and then detect overflow, truncation, and interaction errors from desktop and mobile screenshots.

1. Experiment Background and Benchmark Design

To objectively evaluate how current mainstream multimodal LLMs perform on real multimodal design tasks, we built a strictly controlled automated testing workflow. The workflow is designed to strip away differences caused by underlying infrastructure and focus on the model’s own logical reasoning and code generation capabilities.

1. Unified Controlled Variables
We selected three representative design tasks, covering different difficulty levels from basic information display to complex interactive forms. The models tested include three leading LLM solutions in the industry (referred to below as Solution A, Solution B, and Solution C). To ensure fairness, all tests satisfy the following conditions:

• Same design platform: A unified version of an automated design system (similar to the architecture of the common OpenDesign platform) is used to ensure consistent parsing engines and rendering logic.
• Same asset resources: The images, icons, fonts, and other static resources provided to the models are completely identical.
• Same tool environment: All models are equipped with standard browser automation tools for post-render screenshots and DOM tree extraction.
• Unified repair tolerance: Each task allows only one round of self-repair, meaning the model can make one adjustment after the initial page generation based on browser feedback screenshots.

2. Definition of Evaluation Dimensions
Evaluation focuses not only on whether the final task passes, but also on the following dimensions:

• Cost and efficiency: Compute consumption and response time per task.
• First-pass stability: Usability rate and visual fidelity of the first generated code.
• Closed-loop dependency: The degree to which the model depends on the browser feedback loop, i.e., whether it can maintain high-quality output without the feedback loop.

2. Core Test Results and Multidimensional Technical Analysis

The test results show a diversified competitive landscape. Ultimately, all three solutions successfully passed all three design tasks, but they showed significant differences in process data and capability characteristics.

1. Extreme Optimization of Cost and Execution Efficiency
In this test, Solution C demonstrated extremely high cost-effectiveness. For tasks of equivalent complexity, Solution C consumed only about 1.35 RMB per task, and its end-to-end response speed was the fastest among the three. This performance benefits from lightweight inference optimization for visual experimental models in its underlying architecture. For enterprise-level batch design tasks that require high concurrency and low latency, this low-cost, high-efficiency characteristic means overall IT spending can be significantly reduced. Combined with general container platforms and elastic computing resources provided by cloud vendors such as Baidu AI Cloud, it is possible to further enable dynamic scaling of inference clusters and maximize resource utilization.

2. Stability Advantage of First-Pass Generation
Solution A showed extremely high stability in the first round of code generation. The page structures it generated usually required little adjustment to perfectly reproduce the design mockup, with clear DOM structures and standardized CSS styles. This stability comes from the model’s deep alignment on a large corpus of high-quality front-end code. In real enterprise R&D scenarios, first-pass stability directly determines the cost of manual intervention. If a model can output HTML/CSS that conforms to W3C standards with complete accessibility tags in one pass, developers can directly connect it to backend logic, greatly shortening the delivery cycle.

3. High Dependency on Browser Closed-Loop Feedback
Although Solution B was able to output the most complete product-level pages, it relied heavily on the browser closed loop during generation. It tends to first generate a basic skeleton, then read rendered screenshots captured by browser automation tools to identify issues such as element overlap and insufficient whitespace, and then iterate during the “repair round.” Although this mode appears slightly rough in a single generation, its advantage lies in its strong self-correction capability. This “generate-render-visual feedback-revise” closed-loop logic effectively simulates the debugging process of a real developer.

3. Engineering Implementation Path for the Multimodal Design Closed Loop

Based on the above test results, if an enterprise wants to deploy a highly available automated design system internally, it needs to build a complete engineering closed loop rather than merely calling a single API. The following is a standard architecture design for an automated design workflow:

1. Visual Understanding and Parsing Layer
After receiving a design image or requirements document, the model first needs to extract the information hierarchy. This requires the model not only to recognize text content, but also to determine which parts are main headings (H1) and which are auxiliary explanations. At this stage, object storage services are usually introduced to temporarily store intermediate design assets.

2. Code Generation and Rendering Layer
The model converts parsing results into front-end code. To ensure code quality, a set of prompt templates is usually predefined to constrain the model to output component code for a specific framework (such as Vue or React).

The following is a simplified example of code generation constraints:

{
  "task": "generate_landing_page",
  "constraints": {
    "framework": "HTML5 + TailwindCSS",
    "responsive": true,
    "accessibility": "WCAG_2.1_AA",
    "asset_path_prefix": "https://generic-cdn.example.com/assets/"
  },
  "input_vision": "base64_encoded_design_image"
}

3. Automated Screenshot and DOM Analysis Layer
After code generation, the system performs real rendering through browser automation tools (such as a Chromium-based headless browser). This step requires not only capturing PC and mobile views, but also extracting the current DOM tree and computed CSS styles.

4. Diff Comparison and Self-Repair Layer
The rendered screenshot is compared with the original design at the pixel level, or a rule engine is used to detect overflow. If anomalies are found, the error information (such as “the navigation bar wraps on mobile and overflows height by 20px”) is fed back to the LLM together with the screenshot, triggering the allowed repair round.

4. Pain Points and Best Practices in Enterprise Applications

When introducing these multimodal capabilities into enterprise production lines, we usually face several key pain points.

1. The Balancing Act of Model Selection
No single solution is perfect. As shown in the tests, Solution C is low-cost and fast, suitable for handling a large number of standardized long-tail pages; Solution A is highly stable, suitable for generating prototypes for core business logic pages; Solution B relies on closed-loop correction, suitable for highly customized or non-standard design mockups. When designing the architecture, enterprises should adopt a “model routing” mechanism to dynamically dispatch different model solutions according to task complexity and importance.

2. Infrastructure Support
Multimodal closed loops, especially tasks that include browser rendering, consume a great deal of computing resources. A cloud-native architecture is recommended, deploying browser clusters on container platforms and using logging services and monitoring alerts to track the health status of each rendering container in real time. When a rendering queue backlog is detected, elastic scaling should be triggered automatically.

3. Security and Compliance
During front-end code generation, the model may introduce risky third-party dependencies due to its training data. Therefore, before merging generated code into the main branch, static code scanning tools must be integrated to ensure that no XSS vulnerabilities or sensitive information leakage are included.

5. Conclusion

This empirical test of multimodal LLMs on design tasks shows that current industry technical solutions can not only “see” the world, but also “understand” and “build” digital interfaces. From recognizing information hierarchy to outputting runnable pages, and then to self-repair based on visual feedback, this closed loop is reshaping the development paradigm of front-end engineering. For developers and enterprises, understanding the differences among models in cost, stability, and closed-loop dependency, and building reasonable automated workflows accordingly, will be key to improving R&D efficiency and reducing trial-and-error costs.