Multimodal Design Tasks Evaluated: A Comprehensive Assessment of Visual Generation and Interaction Capabilities

1. Background and Objectives

As multimodal models continue to evolve, visual recognition is no longer a rare capability. Recognizing text, describing images, and identifying content only prove that a model can “see,” but they do not prove it can handle tasks closer to engineering delivery. The real question worth exploring is: Can multimodal models complete a full design loop—from visual understanding to page generation, interaction implementation, and state recovery—under a unified toolchain, with consistent assets and constraints?

To answer this, I designed a comparative experiment, assigning three progressively difficult design tasks to three different models. All models used the same version of the design tool, the same set of assets, and the same browser automation tool. Each task allowed for one round of browser-inspected repair. Evaluation metrics included first-draft quality, visual originality, interaction correctness, dependency on browser-based debugging, time consumed, and resource consumption.

2. Experimental Environment and Task Design

To ensure fairness, variables were strictly controlled. All models shared the same version of an open-ended design tool, which supports generating page structures, styles, and interaction logic from natural language instructions, and allows runtime inspection and repair via browser tools. The asset library included oceanographic data, map tiles, satellite imagery, and audio metadata, none of which contained any branded elements.

The experiment consisted of three tasks:

1. Ocean Memory Exhibition: Inputs included 147 annual data points, world map regional information, maritime data, and a high-altitude image. The model was required to generate a scrollable narrative page where year changes drive a temperature curve, and clicking a sea area switches the map focus and risk alerts. This task focused on original visual expression and data storytelling.
2. Music Map Interaction: The model needed to complete map selection, playback control, track switching, favoriting, playlist synchronization, and state recovery after a refresh. This task closely resembled real-world product requirements, focusing heavily on the completeness of interaction logic rather than mere visual aesthetics.
3. Multi-Source Dashboard: Combining multiple data sources, the model was tasked with building a high-density, hierarchically clear dashboard, ensuring no layout collapse or missing elements during actual browser rendering. This task tested the model’s ability to organize complex information architectures and its runtime repair efficiency.

The three models were labeled Model A, Model B, and Model C. To avoid bias, their sources were not disclosed during the experiment; only behavioral performance and quantitative results were recorded.

3. First-Draft Quality Comparison: Stability vs. Impact

The results of the first task revealed distinct stylistic differences.

Model A delivered the most magazine-like first screen. It split the theme words into two massive text blocks overlaid on the image, creating a strong visual impact. While maintaining information readability, this approach demonstrated a strong grasp of visual rhythm. More importantly, Model A was the most stable in its first drafts across all three tasks, with almost no severe issues requiring a complete redo. For teams prioritizing first-draft delivery efficiency, this stability is highly valuable.

Model B took a restrained approach. A dark starry background paired with a large title and centered descriptions resulted in clean information but slightly weaker visual tension. This style suits data-first scenarios and indicates that Model B tends to avoid over-designing, leaving complexity to the interaction layer.

Model C fell somewhere in between. It also used a large title and high-altitude image background but added a side data scale, achieving high completeness. Visually, it was less aggressive than Model A but richer than Model B. However, Model C encountered an abnormal map area height issue in its first draft, which was only resolved after locating and fixing it via browser inspection.

4. Interaction Implementation: Product Completeness and Debugging Dependency

The second task was the critical differentiator for engineering capabilities. The Music Map required not only correctly parsing reference images but also executing multi-step interactions: playback, track switching, favoriting, synchronization, and state recovery. Any missing logic would break the product loop.

Model B delivered the most complete product for this task. All interaction paths were connected, and state recovery functioned correctly. However, the results also showed that Model B had the strongest dependency on the browser loop. That is, it required repeated runtime feedback from the browser to correct DOM structures, event bindings, and styling issues. Without the browser inspection phase, the reliability of its first draft would drop significantly.

Model A had a high interaction correctness rate in its first draft, with almost no obvious flaws in map selection and player state management. It consumed the least time in browser inspection rounds, indicating that its first-draft code quality was closer to a deployable state.

Model C experienced an issue in its first draft where the map container height collapsed from a normal value to a minimal size. This problem could not be directly identified by statically reading the code; it only surfaced during actual browser rendering. The repair process indicated that when layout systems lack explicit constraints, Model C is prone to omissions when handling percentage heights and minimum height attributes.

5. Cost and Efficiency: Breaking the “More Expensive is Better” Assumption

Quantitative results showed that while all three models ultimately passed all tasks, their resource consumption varied significantly.

Model C completed all tasks for a cost of only approximately $0.19 (1.35 RMB equivalent), with the shortest average time. This result is noteworthy because it breaks the simple assumption that higher quality necessarily corresponds to higher cost. In real-world R&D scenarios, if a task is of medium complexity and allows for one round of automated repair, the differences in cost and latency might impact decision-making more than subtle differences in output quality.

Model A was the most stable in its first drafts, meaning it required the fewest rework rounds. For environments that cannot provide browser-based debugging, or scenarios aiming to minimize interactive repairs in CI pipelines, Model A’s stability translates to lower manual intervention costs.

Model B performed comprehensively in the task with the highest interaction complexity but required more debugging rounds. If placed in a toolchain with weak browser automation capabilities, its advantages would be diminished. Therefore, Model B is better suited for teams with mature browser debugging pipelines.

6. Visual Understanding to Engineering Delivery: Rethinking Capability Layering

The core findings of this experiment can be abstracted into a three-layer capability model:

The first layer is visual parsing capability. All three models passed this layer, proving that “seeing” is no longer a differentiating threshold.

The second layer is structured generation capability. The model must translate visual understanding into reasonably structured page components, style rules, and layout constraints. Differentiation begins to appear here, especially in handling height, stacking, and responsive constraints.

The third layer is engineering loop capability. Whether a model can correct its own errors based on runtime feedback and achieve a usable state within a limited number of rounds is the true watershed determining its readiness for production workflows.

Many evaluation systems only cover the first two layers, often ignoring the third. However, experimental data shows that even if a model “passes,” one might spend only $0.19 and complete the task quickly, while another might require more debugging rounds and time. For engineering teams, this difference is more critical than mere output quality.

7. Model Selection Recommendations for R&D Scenarios

If a team’s core priority is first-draft stability to minimize manual repair and interactive debugging, the more stable model is the appropriate choice. Its output quality fluctuates less, making it suitable for integration into standardized CI pipelines.

If a team possesses a comprehensive browser automation debugging environment, and the project has high interaction complexity requiring a complete product loop, a model with higher interaction coverage will be a better fit.

If a team is sensitive to cost and speed, dealing with large-scale, high-frequency tasks while allowing for a single round of automated repair, the low-cost, high-efficiency model offers a clear advantage. This is especially true in scenarios like batch-generating design drafts, A/B testing assets, or prototype validation, where cost benefits are amplified.

8. Conclusion and Outlook

This experiment ultimately reached a seemingly plain but profound conclusion: all three models completed the tasks, but their methods, costs, and stability were drastically different. The capability assessment of multimodal models should not stop at “can it read an image,” but must go deeper into “can it produce deployable engineering output based on visual inputs.”

For enterprise R&D teams, building a comprehensive evaluation pipeline that includes visual understanding, structured generation, runtime inspection, automated repair, and state validation is more practically meaningful than simply chasing leaderboard rankings. As model capabilities further converge in the future, the maturity of the toolchain and repair efficiency will likely become the key variables impacting the delivery quality of multimodal design tasks.