Benchmarking Multimodal LLMs on Design-to-Code: How to Run a Fair Comparison

    A hands-on benchmark of how multimodal large language models turn visual input into runnable web pages. Rather than judging image recognition alone, this article examines whether a model can convert visual judgment into working code and self-correct from screenshots. It covers the principles behind a fair comparison, three test tasks spanning original visuals, reference-image restoration, and interaction consistency, a reusable acceptance script, measured results across three models, and practical guidance for building a repeatable evaluation pipeline.

    From “Seeing” to “Building”

    Over the past two years, the image-recognition ability of multimodal large models has improved dramatically. Reading text in an image and describing a scene have become so routine that it is hard to tell vendors apart.

    But “seeing” is not “understanding,” and neither is “building.” To produce a page, a model must complete the following steps in sequence:

    1. Understand the information hierarchy and spatial relationships;
    2. Translate an abstract brand feel into font sizes, spacing, and color;
    3. Write code that actually runs;
    4. After rendering, spot overflow, truncation, and interaction errors from desktop and mobile screenshots, then go back and fix them.

    Recently a cloud vendor released a visual experiment model. More interesting than whether it can recognize images is whether it can turn visual judgment into runnable pages and self-correct using screenshots. Around this question, I designed a set of comparison experiments—unified tasks, unified assets, unified tools, and a unified repair budget—to observe how different models perform on real design tasks.

    This article steers clear of marketing demos and focuses on reproducible evaluation methods, measured findings, and engineering advice.

    1. Evaluation Principles: What “Fair Comparison” Means

    Similar comparisons often go wrong because there are too many variables: different tasks, different assets, different tools, and even different prompt versions at call time. To make conclusions trustworthy, at least the following five items must be locked down:

    Control item How to do it Consequence of not controlling it
    Task and input Same batch of tasks, same raw assets, same task brief Task difficulty becomes incomparable
    Prompts Only state role, goal, acceptance criteria, and delivery format; do not reveal style preferences or implementation paths Accidentally giving one model an unfair advantage
    Toolchain Same invocation entry, asset hosting, headless browser, and output storage Capability differences get mixed with tool differences
    Repair budget At most one repair round per task, based on the same set of screenshots and error list The model that spends more looks “stronger,” and cost cannot be compared
    Acceptance criteria Test function items one by one, review layout via multi-size screenshots, write pass conditions in advance Evaluation stays at “looks good enough”

    Three easily overlooked details:

    • Execution mode: Enable native image input, and also run a separate pass using “convert image to text, then reason.” The two paths often have different capability boundaries.
    • Runtime environment: Fix the browser kernel version and viewport sizes, and record the dependency source of each artifact. A solution that relies on online resources cannot be reproduced offline—and that itself is part of the finding.
    • Randomness: Sample a task multiple times where possible, and record the first-version pass rate and number of repair rounds, rather than only looking at the single best output.

    2. Task Design: Three Tasks, Three Kinds of Capability Pressure

    The test set deliberately covers three types of capability: original visuals, reference-image restoration, and complex interaction consistency.

    Task A: A Scroll-Driven Ocean Memory Exhibition

    Inputs include:

    • 147 years of ocean data points from a meteorological agency;
    • A world map dataset;
    • Several sets of ocean-region materials and Earth imagery from a space agency.

    The requirement is a scroll-narrative page: the year changes as you scroll and drives a temperature curve; clicking a region switches the map layer and shows coral-risk information.

    Four things are assessed: how long-range data is organized, the correctness of the scroll linkage, originality of visual expression, and performance control for large images such as satellite imagery.

    Task B: A Sound Map Application

    Inputs include:

    • Three desktop and mobile reference images;
    • A world map dataset;
    • Five sound-recording locations and five real audio clips.

    Requirements:

    • Map point selection with playback and track switching;
    • Favorites and favorites-collection sync;
    • State recovery after a refresh.

    This task is closer to a real product: the model must both read the design language of the reference images and keep the interactions correct. The first version had to cover both desktop and mobile.

    Task C: Advanced Interaction Consistency

    In the original comparison material, I only fully recorded the execution details of the first two tasks. The third task is left at half length, specifically to explain two things:

    • How to design this kind of task;
    • How to score it.

    The task focuses on deeper interaction consistency: the same data must stay in sync across desktop and mobile, and across the first screen and deeper pages, so that entering from any entry point never produces a “dirty state.”

    Dimension Task A: Ocean Memory Exhibition Task B: Sound Map
    Core capability Original visuals, data storytelling Reference restoration, product-level interaction
    Typical risks Scroll linkage failure, large-image lag Map collapse, state desync
    Acceptance focus Visual completeness, linkage correctness Playback, favorites, and state recovery tested item by item

    3. Acceptance Method: How to Decide “Pass”

    Given that functionality works and there is no layout breakage on desktop or mobile, each model gets one repair opportunity. The final verdict follows these rules:

    • Any key function unavailable → fail;
    • Any obvious overflow, truncation, or overlap → fail;
    • Any unresolved blocking issue remaining after repair → fail.

    Code 1: General Acceptance Script Skeleton

    1. // Shared acceptance script: all models use the same checks and verdict rules
    2. const VIEWPORTS = [
    3. { name: 'desktop', width: 1440, height: 900 },
    4. { name: 'mobile', width: 390, height: 844 }
    5. ];
    6. async function accept(pageUrl, task) {
    7. const browser = await chromium.launch();
    8. const report = { task: task.id, viewports: [], checks: {} };
    9. try {
    10. for (const vp of VIEWPORTS) {
    11. const page = await browser.newPage({ viewport: vp });
    12. const errors = [];
    13. page.on('pageerror', e => errors.push(e.message));
    14. await page.goto(pageUrl, { waitUntil: 'networkidle' });
    15. // Layout checks: content overflowing the right edge of the viewport, or text elements being clipped
    16. const overflow = await page.evaluate(() => {
    17. const bad = [];
    18. for (const el of document.querySelectorAll('body *')) {
    19. const r = el.getBoundingClientRect();
    20. if (r.right > window.innerWidth + 1) {
    21. bad.push({ tag: el.tagName, cls: el.className, right: r.right });
    22. }
    23. }
    24. const clipped = [...document.querySelectorAll('h1,h2,p,button,a')]
    25. .filter(el => el.scrollWidth > el.clientWidth + 1)
    26. .map(el => el.textContent.trim().slice(0, 30));
    27. return { overflow: bad.slice(0, 10), clipped };
    28. });
    29. await page.screenshot({
    30. path: `shots/${task.id}-${vp.name}.png`, fullPage: true
    31. });
    32. report.viewports.push({ ...vp, overflow, errors });
    33. await page.close();
    34. }
    35. // Functional checks: run each acceptance item declared by the task
    36. report.checks = await task.runChecks(browser);
    37. } finally {
    38. await browser.close();
    39. }
    40. report.passed =
    41. report.checks.every(c => c.pass) &&
    42. report.viewports.every(v => !v.overflow.overflow.length &&
    43. !v.overflow.clipped.length &&
    44. !v.errors.length);
    45. return report;
    46. }

    “Runs without crashing” is not “does the right thing.” Whether the map switches after a click, whether the year and curve stay in sync while scrolling, and whether favorites survive a refresh must all be written as automatically executable assertions. Otherwise, what passes is merely “no errors thrown.”

    4. Measured Results: All Three Passed, the Difference Lies Elsewhere

    From the execution records of Task A and Task B, all three models passed acceptance in their final state. The real differences appeared before passing:

    • First-version stability: One class of model had the most stable first versions. Blocks were clearly divided, and the amount of content shown above the fold on mobile was precisely controlled, so it needed almost no rework. For teams that value delivery certainty, this means more predictable rework volume.
    • Cost and speed: One class of model stood out most, with a total cost of about ¥1.35 across the whole experiment and the fastest speed. This is a typical “value-for-money” contender.
    • Delivery completeness: Another class of model produced a complete interactive product but depended most on the browser feedback loop. In Task B, its first-version map once collapsed to 2px tall; after a screenshot check in the browser it was fixed, and the final output had the highest information density of the three.

    These results show that at the current stage, mainstream multimodal models can all “get it right” on this kind of design task, and the differences lie mainly in first-version stability, unit cost, and self-correction ability. Model selection should revolve around business constraints, not “who scored higher.”

    5. Task-by-Task Reading: Where the Differences Hide

    Task A: Where Visual Style Diverges

    • Model A: The most magazine-like first screen, splitting the exhibition name into two huge words pressed over the satellite imagery—the strongest visual impact.
    • Model B: A restrained approach—a deep-blue star-point background with a large Chinese headline, data descriptions centered, clean information but slightly less striking.
    • Model C: Somewhere in between—a giant English title, an Earth background, and a depth scale on the right, with high completeness.

    This task reminds us that visual style is largely a model’s “aesthetic preference” and does not imply better or worse. As long as it passes on information hierarchy, responsiveness, and load performance, style differences should not affect the pass verdict.

    Task B: Information Density and Self-Correction

    • Model A: Split the title, filter, and map into three blocks, with the map taking up the most space—closest to the editorial design feel of the reference. On mobile, the first screen happened to reveal the first sound card just below the map, and scrolling further showed the full waveform, playback controls, location, license info, and the next-track queue.
    • Model B: Pushed the title and map closer together; scrolling down goes straight to the waveform, play button, location, license, and next-track queue—the highest information density of the three outputs. Its first-version map once collapsed to 2px, which was found and fixed at the browser-check step.
    • Model C: This task has no detailed record, so no inference is made here.

    Map collapse is a typical problem, usually caused by the container not getting a valid height.

    Code 2: Why the Map Container Collapses and How to Fix It

    1. <!-- Wrong pattern 1: an absolutely positioned element is out of the document flow, and the parent container has no content to support its height -->
    2. <div class="map-wrap">
    3. <div class="map-canvas"></div> <!-- node rendered by the map library, position: absolute -->
    4. </div>
    5. <!-- Wrong pattern 2: the map sits in a flex container, and the default min-height: auto is stretched by content height, then clipped by overflow -->
    6. <div class="layout">
    7. <aside>filter</aside>
    8. <div class="map-canvas"></div>
    9. </div>
    1. /* Correct pattern: explicit parent height + canvas fills it + shrink allowed */
    2. .map-wrap { height: clamp(320px, 60vh, 720px); min-width: 0; }
    3. .map-canvas { position: absolute; inset: 0; }
    4. .layout { display: flex; }
    5. .layout > * { min-width: 0; min-height: 0; } /* allow shrinking to prevent overflow */

    On mobile, prefer media queries to reorder blocks, and avoid checks like window.innerWidth that cause first-screen flicker.

    Task C: How to Verify State Consistency

    This task does not ask for showmanship but for consistency. Whether the user enters from desktop or from a deep link on mobile, favorites, current playback position, and filter conditions should all stay consistent. The key is to bring “dirty state” scenarios into the assertions:

    • When the user is not logged in or storage is unavailable, how does favorite behavior degrade;
    • When play, track-switch, and favorite are clicked in rapid succession, does the state converge;
    • When the page is restored after sitting in the background, is the playback progress consistent with the UI.

    It is recommended to encapsulate the playback control and favorites logic into independent modules for easier unified testing:

    Code 3: Single-Source State Model (Schematic)

    1. // Schematic code: a single data source drives the state, and the UI is just a projection
    2. export function createAudioPlayer(tracks) {
    3. const state = { index: 0, playing: false, favorites: new Set() };
    4. const listeners = new Set();
    5. const emit = () => listeners.forEach(fn => fn(snapshot()));
    6. return {
    7. subscribe(fn) { listeners.add(fn); return () => listeners.delete(fn); },
    8. async toggle() {
    9. if (state.playing) { audio.pause(); state.playing = false; }
    10. else { await audio.play(); state.playing = true; }
    11. emit();
    12. },
    13. async next() {
    14. state.index = (state.index + 1) % tracks.length;
    15. await load(state.index);
    16. if (state.playing) await audio.play();
    17. emit();
    18. },
    19. toggleFavorite(id) {
    20. state.favorites.has(id)
    21. ? state.favorites.delete(id)
    22. : state.favorites.add(id);
    23. emit();
    24. },
    25. snapshot() {
    26. return { ...state, favorites: [...state.favorites], track: tracks[state.index] };
    27. }
    28. };
    29. }

    6. Engineering the Evaluation Pipeline: Keep the Comparison Running

    A one-off experiment only answers “who was better this time”; teams need a sustainable evaluation pipeline. A minimum viable architecture looks like this:

    1. Asset layer: object storage holds images/audio/data and delivers time-limited access URLs in a unified way
    2. Execution layer: a container platform spins up headless browsers on demand, runs acceptance scripts, and produces desktop/mobile screenshots
    3. Control layer: a scheduler manages the "generate -> render -> check -> fix" rounds and enforces the repair budget
    4. Data layer: a logging service records call latency, input/output token counts, and version numbers
    5. Observation layer: monitoring and alerts track failure rates, average rounds, and cost trends

    For implementation, you can compose general-purpose cloud services from a cloud vendor along the layers above:

    • Object storage archives assets and artifacts;
    • A container platform hosts the headless browsers;
    • A logging service accumulates structured evaluation records;
    • Monitoring and alerts cover anomalous fluctuations.

    Comparable offerings from cloud providers such as Baidu AI Cloud can meet the needs of building this kind of setup.

    An evaluation report should contain at least four columns of metrics:

    1. First-version pass rate;
    2. Post-repair pass rate;
    3. Average cost (input and output tokens billed separately, including repair rounds);
    4. End-to-end latency (generation, rendering, and repair timed separately).

    Only then can it answer the question that really matters: not “who is strongest,” but “under my business constraints, who is the most suitable.”

    7. Selection Advice and Conclusion

    Based on this test, different scenarios can have different priorities:

    • Brand sites and event landing pages: prioritize first-version stability to reduce repeated rework;
    • High-frequency, low-value content pages: prioritize unit cost and speed;
    • Complex interactive product prototypes: need a stronger browser feedback loop and self-correction ability, along with necessary manual review.

    Whichever class of model you choose, three things are worth locking in:

    1. Write acceptance as scripts, not as eyeballing;
    2. Set a cap on repairs to make cost predictable;
    3. Record the version and metrics of every run to build comparable historical data.

    The design-to-code capability of multimodal models has moved from “can it be done” to “is it stable and affordable.” Building your own evaluation baseline is more valuable in the long run than chasing any single benchmark score.