Multimodal Image Generation: From Random Sampling to Controllable Output

    This article traces how multimodal image generation has moved beyond random sampling toward controllable image generation. It explains why early text-to-image models failed on prompts that combine multiple entities, attributes, and spatial relations, and how a two-stage architecture of semantic planning plus diffusion rendering addresses those failures through structured intermediate representations. It then covers engineering complex instruction following, rendering legible in-image text, turning generated visuals into polished and editable decks, and designing the supporting cloud infrastructure. It closes with a practical checklist for evaluating image generation solutions and an honest look at the remaining limitations.

    1. The Starting Point: Being Able to Draw Is Not the Same as Drawing It Correctly

    Over the past two years, nearly every team that has integrated text-to-image generation into a business system has hit the same kind of problem. Give an early model a prompt with multiple entities, attributes, and relative positions, such as “A girl in a red raincoat sits to the left of a blue bench, and a dog that doesn’t eat apples lies on the right”, and the result is often wrong at the semantic level. Colors get mixed up between objects, an apple ends up in the dog’s mouth, and the textures of the bench and the raincoat blend into each other.

    The problem is not that the image fails to look beautiful. It comes from three structural capability gaps.

    First, models lack attribute binding. The model compresses the entire sentence into a single global conditioning vector. As a result, “red” no longer belongs to the “raincoat”. Instead, it bleeds evenly across the whole image, which is almost unavoidable when the prompt contains multiple entities.

    Second, spatial and negative relations are not modeled. Early diffusion models respond weakly to relations such as “on the left” and “on the right”, and to “contained in” versus “containing”, and even more weakly to negation. In embedding space, “doesn’t eat apples” and “eats apples” are highly similar to each other, so the conditioning vector can barely tell them apart.

    Third, in-image text is treated as texture. To people, text is a set of symbols. But in a framework that models pure pixels, it is no different from a pattern printed on clothing. The result is posters covered in garbled characters that vaguely resemble ancient Greek inscriptions. Someone then has to import the image into a general-purpose image editing tool and type all the text again by hand.

    In short, a traditional diffusion model is like an extremely talented painter who is also extremely drunk. The brushwork is impressive, but the painter never really understands the logic of the scene.

    2. The Core Paradigm: Think First, Then Draw

    The past two years of technical progress have produced an answer that has been validated again and again: separate “thinking” from “drawing”.

    2.1 Two-Stage Architecture: A Semantic Planner and a Pixel Renderer

    The mainstream two-stage architecture can be abstracted as follows:

    • Brain (an autoregressive semantic planner): built on a generative language model backbone with billions of parameters. It understands long instructions, breaks them down into entities and constraints, orders the composition, and outputs a structured intermediate representation.
    • Hands (a diffusion renderer): generates high-fidelity pixels under the given layout and attributes. It focuses on materials, lighting, perspective, and fine textures.

    The key change lies in the intermediate representation. The model no longer maps a sentence directly to an image. Instead, it first produces an inspectable “semantic blueprint”: it sets the canvas size, assigns a region and attribute labels to each entity, specifies the relative relations between entities, and rewrites negative constraints as explicit negative conditions. Only after planning is complete does the renderer fill in the pixels.

    2.2 What a Semantic Blueprint Looks Like

    A typical intermediate representation looks roughly like this (illustrative structure):

    1. {
    2. "canvas": { "width": 1024, "height": 1024 },
    3. "regions": [
    4. {
    5. "id": "girl",
    6. "bbox": [40, 300, 420, 900],
    7. "attributes": { "clothing": "red raincoat", "pose": "sitting" },
    8. "relations": { "relative_to": "bench", "direction": "left" }
    9. },
    10. {
    11. "id": "dog",
    12. "bbox": [600, 480, 900, 880],
    13. "attributes": { "pose": "lying down" },
    14. "relations": { "relative_to": "bench", "direction": "right" }
    15. },
    16. {
    17. "id": "bench",
    18. "bbox": [300, 620, 760, 780],
    19. "attributes": { "color": "blue" }
    20. }
    21. ],
    22. "negative": ["apple in the dog's mouth", "extra dogs"],
    23. "text_blocks": []
    24. }

    This blueprint brings three direct engineering benefits. It binds each attribute one-to-one to a region, which solves color bleeding. It makes negative conditions explicit, which solves the failure with negated statements. And its output can be validated programmatically, which solves the problem of quality being judged by the human eye alone.

    2.3 Why the Planning Stage Suits an Autoregressive Backbone

    Planning is essentially a reasoning task with sequential dependencies. You first decide the canvas and the main composition, then place the subjects, and finally add the background and lighting. Autoregressive modeling naturally unfolds token by token. It can write intermediate conclusions such as “the red attribute is locked to the raincoat” explicitly, so that later steps can refer to them. In contrast, a single encoding pass compresses a long instruction into a lossy fixed-length vector. This is the fundamental reason planning-style architectures are more stable with long instructions and multiple constraints.

    3. Engineering Complex Instruction Following

    To put this paradigm into production, you also need to handle several specific language phenomena.

    • Counting and uniqueness: quantity constraints must be written explicitly into the plan and checked by counting entities after rendering. If the count is off, you resample or fall back to human review.
    • Relative positions: relations such as left/right, front/back, and above/below are best converted into normalized coordinate ranges. Leaving them in natural language forces the renderer to guess.
    • Negation: negation should not just be stacked as negative prompts. It should become a structured constraint such as “this region must not contain a given object”.
    • Conflicting global style and local attributes: separate “global style conditions” from “region attribute conditions”, and define a priority order between them. Otherwise a style description can override a specific color.

    Evaluation needs the same engineering rigor. A workable combination includes:

    • Rule-based checks: whether attributes fall in the correct regions, and whether negative objects are avoided.
    • OCR read-back: whether the in-image text is legible and its content is correct.
    • Structured comparison: differences between the blueprint and the set of entities actually generated.
    • Small-sample human preference scoring: used to judge trends.

    Relying only on a subjective “it looks better” to sign off on quality will eventually get out of control.

    4. In-Image Text and Layout: From Gibberish to Readable

    Text rendering is hard because the model has to master both glyph structure and layout semantics at the same time.

    An effective approach is to treat text as a separate modality. A dedicated text encoder and glyph-level conditioning are introduced for text regions, so the model knows which stroke components make up a Chinese character and how the skeleton of a Latin letter relates to its serifs. During conditioning injection, text regions are also decoupled from image regions, so that text descriptions do not pollute background generation.

    You should be clear-eyed about the limits. Very small font sizes, dense long-form text, and complex mixed Chinese-English layouts remain difficult. For commercial delivery, a hybrid strategy should take priority: keep only a short headline inside the image, and typeset the body copy in editable text layers.

    5. From a Single Image to a Finished Product: Layout Design and Editable Export

    The real leap toward a product is moving from “generating an image” to “generating a deliverable document”. This rests on three capabilities.

    The first is layout design. It includes the grid and whitespace system, visual hierarchy (a graduated scale of font size and weight for titles, body text, and annotations), color consistency (restrained use of primary, secondary, and accent colors), and image-to-text density control. What people call “better-looking output” essentially means these layout rules have been parameterized and built into the generation pipeline. It does not simply mean a higher-resolution model.

    The second is automatic composition from content to a slide deck. First, the topic is expanded into a structured outline. Next, a layout template is selected for each page. Finally, titles, key points, images, and charts are filled in.

    The third is editable export. A generated presentation must be exported as native editable text and vector elements, not pasted onto each page as a bitmap. Otherwise, users cannot change a single word in the file they receive, and the delivery loses much of its value.

    1. def compose_deck(spec, render_fn, exporter):
    2. plan = planner(spec) # 1) Structured plan: page count, layouts, image/text budget
    3. assets = {}
    4. for slot in plan.image_slots:
    5. assets[slot.id] = render_fn(
    6. prompt=slot.prompt, negative=slot.negative,
    7. size=slot.size, seed=slot.seed,
    8. ) # 2) Render visual assets by slot
    9. deck = exporter.new_deck(size=plan.page_size, theme=plan.theme)
    10. for page in plan.pages:
    11. slide = deck.add_slide(page.layout)
    12. for block in page.blocks:
    13. if block.kind == "image":
    14. slide.add_picture(assets[block.ref], block.rect)
    15. else:
    16. slide.add_text(block.content, block.rect, style=block.style)
    17. for note in page.speaker_notes:
    18. slide.set_notes(note)
    19. return deck.save(plan.out_path) # 3) Text is written to native text objects and stays editable

    Accessibility is worth noting as well. Legible in-image text, sufficient contrast between foreground and background, and alternative text descriptions for decorative elements are not just improvements to the user experience. They are also basic requirements for accessibility compliance.

    6. Production Architecture: A Cloud Engineering Checklist

    Turning these capabilities into a stable service calls for a layered approach on the infrastructure side.

    • Inference tiers: semantic planning runs on the text inference pipeline, and rendering runs on the image inference pipeline. The two scale independently. Planning tasks are short and bursty, while rendering tasks are long-running and GPU-intensive. Mixing them only slows both down.
    • Caching and cost reduction: cache results by the semantic fingerprint of the normalized prompt. Set resolution tiers for each business scenario, using low resolution for previews and high resolution for final output. For non-sensitive scenarios, fix the random seed to ensure reproducibility.
    • Storage and delivery: store assets in object storage and use a content delivery network (CDN) to reduce time to first image. Use a message queue to smooth out traffic spikes in rendering tasks and handle asynchronous callbacks.
    • Observability: connect logging and monitoring/alerting services, and track time to first byte, end-to-end latency, resample rate, and human rework rate as core dashboard metrics. A sudden rise in the resample rate usually points to a problem in the planning stage, not to render quality.
    • Content safety: review prompts for compliance on the input side, and review both images and in-image text on the output side. Tag generated content so that it can be traced.

    As a reference architecture, Baidu AI Cloud offers building blocks for each of these layers. Its container services can run inference services with elastic scheduling, object storage can hold assets and outputs, a message queue can handle asynchronous rendering tasks, and its logging and monitoring/alerting services provide end-to-end observability. These are generic infrastructure capabilities, not features tied to any specific model.

    7. Selection and Acceptance: An Actionable Checklist

    When evaluating a new image generation solution, use these questions instead of subjective impressions:

    1. Can attribute accuracy be measured in multi-entity scenarios, and does the solution support negative constraints?
    2. How does performance degrade with long instructions (more than 50 characters and more than five constraints)?
    3. How readable and accurate is the in-image text, and how does the model handle mixed Chinese and English?
    4. Does it support region-level conditioning and targeted repainting of local areas?
    5. Does it output structured intermediate results that make programmatic validation and further editing easy?
    6. Can generated output be exported in an editable format, or only as a bitmap?
    7. How do per-image generation cost and resample rate affect unit economics for the business?

    8. Limitations and Boundaries

    Even with a two-stage architecture that combines planning and rendering, several issues remain:

    • Key constraints still decay as instructions get longer.
    • Dense long-form text and stylized fonts are still weak spots.
    • Training-data bias and copyright risks need to be assessed separately by the business.
    • Compute cost makes it difficult to cover every real-time scenario.

    For this reason, the most robust approach in practice is a three-stage pipeline of model generation, rule-based constraints, and human fallback. It is better than betting all your trust on a single model call.

    The essence of moving from random sampling to controllable image generation is taking a black-box generation process apart and rebuilding it as a pipeline that is plannable, verifiable, and editable. Understanding this underlying thread is more valuable than memorizing any particular version number.