Controllable Multimodal Generation: A Technical Breakdown of Moving from Trial-and-Error to Precision

    This article breaks down how multimodal generation is evolving from one-shot, unpredictable output toward controllable, plan-first pipelines. It explains why single diffusion models struggle with attribute binding, spatial relations, negation, and in-image text; how hybrid "plan-then-render" architectures decouple semantic planning from pixel generation; techniques for rendering text as structured symbols rather than textures; and engineering practices such as prompt rewriting, layout control, and multi-candidate selection. It also covers how generation is expanding from single images to structured, editable documents, and closes with practical guidance on deployment, cost planning, safety and compliance, and a measurable evaluation checklist for building production-grade controllable generation pipelines.

    1. Where the problem starts: the more detail you describe, the worse the result

    Many developers who have used text-to-image tools have run into an awkward situation. You write a prompt like “a girl in a red raincoat sits to the left of a blue bench, and to the right is a dog that is not eating an apple,” and the model generates a red dog, a blue raincoat, or the dog happily munching on an apple.

    The problems fall into three areas:

    • Attribute misbinding: When multiple objects each carry attributes, color and material tend to “shift” between objects. The model can’t reliably bind an attribute to the correct entity.
    • Spatial relation failure: Relative positions such as left/right, up/down, and front/back depend on a reference frame. If the reference object is missing or misplaced, the entire relation chain collapses.
    • Negation semantics failure: “Not eating an apple” is a logical constraint. The model can’t translate it into pixels, and the easiest thing to do is often to draw the negated action directly.

    Text in images behaves the same way. Models often render a pattern that “looks like text” but turns out to be gibberish on closer inspection, with dropped or extra strokes. The root cause is that the model treats text as ordinary texture to be interpolated, rather than as a symbol with definite structure.

    Mainstream approaches are shifting from “one-step direct output” to “plan the semantics first, then generate pixels,” and the industry is using compositional generation benchmarks to test exactly these problems: for a fixed number of objects, checking item by item whether attributes, counts, and spatial relations are all correct.

    2. Architecture evolution: from “one-step output” to “think first, then draw”

    2.1 The bottleneck of a single diffusion model

    Take the latent diffusion model, which is widely used in text-to-image generation. A typical pipeline works like this:

    1. Encode the prompt into a sequence of embeddings;
    2. Inject text information into the image denoising process via cross-attention;
    3. Iteratively denoise to recover pixels.

    This pipeline excels at image texture but falls short on semantic precision:

    • Text conditioning lacks structured organization, so semantically similar words interfere with each other.
    • Attention has no notion of an “entity,” so attributes drift between objects.
    • Generation is one-shot; when it goes wrong, there’s no chance to go back and fix it.

    2.2 Hybrid architecture: separating the planner from the renderer

    A recent upgrade to multimodal models introduced a two-stage “plan-then-generate” paradigm:

    • Planning stage: an autoregressive multimodal large model with parameters in the tens-of-billions range or more handles understanding the instruction, decomposing objects, binding attributes, and arranging spatial relations;
    • Generation stage: a diffusion or flow-matching model renders the image according to the plan.

    The idea isn’t entirely new. Promptist, a chained-prompt diffusion approach proposed in 2022, already handed prompt rewriting to a language model and then generated images with a diffusion model. The difference is that the new generation of models folds both into the same model system:

    • Planning and generation can be trained jointly end-to-end;
    • The intermediate semantic representation is more structured;
    • The gap between the two stages is markedly narrowed.

    The intermediate representation is mainly implemented in three ways:

    1. Image token sequences: Discretize the image into tokens and predict them autoregressively, segment by segment. In practice, a temporary end-of-sequence token is often inserted mid-sequence to close out details that are no longer in effect early, concentrating compute on the main subject.
    2. Structured layout descriptions: Output a scene description containing objects, attributes, bounding boxes, and relative relations, then hand it to the generation model to render according to the layout.
    3. Interleaved text-image representation: Organize text descriptions and image segments alternately within a unified sequence, naturally supporting multi-turn editing.

    2.3 Sample code

    1. from dataclasses import dataclass
    2. @dataclass
    3. class ObjectSpec:
    4. name: str
    5. attributes: dict
    6. bbox: tuple # normalized coordinates (x1, y1, x2, y2)
    7. relations: list # e.g. [{"type": "left_of", "target": "bench"}]
    8. @dataclass
    9. class SceneSpec:
    10. canvas: tuple # (width, height)
    11. background: str
    12. objects: list
    13. text_overlay: list # text that needs precise rendering
    14. def parse_prompt(prompt: str) -> SceneSpec:
    15. """Planning stage: understand the instruction, output a structured scene description"""
    16. return vision_llm.parse(prompt) # illustrative interface, not a real call
    17. def render(scene: SceneSpec, seed: int = 0, steps: int = 28) -> bytes:
    18. """Generation stage: perform denoising sampling based on the scene description"""
    19. latents = diffusion_pipeline(
    20. prompt=scene.to_text(),
    21. layout=scene.objects,
    22. text_regions=scene.text_overlay,
    23. seed=seed, steps=steps,
    24. )
    25. return decoder.decode(latents)
    26. def generate(prompt: str, retries: int = 2) -> bytes:
    27. for attempt in range(retries + 1):
    28. image = render(parse_prompt(prompt), seed=attempt * 7919)
    29. report = quality_gate(image) # combined validation of attributes, relations, text
    30. if report.passed:
    31. return image
    32. raise RuntimeError(f"Quality gate still not passed after multiple samples: {report.summary}")

    2.4 Why it works

    • Global planning: Attributes are bound to objects during planning and stay consistent throughout generation, significantly reducing the misbinding rate.
    • Explicit relation modeling: Spatial relations are explicitly expressed as “who is to the left of whom,” rather than relying on incidental alignment from attention.
    • Controllable intermediate artifacts: The structured description can be manually inspected, modified, cached, and audited, laying the groundwork for local inpainting and multi-turn editing.
    • Improved prompt robustness: Long sentences and nested descriptions are decomposed uniformly during planning, reducing the impact of ordering and wording.
    • Chinese and local elements: Multimodal models are typically trained jointly on massive bilingual Chinese-English image-text data, so they understand Chinese characters, traditional festivals, and local contexts more directly, without repeatedly translating or rewriting prompts.

    2.5 Costs and optimization

    • Latency: The two stages run serially, and long-text understanding in the planning stage raises overall latency, so streaming or asynchronous interfaces are a better fit.
    • Compute overhead: The generation stage is still the bulk of the compute; you can use few-step sampling with consistency distillation, model quantization, operator fusion, and continuous batching.
    • Semantic caching: Hash normalized prompts and return cached results directly for identical requests.
    • Quality tiering: Route simple tasks to lightweight models and trigger multi-candidate sampling only for complex tasks, keeping unit cost under control.

    3. Text rendering: treat “text” as a symbol, not a pattern

    The essence of the gibberish problem is that the model lacks a “symbol prior” for text. The industry has addressed this systematically on three levels.

    Model level: glyph condition injection. GlyphDraw, an early representative proposed by BAAI (Beijing Academy of Artificial Intelligence), is a well-known example, and several teams followed suit. These methods convert the characters to be rendered into an image-form feature and inject it as an additional condition into the denoising network:

    • Latin letters’ strokes and glyph differences rely more on fine structure, so a pixel-level glyph map is usually used;
    • Chinese characters have many strokes and complex structures, so abstract representations such as skeletons, contours, and heatmaps are more suitable.

    Data level: character-level annotation. Training data annotates each text region with its content, font, size, and position, teaching the model to “render the right character in the right place.”

    Engineering level: a closed loop of generation and recognition. After generation, optical character recognition is run on the text region and compared with the expected text; on a mismatch, the text condition is adjusted and the region is redrawn.

    The sample code is shown below:

    1. def refine_text_region(image, box, target_text, font_spec, max_rounds=2):
    2. for _ in range(max_rounds):
    3. glyph = glyph_encoder(target_text, font_spec) # map characters to glyph features
    4. image = inpaint(image, mask=box, glyph_condition=glyph)
    5. recognized = ocr(image.crop(box))
    6. if normalize(recognized) == normalize(target_text):
    7. break
    8. return image

    In practice, for posters, covers, and other scenarios with high text-precision requirements, a safer approach is “the image handles the background, and text is overlaid by a typesetting engine”:

    • Leave clean whitespace during generation;
    • Precisely control font, size, and letter spacing through typesetting parameters.

    This avoids stroke errors at the root and also makes it easy to swap in multiple languages and brand fonts.

    4. Controllability engineering: putting randomness in a cage

    Upgrading model capability is only the first step. To make output stable and usable, you also need engineering support on the side.

    Prompt rewriting. Users often speak colloquially and omit subjects. Before calling the generation model, you can first use a large model to normalize the input into a structured description, fill in implicit objects, unify attribute wording, and convert negative constraints into positive descriptions:

    1. Original input: a girl sitting to the left of a bench, a dog to the right that isn't eating an apple
    2. Rewritten:
    3. {
    4. "canvas": "16:9",
    5. "objects": [
    6. {"name": "girl", "attributes": {"coat": "red raincoat"},
    7. "relation": {"type": "left_of", "target": "bench"}},
    8. {"name": "bench", "attributes": {"color": "blue"}},
    9. {"name": "dog", "attributes": {"color": "reddish-brown"},
    10. "relation": {"type": "right_of", "target": "bench"}}
    11. ],
    12. "constraints": ["apple placed on the ground, away from the dog"],
    13. "text_overlay": []
    14. }

    Layout control. Models that support control signals such as region conditioning, keyframes, and edge maps let you precisely control subject position and compositional whitespace. For human-pose scenes, you can first use a pose-estimation algorithm to extract the skeleton as a condition.

    Sampling strategy. A fixed random seed, combined with adjusting prompt-guidance strength, can stably reproduce an image or tune style intensity. Multi-candidate sampling plus automatic filtering has a noticeably higher success rate than a single generation.

    Failure fallback. Generation models can still err on counts and exact text, so mature products often mix several techniques:

    • Overlay exact text with a typesetting engine;
    • Use a vector asset library for logos;
    • Draw charts with a visualization library;
    • Composite the final output.

    5. From one image to a document: generation’s boundaries keep expanding

    The multimodal upgrade goes beyond image quality and semantic precision and extends into structured output. On some model service platforms, you can already generate presentations and export them to editable presentation file formats, showing that generative models are becoming the front end of document production.

    Technically, this capability breaks down into three layers:

    1. Content planning: The model generates an outline, key points, and copy;
    2. Layout arrangement: Structured data is applied to master layouts and templates from a template library;
    3. Formatted rendering: A document rendering engine produces editable files.
    1. def build_deck(topic: str, page_count: int, template_id: str):
    2. outline = content_llm.plan(topic=topic, pages=page_count) # planning
    3. deck = document_engine.new_deck(template=template_id) # layout
    4. for page in outline.pages:
    5. slide = deck.add_slide(layout=page.layout)
    6. slide.set_title(page.title)
    7. slide.set_body(page.bullets)
    8. if page.chart_spec:
    9. slide.add_chart(page.chart_spec)
    10. if page.image_prompt:
    11. slide.add_image(render(page.image_prompt, seed=page.seed))
    12. return deck.export(target_format="editable") # export

    Compared with generating images directly, structured generation has clear advantages:

    • Text is real and editable, with no gibberish;
    • Objects are structured elements, easy to modify later;
    • Enterprise templates can be reused for consistent brand visuals;
    • The generation process is inherently traceable, aiding review and retrospection.

    Engineering implementation still needs strengthening:

    • Content authenticity: The model may fabricate data, so key numbers and facts need retrieval-based verification;
    • Template fit: Auto-generated title lengths and point counts may exceed placeholder capacity, requiring overflow detection and font-size adaptation;
    • Asset licensing: Images and fonts must use licensed assets;
    • Audit trail: Record prompts, model versions, and generation time to meet compliance requirements.

    6. Engineering implementation: integration, deployment, and evaluation

    Interface form. These models are typically provided as cloud services. A typical integration flow:

    1. Create a service in the console and obtain call credentials;
    2. Send requests per the API specification, passing parameters such as prompt, resolution, and random seed;
    3. Receive image data or a generation task identifier, and poll for the task result;
    4. Store generated artifacts in object storage and accelerate delivery via a content delivery network.
    1. import base64, hashlib, json, time, urllib.request
    2. def call_text_to_image(endpoint, token, prompt,
    3. width=1024, height=1024, seed=0):
    4. """Illustrative code: both synchronous and asynchronous task forms"""
    5. key = hashlib.sha256(
    6. f"{prompt}|{width}x{height}|{seed}".encode()
    7. ).hexdigest() # semantic cache key
    8. # cached = cache.get(key); if cached: return cached
    9. payload = {"prompt": prompt, "width": width,
    10. "height": height, "seed": seed}
    11. req = urllib.request.Request(
    12. endpoint, data=json.dumps(payload).encode(),
    13. headers={"Content-Type": "application/json",
    14. "Authorization": f"Bearer {token}"})
    15. for _ in range(30):
    16. with urllib.request.urlopen(req, timeout=60) as resp:
    17. result = json.loads(resp.read())
    18. if result.get("status") == "SUCCEEDED":
    19. return base64.b64decode(result["image_base64"])
    20. if result.get("status") == "FAILED":
    21. raise RuntimeError(result.get("error"))
    22. time.sleep(2) # asynchronous task: poll or callback
    23. raise TimeoutError("Generation task timed out")

    Deployment form. Inference services can adopt pay-per-call or resource-point deduction models and scale elastically with demand, avoiding reserving resources for peak load over the long term. For scenarios with special latency or data-compliance requirements, private deployment is an option.

    Cost and capacity planning. Key dimensions to focus on:

    • Estimate per-generation cost by resolution and step count;
    • Generation tasks are naturally suited to queue-based peak shaving and asynchronous callbacks;
    • Use a logging service and a monitoring/alerting system to continuously observe call volume, success rate, latency, and failure reasons.

    Security and compliance. Image generation carries risks at both the input and output ends:

    • Input side: prompt moderation to intercept violating content;
    • Output side: image content-safety detection to avoid generating sensitive content;
    • Compliance labeling: embed implicit labels in file metadata or the interface, and add explicit notices where appropriate, per applicable AI-generated content labeling requirements;
    • Data side: clarify where user-generated artifacts are stored and their access permissions, and set a reasonable content retention period.

    Evaluation checklist. To accept a generation model, it’s recommended to build a quantifiable evaluation set across the following dimensions:

    Dimension Metric Evaluation method
    Semantic adherence Accuracy of attribute binding, counting, and spatial relations Compositional generation benchmark
    Text rendering Text accuracy and readability Recognition comparison, manual spot checks
    Image quality Aesthetic score and structural plausibility Automated scoring, blind human evaluation
    Stability Consistency rate across repeated generations of the same prompt Fixed-seed repeated sampling
    Engineering performance First-token latency, P95 latency, success rate Load testing, production monitoring
    Cost Cost per valid image Cost accounting

    7. Conclusion

    From trial-and-error to control, the upgrade can be summarized in four points:

    1. Architecturally, the hybrid “plan-then-generate” paradigm decouples semantic understanding from pixel generation;
    2. For text rendering, characters are treated as structured symbols rather than textures;
    3. In engineering, prompt rewriting, layout constraints, and multi-candidate filtering provide fallbacks;
    4. In product form, output expands from a single image to structured, editable artifacts such as presentations.

    For developers, rather than expecting a new model to solve everything once and for all, it’s better to build a “controllable generation pipeline” around your business scenario: combining the model’s generative capability with rule-based constraints, validation, and review is what truly puts randomness in a cage and lets generation results reliably enter production.