Controllable Image Generation: From Random Prompting to Precise Control

    Diffusion models produce striking visuals but struggle with structured reasoning, which causes errors in spatial relationships, negation, counting, attribute binding, and in-image text. This article traces the root of these failures and explains the shift to a two-stage "plan-then-render" architecture that pairs a reasoning planner with a diffusion renderer. It also covers practical techniques for rendering text as symbols rather than textures, outlines the capability boundaries and engineering considerations, and provides a reproducible evaluation approach along with the open challenges ahead. Readers will gain a clear view of how controllable image generation moves from random luck to reliable, structured output.

    1. The Root of the Problem: Models “Recognize Words” but Don’t “Understand Scenes”

    In recent years, almost everyone who has used AI to generate images has experienced the frustration of a “random pull”: you enter a complex description containing positions, attributes, and negation, yet the model puts “left” on the right, turns “a dog that doesn’t eat apples” into a dog gnawing on an apple, or assigns a red attribute to the wrong object. These failures are not the result of a poorly written prompt; they stem from a structural flaw in the image generation paradigm itself. This article starts from the root cause, systematically reviews the evolution of controllable image generation, and offers judgment criteria from an engineering perspective.

    Early mainstream text-to-image approaches center on the diffusion model. You can think of it as a gifted but temperamental painter: given a set of keywords, it can quickly bloom stunning colors and textures onto the canvas, yet it often fails to guarantee semantic rigor.

    Its core weakness is that the denoising process of a diffusion model is essentially an iterative optimization of the global pixel distribution of the entire image. It is naturally good at texture, lighting, and style, but weak at structured reasoning. This shows up as three common failure modes:

    • Spatial relationship errors: The model understands nouns and attribute words like “red,” “girl,” and “dog,” but does not understand relative positional constraints such as “left,” “right,” and “above.”
    • Negation and counting failures: For logical and quantitative constraints like “doesn’t eat apples” or “two, not three,” the model often treats them as positive semantics, resulting in the opposite output.
    • Attribute binding errors: In multi-subject scenes, attributes such as color and material are easily bound to the wrong adjacent object, a phenomenon known as “attribute leakage.”

    In-image text rendering is even trickier. Under the traditional diffusion paradigm, text is treated as a kind of texture, no different in essence from clothing patterns or brick-wall designs. So when users ask for explicit text like “SALE” or a grand-opening sign, the model tends to output gibberish that merely resembles the characters, and the text still has to be added by hand in an image editor.

    In short, the old paradigm stacks pixels rather than building a scene.

    2. A Paradigm Shift: Plan First, Then Render

    To solve these problems at the root, the key idea is to make the model “think it through” before it starts drawing. The effective path the industry has explored is to split generation into two stages handled by two models with different specialties:

    • Planner: Usually a multimodal or autoregressive language model with strong reasoning ability, responsible for parsing a natural-language instruction into a structured scene blueprint.
    • Renderer: Usually a diffusion model, responsible for painting lighting, texture, and detail in pixel space according to the blueprint.

    The point of this combination is that the two capabilities complement each other: autoregressive models excel at token-by-token, symbol-level reasoning and maintaining logical constraints, while diffusion models excel at high-fidelity visual synthesis. When an instruction is complex, the planner does not rush to produce an image; instead, it first sorts out the logic of the scene like an outline—“compose the layout: place the girl in red on the left and lock in the red attribute; place the dog on the right and lock in the ‘not eating apples’ state; the background is a blue bench”—and then hands this logic to the renderer to fill in the pixels.

    The following is a high-level illustration of this flow (pseudocode that only conveys the structured idea, not any specific product API):

    1. # High-level illustration: a plan-based image generation pipeline
    2. instruction = "A girl in a red raincoat sits to the left of a blue bench; to the right is a dog that does not eat apples"
    3. # Step 1: The planner parses the natural-language instruction into a structured blueprint
    4. plan = planner.parse(instruction)
    5. # plan can be roughly structured as:
    6. # {
    7. # "entities": [
    8. # {"id": "girl", "attrs": ["red raincoat"], "position": "left"},
    9. # {"id": "dog", "attrs": ["not_eating", "apple"], "position": "right"},
    10. # {"id": "bench", "attrs": ["blue"], "role": "background"}
    11. # ],
    12. # "relations": ["girl left_of bench", "dog right_of bench"]
    13. # }
    14. # Step 2: The renderer synthesizes pixels based on the structured blueprint
    15. image = renderer.generate(plan, seed=fixed_seed)

    This “plan-then-render” hybrid architecture turns image generation, originally completed in one shot, into a two-stage pipeline of “symbolic reasoning plus visual synthesis.” For logical relationships such as position, above/below, containment, and negation, the planning stage can express explicit constraints symbolically, achieving something close to a “dimension-reducing strike”—it no longer relies on probability and luck, but guarantees semantic correctness through structured reasoning.

    3. Tackling In-Image Text: From “Texture” to “Symbol”

    The breakthrough in text rendering comes from a shift in mindset: making the model treat text as a symbolic system rather than a visual texture.

    In such approaches, a character-level or subword-level encoding module with strong text understanding is typically introduced. It first encodes the text to be rendered into semantic and structural representations, then aligns them with the image generation process. The key is to let the model “recognize” the composition of characters—it knows that the letter H is made of two vertical strokes and one horizontal stroke, and that the Chinese character “中” consists of specific strokes and an enclosing structure—so that when drawing, it follows the glyph structure instead of casually producing look-alike gibberish.

    From an engineering perspective, several practical techniques improve text controllability:

    • Explicit text-region placement: The planning stage reserves clear layout regions and hierarchy for text, preventing text and subjects from interfering with each other.
    • Character-level supervision: Use glyph structure as a constraint signal to reinforce stroke-level correctness, rather than optimizing only for overall visual similarity.
    • Multilingual adaptation: Model the glyph differences of different writing systems separately, so that Chinese, Latin letters, and Arabic numerals do not “bleed into” one another on the same layout.

    The result is that scenarios with “text in the image”—posters, covers, infographics—can for the first time directly generate usable finished images, greatly reducing later rework.

    4. Capability Boundaries and Engineering Considerations

    To view this technology rationally, we still need to be clear about its capability boundaries and engineering considerations:

    1. The planner is the ceiling: The upper bound of the hybrid architecture depends on the reasoning quality of the planner. If an instruction is ambiguous or self-contradictory (for example, “on the left and also on the right”), no matter how strong the renderer is, it cannot fix it, so the structured expression of the prompt still matters.
    2. Balancing controllability and randomness: By fixing the random seed and adjusting sampling steps and guidance strength, you can trade off between “stable and controllable” and “rich visuals.”
    3. Cost and latency assessment: Compared with a single-stage approach, a two-stage pipeline usually brings extra inference overhead; production environments need to optimize throughput and cost through means such as caching planning results, batch scheduling, and asynchronous generation.
    4. Systematic integration: In real business, image generation is rarely an isolated capability. It often relies on object storage to host assets, message queues to smooth out traffic peaks, container platforms for elastic scaling, and logging and monitoring alerts to ensure stability. Only by combining these general cloud capabilities can you turn a demo into a service that can be operated sustainably.

    5. Evaluation: How to Judge Whether a Solution Is Truly “Controllable”

    Faced with a flood of marketing claims, developers can use a reproducible evaluation approach to make objective judgments:

    • Instruction following: Construct a set of combination instructions with multiple subjects, positions, and attributes, and measure the accuracy of attribute binding and spatial relationships.
    • Negation and counting tests: Specifically cover constraints such as “no,” “none,” and “two/three,” and verify whether logical handling is reliable.
    • Text rendering accuracy: Use decidable short text (letters, numbers, common words) to assess glyph correctness and readability.
    • Consistency tests: Fix the semantics and vary the random seed to observe whether the results are stable and controllable.
    • End-to-end latency and cost: Measure average and tail latency under the target concurrency and assess the cost per unit of output.

    Comparing different solutions horizontally with the same test set reflects the real level far better than any single example.

    6. Challenges and Future Directions

    Although the hybrid architecture is remarkably effective at semantic control, it still faces several challenges: how to align the interface between planning and rendering more efficiently, the planning stability of complex long instructions, the robustness of text rendering at small font sizes and with artistic fonts, and the room for reducing overall inference cost. It is foreseeable that as the planner’s reasoning ability grows, the renderer’s fidelity improves, and their collaboration mechanism is continuously optimized, “say it clearly and it draws correctly” will gradually shift from a selling point to a baseline capability.

    For developers, understanding the value of this paradigm lies not only in knowing how to use a particular tool, but more in grasping its underlying logic: push as much uncertainty as possible forward to the reasoning-based symbolic stage, and leave visual expressiveness to a dedicated rendering module. This design philosophy of “layered decoupling, each doing what it does best” is precisely what moves controllable multimodal generation from “random pull” to “precise output.”