Controllable Text-to-Image: Think First, Then Draw — Model Selection and Implementation

    Text-to-image generation often feels like a lottery: the images look good, but object positions, attribute bindings, and negation logic frequently go wrong. This article explains why pure diffusion models struggle with these constraints and how a hybrid, plan-first architecture—combining LLM semantic planning with diffusion rendering—improves controllability. It also covers glyph-aware text rendering for readable in-image text, compares pure diffusion, hybrid architecture, and glyph-aware approaches, and walks through the engineering pipeline for production. Finally, it provides reusable test prompts and a decision checklist for evaluating text-to-image models, along with the risks and boundaries of hybrid architectures.

    Why Text-to-Image Still Feels Like a Lottery: The Shift Toward Controllable Generation

    Over the past two years, many users have experienced the “gacha” feeling of text-to-image generation: you enter a complex description, and the model produces an image that may have beautiful colors and decent texture, but the object positions, color bindings, and quantity relationships are often wrong. For example, if you ask for “a girl in a red raincoat sitting on the left side of a blue bench, with a dog that won’t eat an apple on the right,” the model might turn the dog red, color the bench like the raincoat, or leave an apple still in the mouth of the dog that “won’t eat an apple.”

    The core of the problem is not just image quality, but whether semantic understanding is accurate. When the task is simply to generate an atmosphere image, concept art, or background material, an occasional “misunderstanding” by the model may be acceptable. But once you enter production scenarios like posters, e-commerce assets, cover designs, presentation illustrations, and long infographic images, users begin to demand: objects must appear, attributes must be correct, positions must match the description, text must be clear and readable, and results must be reproducible and stable.

    As a result, text-to-image technology is shifting from “gacha-style generation” to “controllable generation.” The goal of controllable generation is not merely to improve aesthetics, but to enable the model to understand complex instructions, decompose semantic structure, and remain constrainable, verifiable, and debuggable throughout the generation process.

    Why Pure Diffusion Models Struggle With Attribute Binding and Negation Logic

    To understand the technical evolution of text-to-image, you first need to understand the limitations of pure diffusion models. Diffusion models synthesize pixels through repeated denoising and are highly mature in texture, color, lighting, and stylization. But their essence leans more toward “recovering a visual result from noise” rather than “first building a complete scene logic, then painting it layer by layer.”

    As a result, pure diffusion models excel at generating images with overall aesthetic appeal, but tend to be unstable on three types of constraints:

    1. Spatial relationships: Relative positions like left/right, up/down, front/back, contains/contained-by, and parallel/adjacent require the model to not only identify objects but also understand the geometric relationships between them.
    2. Logical constraints: Negation, quantity, qualification, and exclusion conditions—such as “won’t eat an apple,” “no umbrella,” or “no people should appear”—require the model to reliably parse logical negation in natural language.
    3. Attribute binding: Binding “red” accurately to “raincoat” rather than to “dog,” and “blue” to “bench” rather than to “background.” In multi-object scenarios, attribute mismatch is one of the most common problems.

    In other words, pure diffusion models are better at “organizing good-looking pixels,” but under complex semantics they cannot necessarily build a self-consistent, complete, and verifiable scene. This is also why a single model frequently makes mistakes with complex instructions.

    How Hybrid Architectures Achieve Plan-First, Render-Later

    A more robust technical approach today is to decouple “thinking” and “drawing” into two stages: a language model or autoregressive model first performs semantic planning, then a diffusion model completes the visual rendering. This architecture can be understood as “think first, then draw.”

    The first stage is the brain: the semantic planning module. It typically uses a large or autoregressive model with strong instruction-following ability, responsible for parsing user input, identifying subjects, attributes, relationships, quantities, and negation conditions, and generating a structured scene blueprint, such as a scene graph. This blueprint can include:

    1. {
    2. "objects": [
    3. {"id": "girl", "attrs": {"color": "red", "cloth": "raincoat"}},
    4. {"id": "dog", "attrs": {"negate": "eating_apple"}}
    5. ],
    6. "layout": {"girl": "left", "dog": "right", "shared": "bench"},
    7. "text": ["SALE", "中秋快乐"]
    8. }

    The second stage is the hands: the diffusion rendering module. It no longer generates pixels directly from fully open-ended natural language, but instead completes rendering under the constraints of the scene blueprint. The key question here is not “is there a blueprint?” but “how does the blueprint actually condition the diffusion process?”

    Specifically, the scene graph can be injected into the diffusion model in several ways:

    1. Layout and region conditioning: Convert object bounding boxes, region masks, and relative position relationships into spatial condition maps, and feed them into the diffusion model’s cross-attention, control networks, or region control modules. This gives the model a clearer spatial prior when generating “girl on the left, dog on the right.”
    2. Attribute conditioning and binding: Bind object IDs with attribute embeddings—for example, “girl + red raincoat,” “dog + negate apple”—reducing errors like a red raincoat being misbound to the dog or a blue bench tinting the background.
    3. Negation conditioning: Convert negation conditions like “no apples,” “no people,” or “no umbrellas” into negative prompts, exclusion regions, or exclusion items in the planning stage, reducing the chance of the model inventing things.
    4. Control strength tuning: Use a parameter like control_scale to adjust the strength of the blueprint constraints. If control strength is too low, results are more free but less controllable; if too high, the layout constraints are too strong and may affect the naturalness of the image. In real projects, this needs to be tested against the business scenario.

    A simplified flow is shown below:

    1. def controllable_text_to_image(prompt: str) -> Image:
    2. # Stage 1: Semantic planning
    3. plan = autoregressive_planner(prompt)
    4. scene_graph = plan.to_scene_graph()
    5. # Stage 2: Convert the scene graph into diffusion control conditions
    6. control_conditions = build_control_conditions(
    7. boxes=scene_graph["layout"],
    8. attrs=scene_graph["objects"],
    9. negatives=scene_graph.get("negatives", [])
    10. )
    11. # Stage 3: Diffusion rendering under structural constraints
    12. latent = diffusion_renderer(
    13. scene_graph,
    14. control_conditions=control_conditions,
    15. control_scale=0.6
    16. )
    17. image = vae_decode(latent)
    18. # Stage 4: Precise text-layer compositing
    19. image = compose_text_layer(image, scene_graph["text"])
    20. return image

    The value of this decoupling is not only “more stable results” but, more importantly, that it is debuggable and interventable. If generation fails, developers can determine whether the error lies in the planning stage (a misunderstanding) or the rendering stage (an execution error). The business side can also inspect the scene graph, adjust layout constraints, and modify negation conditions, rather than repeatedly retrying the same prompt.

    From a technical-direction perspective, hybrid architectures typically involve three capabilities: first, LLM scene planning, responsible for instruction parsing and structuring; second, layout/region conditional control, such as methods like ControlNet, GLIGEN, and T2I-Adapter, used to constrain object positions and regional generation; and third, text-layer compositing and glyph-aware encoding, used to improve text readability within the image. Together, these directions support the controllable generation pipeline of “plan first, then render.”

    Why Text Generation Gets Garbled, and How to Improve Glyph Accuracy

    Another long-standing pain point of text-to-image is inaccurate text within the image. Early models often treated text as ordinary texture, no different from clothing patterns or wall designs. So when generating text like “SALE,” “中秋快乐,” or “limited-time offer,” problems such as broken strokes, distorted glyphs, and chaotic Chinese-English mixed layout frequently appear. For posters, covers, and event graphics, these errors directly affect usability.

    The improvement approach is to shift the model from a “texture perspective” to a “symbol perspective.” That is, text is not random pattern but a symbol system composed of characters, strokes, radicals, and structural units arranged according to language rules. By introducing a glyph-aware text encoding mechanism, the model can better understand the stroke structure of Chinese, English, numbers, and common symbols, and generate clearer, more regular text at designated positions.

    This capability typically involves several layers:

    1. Character recognition: Identify which parts of the prompt need to be rendered as text.
    2. Glyph encoding: Convert text into stroke, structural, or symbol feature representations.
    3. Position layout: Decide which region of the image the text appears in, such as the title area, corner-badge area, or bottom slogan area.
    4. Layer compositing: Blend the text layer with the background image to reduce blur, distortion, and inconsistent styling.

    When glyph-aware capability is strong, text-to-image can support poster titles, e-commerce selling points, cover slogans, and holiday copy, reducing the rework of “generate the image first, then manually add text.”

    How to Choose: Pure Diffusion vs. Hybrid Architecture vs. Glyph-Aware Rendering

    These approaches are not mutually exclusive but serve different needs. Here is a simplified comparison:

    Approach Spatial Relationships Attribute Binding Negation Logic Text Accuracy Controllability Better-Suited Scenarios
    Pure diffusion model Weak Fair Weak Weak Medium Atmosphere images, concept art, backgrounds
    Hybrid architecture Strong Strong Strong Depends on text-layer capability High Complex compositions, posters, e-commerce assets
    Glyph-aware text rendering Doesn’t directly address Doesn’t directly address Doesn’t directly address Strong High Text-heavy designs, slogans, title graphics

    In practice, you cannot judge by a single metric. For example, an image focused mainly on brand atmosphere may be lighter with a pure diffusion model; but an event poster that needs multiple objects, explicit positions, attribute binding, and readable text calls more for a hybrid architecture with text-layer capability.

    How to Bring Controllable Text-to-Image Into the Engineering Pipeline

    Model capabilities must ultimately turn into a stable service. An image generation system with “think first, then draw” and “text rendering” capabilities usually involves multiple engineering modules:

    • Object storage: Store user input, reference assets, intermediate control conditions, generated images, and export files.
    • Container platform: Handle elastic scaling for services such as planner inference, diffusion rendering, and text compositing.
    • Message queue: Make long-running generation tasks asynchronous to avoid queuing and timeouts during request peaks.
    • Monitoring, alerting, and logging services: Track generation success rate, average latency, failure reasons, and content moderation results to support continuous optimization.
    • Task orchestration and state management: Record the state of each task from parsing, planning, rendering, to post-processing, making retries and troubleshooting easier.

    In mainstream cloud environments such as Baidu AI Cloud, these capabilities can typically be used as combinations of standardized cloud products. For image generation businesses, the engineering focus should always stay on the image result itself: how assets come in, how the blueprint is generated, how diffusion renders, how text is composited, and how images are validated and exported—rather than blindly expanding into unrelated businesses.

    How to Choose Text-to-Image: Reusable Test Prompts and a Decision Checklist

    If an enterprise is preparing to evaluate a text-to-image model or build a production pipeline, it is advisable not to judge only by sample beautiful images, but to run a set of complex instructions as a benchmark. The following is a directly reusable set of test prompts.

    Spatial Relationship Tests

    1. A red umbrella at the upper left of a wooden table, a blue cup at the lower right.
    2. A white cat sitting on the left of a wooden box, a black cat crouching on the right of the wooden box.
    3. A boy standing on the left of a bicycle, a dog lying on the right of the bicycle.

    Negation Logic Tests

    1. A dog with no apple in its mouth, and no ball nearby.
    2. No people in the background, no umbrella on the ground.
    3. A white cup on the table, with no coffee in the cup.

    Attribute Binding Tests

    1. A girl in a red raincoat sitting on the left of a blue bench, with a dog that won’t eat an apple on the right.
    2. A man in a black suit holding a white cup, a woman in a red dress holding a green bag.
    3. A silver car on the left, a blue bicycle on the right, a red balloon at the top.

    When evaluating, you can establish the following criteria:

    • Whether the object count is correct;
    • Whether object positions match the description;
    • Whether attributes are bound correctly;
    • Whether negation conditions are followed;
    • Whether text is clear and readable;
    • Whether the overall style is usable;
    • Whether results are stable across multiple generations.

    For the selection decision, refer to the following checklist:

    1. If you mainly generate atmosphere images or concept art with low demands on semantic precision, prioritize a lightweight pure diffusion approach.
    2. If there are multi-object, complex spatial relationship, or attribute binding needs, prioritize evaluating a hybrid architecture.
    3. If the image contains a lot of readable text, focus on testing glyph-aware capability rather than judging only by image aesthetics.
    4. If the business requires batch production, evaluate asynchronous tasks, elastic compute, failure retries, and quality sampling mechanisms.
    5. If you need traceability and debuggability, require the system to output intermediate results such as the scene graph, layout conditions, and control parameters.

    Risks and Boundaries of Hybrid Architecture: When Not to Use It

    While hybrid architecture improves controllability, it is not suitable for all scenarios. There are several key trade-offs:

    1. Increased latency: With LLM planning first and diffusion rendering second, the overall pipeline is usually longer than a single-pass generation.
    2. Higher compute cost: The planning model, diffusion model, and text compositing module all consume resources.
    3. Error propagation: If the planning stage misinterprets an instruction, the rendering stage may amplify the error. For example, if “left” is understood as “right,” even a refined image afterward will not meet business requirements.
    4. Increased tuning complexity: Control strength, region constraints, and text-layer parameters all need testing and cannot rely on a single default value.

    Therefore, hybrid architecture should be adopted cautiously in the following scenarios:

    • Early creative stages that only pursue rapid iteration;
    • Images with a single subject, atmosphere-first, and no need for complex semantics;
    • Lightweight interactions that are extremely latency-sensitive and expect immediate returns;
    • Teams that currently lack the ability to maintain complex inference pipelines and resource scheduling.

    If you adopt a hybrid architecture, it is advisable to add scene-graph validation, rule-based constraints, manual sampling, and failure fallback mechanisms. For example, after the planner outputs the blueprint, you can first use rules to check whether the object count, negation items, and spatial relationships are reasonable, then decide whether to proceed to the rendering stage.

    Summary

    Text-to-image technology is moving from “gacha” to “controllable.” Its essence is not simply swapping in a larger model, but restructuring single-pass pixel generation into a layered flow of “semantic planning, structural constraints, diffusion rendering, and text compositing.” Hybrid architecture lets the model understand first and draw later; glyph-awareness turns text from texture into symbols; and engineering capabilities carry image generation from demo effects to stable production.

    For developers and enterprises, understanding this evolutionary logic is more valuable in the long run than chasing a single parameter. A truly usable text-to-image system must not only generate beautiful images but also remain stable, interpretable, and optimizable under complex instructions.