Planning and Text in Image Generation: Why Structure Beats Pure Diffusion
Complex image generation has long struggled with structural consistency—spatial relationships, attribute binding, and legible text rendering. This article explains why purely diffusion-based approaches fall short, how a hybrid architecture that separates scene planning from pixel rendering can fix the problem, and what engineering practices make text and logic constraints actually work in production.
Diffusion-based image generation has made impressive strides in visual quality, but two persistent problems remain: unstable spatial and logical relationships between objects, and illegible in-image text. Constraints like left-of, above, contains, or negation are frequently ignored. Meanwhile, generating readable text in posters, document mockups, and UI screenshots rarely works on the first pass.
Attributing these failures solely to “the model is not big enough” misses the deeper issue. Pure diffusion generation excels at filling pixels across a canvas but lacks an explicit scene-planning step. In essence, it treats semantic structure as if it were texture statistics.
Why “Painting Directly” Fails
The diffusion process gradually approximates a target distribution from noise. It handles local appearance, lighting, and material fitting well, but it is weak at honoring long-range constraints. When a prompt contains multiple subjects, relative positions, and negations—such as “the dog is not eating an apple”—the model tends to stack highly correlated patterns together. The color red might incorrectly attach to a dog, a bench could be rendered as raincoat texture, and negation words are often ignored as noise.
Text is an even more extreme case. For diffusion models, characters are often treated as high-frequency textures. The model may produce “brush strokes that look like letters” without a stable glyph topology: whether horizontal and vertical strokes align, whether enclosed areas close, or whether Chinese stroke order is intact. The result looks “like a poster” but the text is unreadable. A common engineering workaround is to overlay text with a layout tool afterward, but this breaks end-to-end generation and fails in cases where text must be integrated with scene lighting and perspective.
To meaningfully improve controllability, the logical structure of composition must be separated from pixel synthesis. First, form a verifiable scene blueprint; then let a rendering module fill in the pixels.
Hybrid Architecture: a Planning Brain and a Rendering Hand
One widely explored route is chaining autoregressive planning with diffusion-based rendering.
On the planning side, an autoregressive sequence model converts natural-language instructions into a structured intermediate representation. This intermediate representation does not need to be final pixels; it can be a list of subjects, attribute bindings, a spatial relationship graph, negation constraints, and text regions with exact string content. The goal of this step is to produce something inspectable and correctable. For example:
- Subjects: girl, bench, dog
- Attributes: raincoat = red, bench = blue
- Relationships: girl is to the left of bench, dog is to the right
- State: dog is not holding an apple (explicitly prohibits “apple in mouth”)
- Text: if a poster requires a title, specify its glyphs, font size, and alignment separately
On the rendering side, a diffusion model takes the plan as conditioned input—not loosely coupled text. Depth maps, layout boxes, and masks can supplement the condition. The diffusion module handles lighting, material, edges, and stylistic consistency. Because spatial slots are already fixed, the rendering phase mainly focuses on “looking right” rather than re-guessing “who goes where.”
This division mirrors how humans draw: compose first, then paint. The engineering benefit is straightforward: logic errors can be caught during planning through rules or secondary validation, instead of relying on human inspection after generating the full image.
Space, Negation, and Binding: What the Planning Phase Must Encode
“Think before you paint” alone is not enough. The plan must express three types of constraints explicitly.
First, spatial relationships. Left-of, front-of, and contains must be converted into coordinates or relative slots, not natural-language phrases. Use normalized bounding boxes, a nine-grid slot system, or scene-graph edges. When the prompt contains “left” or “right,” the planner must output mutually exclusive regions to avoid overlapping subjects or mirrored flips.
Second, attribute binding. Colors, clothing, and quantities must attach to specific subject IDs instead of sitting in a global bag of words. Otherwise, “red” tends to drift toward the nearest high-saturation object. Implement this as subject-attribute key-value pairs, then inject the corresponding text or embedding into each masked region during rendering.
Third, negation and state. Negation cannot rely on adding “don’t” to the prompt. A safer approach is to generate a prohibition list during planning, downweight conflicting concepts during sampling, or remove the corresponding object slot from the layout entirely. For example, “the dog is not eating an apple” should remove the “apple” subject altogether, not depend on the model understanding verb negation.
Once these three constraint types are structured, the stability of complex instructions improves significantly over text-only diffusion. Evaluation should track relationship hit rate, attribute-binding accuracy, and negation compliance—not just aesthetic scores.
In-Image Text: from Texture to Symbols
To make generated images pass a basic spelling test, text must be treated as a sequence of symbols, not as a decorative pattern.
Feasible approaches include glyph-conditioned encoding, per-character rendering, and separating layout from content. The layout module first determines text box position, alignment, and line width; the content module then renders strokes according to glyph topology. This is especially important for Chinese characters: a structured character such as 中 must have a continuous vertical stroke, not two disconnected vertical lines. For Latin letters, enclosed areas and serif consistency must be preserved.
A practical engineering suggestion is to extract text requirements from the main prompt into a dedicated field. For example:
layout:- id: titletext: "限时优惠"bbox: [0.1, 0.08, 0.9, 0.18]align: centerlang: zhsubjects:- id: girlattr: {coat: red, pose: sitting}region: left
At render time, text regions can receive higher-weighted conditioning. Alternatively, use a two-stage approach: generate the text-free scene first, then draw readable text within the specified mask and blend it with lighting. This reduces the rate of garbled output and allows the user to redraw only the text region without regenerating the whole image.
Readable is not the same as beautiful. Excessively small fonts, strong perspective, or bright reflections can cause OCR to fail. In production, “human-readable + machine-recognizable” should both be acceptance criteria.
Prompting and Workflow Recommendations
To use these capabilities effectively, prompts should read like compilable requirements rather than descriptive prose. Consider fixing the field structure: subject, quantity, attributes, space, prohibitions, text, style, and aspect ratio. Use relative positions with reference objects for spatial instructions. Encode prohibitions by removing objects rather than negating verbs. Provide exact text strings instead of expecting the model to infer slogans from context.
A practical workflow:
- Plan generation
- Rule-based validation
- Layout visualization
- Diffusion rendering
- OCR and relationship checks
- Localized redraw
If validation fails, write the correction back to the plan rather than shipping a broken image. Local redraws should support resampling per subject mask or per text box, avoiding a full restart on every failure.
The evaluation set should include: multi-subject left-right relationships, color binding, quantities, negation, mixed Chinese-English layouts, and small-font posters. Report accuracy for each category separately to avoid masking structural defects with an acceptable average aesthetic score.
Boundaries and Common Pitfalls
The hybrid architecture is not a cure-all. Errors in the planning phase get amplified by the rendering phase. If the intermediate representation is too coarse, the renderer will still “freely improvise”; if it is too fine-grained, latency increases. There is a balance to strike between structural granularity and generation speed. Style transfer and artistic deformation inherently conflict with strict typography. Choose the mode based on the business scenario: layout-first for design drafts, atmosphere-first for concept art.
A single impressive demo should not be mistaken for reproducible capability. Without constraint validation and a regression suite, production will still fall back into chance-based iteration. Real improvement comes from three things: structured conditioning, inspectable intermediate states, and treating text as symbols. Get these three right, and stable output on complex instructions becomes a process—not a lucky roll.
