Controllable Image Generation: Separating Reasoning and Rendering in Practice
This article examines why single diffusion models struggle to follow complex instructions and introduces a two-stage architecture that separates reasoning from rendering. It explains how a large autoregressive "planner brain" converts natural language into a structured blueprint that a diffusion "rendering hand" then paints, enabling explicit handling of attribute binding, spatial layout, negation, and text rendering. Readers will find a technical breakdown of these capabilities, a pseudocode illustration of the pipeline, and practical engineering guidance on schema design, evaluation, post-processing, and safety for building controllable generation systems.
Introduction: From “Lottery-Style” Output to True Control
Image generation has spread rapidly over the past two years, yet many users still run into “lottery-style” results after issuing complex prompts: bound attributes end up in the wrong place, multiple objects blur into one another, and negation conditions are ignored or even reversed. These problems are not tied to any single product. They are common technical challenges across the multimodal generation field. From a technical perspective, this article discusses why controllable image generation is hard and how the mainstream idea of separating reasoning from rendering systematically improves results.
1. Root Cause: Why a Single Diffusion Model Struggles with Complex Instructions
Traditional image generation is dominated by diffusion models. At its core, the process progressively denoises noise back into an image, learning an overall mapping from text to the pixel distribution. Its strength lies in rendering texture, light, shadow, and style, but it has an inherent weakness in compositional semantics:
- Weak attribute binding: It struggles to reliably tie “red” to “the girl” rather than “the dog,” which leads to attribute drift.
- Blurred spatial relations: It under-models relative directions such as “left,” “right,” and “above,” so element positions get jumbled.
- Ineffective negation: Negated semantics like “not eating an apple” are often ignored or even understood in reverse under pure conditional generation.
- Counting and attribution difficulties: In multi-object scenes, quantities, ownership, and hierarchical relationships are easy to confuse.
The root cause is that a single diffusion model “stacks pixels directly” instead of first establishing scene logic. It compresses a complex instruction into one holistic condition vector, lacking any explicit step for semantic decomposition and planning.
2. Core Idea: A Two-Stage Architecture That Separates Reasoning and Rendering
The industry has gradually converged on a shared insight: let the module responsible for “understanding” and the one responsible for “drawing” divide the labor and collaborate.
- Planner brain: A large autoregressive model (typically on the order of billions of parameters). It excels at sequence modeling and logical reasoning, first breaking the natural-language instruction into a structured “scene blueprint” — which objects exist, their respective attributes, relative positions, negation constraints, background settings, and so on.
- Rendering hand: A diffusion model. It receives the already-structured planning signals and focuses on fine-grained pixel-level rendering of light, material, and composition.
This biomimetic “think first, then draw” logic splits the “understanding” and “generation” that were previously coupled in a single step into two stages, allowing semantic correctness and image quality to be optimized together. A schematic flow is shown below (pseudocode, for illustration only):
def controllable_image_gen(instruction, planner, renderer):# Stage 1: the planner brain turns natural language into a structured blueprintblueprint = planner.to_structured_plan(instruction,schema={"objects": [{"name": str, "attributes": dict, "position": str}],"negations": [str],"background": dict,},)# Stage 2: the rendering hand fills in pixels based on the blueprintimage = renderer.render(plan=blueprint,steps=50, # illustrative parameter; tune to the taskguidance_scale=7.0, # illustrative parameter; controls constraint strength)return image
The key lies in the interface between the two stages: the blueprint is not free-form text but a structured representation that downstream components can parse reliably. Constraints such as position, attributes, and negation can therefore be passed explicitly instead of being drowned in the condition vector.
3. Key Capability One: Accurate Rendering of Text and Symbols
In scenes such as posters, banners, and greeting cards, on-image text is often a hard requirement. Traditional models treat text as a kind of “texture” to imitate, so they frequently produce garbled characters or “alien inscriptions.”
To render text correctly, it must be treated as a “symbol” rather than a “texture”:
- Introduce dedicated text-perception and encoding components so the model understands the stroke composition and glyph structure of characters, rather than learning only the statistical features of their appearance.
- Strengthen text–image alignment during training so the model knows which strokes each character is made of and how they are arranged.
- Design separate glyph and layout constraints for scenarios such as multilingual text, mixed Chinese–English typesetting, and artistic fonts.
The goal is to make the model not merely “look like text” but “write the correct characters,” substantially reducing manual rework afterward.
4. Key Capability Two: Understanding Spatial, Logical, and Negation Relations
Another difficulty in controllable generation is accurately expressing relations like “who is where” and “who does not do what.”
- Spatial relations: Use structured position fields (left/right/above/below/front/back) to constrain object layout, together with a relative reference frame, to reduce positional errors.
- Logical relations: For relations such as “contains,” “belongs to,” and “adjacent to,” annotate them explicitly at the planning stage, then constrain the spatial topology of elements accordingly at the rendering stage.
- Negation conditions: Extract negation semantics such as “not eating,” “do not want,” and “none” separately and pass them to the rendering module as negative constraints, so they are not overridden or reversed by positive semantics.
Precisely because these constraints are explicitly modeled at the planning stage, the model’s adherence to complex, compound instructions can improve markedly.
5. Engineering Practice Recommendations
To bring this kind of controllable generation into production, evaluate and integrate from the following angles:
- Define task boundaries: Distinguish scenarios that emphasize “texture/style” from those that emphasize “semantic accuracy.” The former can use a single diffusion approach, while the latter is better suited to a reasoning–rendering separation architecture.
- Design structured constraints: Define a clear blueprint schema for your business scenario (objects, attributes, positions, negations, background). This is the core of controllability.
- Build a quality evaluation system: Establish a multi-dimensional evaluation covering attribute correctness, spatial-relation hit rate, text legibility, and negation adherence — rather than judging only whether the image looks appealing.
- Post-processing and iteration: Even as model capabilities grow, it is still advisable to keep optional post-processing (such as text fine-tuning and local inpainting) as a fallback.
- Compliance and safety: Run content safety checks on generated content to avoid violations and infringement risks.
In addition, you can package the whole pipeline as a service, paired with object storage for outputs, a logging service to record the call chain, and monitoring and alerting to track quality fluctuations — forming a generation system that can be operated sustainably.
Conclusion
Moving from “lottery” to “control,” the key to image generation is not simply piling on parameters, but decoupling “understanding” from “generation” so the model plans first and then renders. A two-stage architecture that separates reasoning from rendering, combined with explicit spatial, logical, and negation constraints and dedicated text handling, can systematically improve adherence to complex instructions. For developers and enterprises, mastering this methodology matters far more than chasing any single product.
