Planning and Rendering for Complex Instruction-Based Image Generation
Complex image generation is not primarily a style problem—it is a constraint-execution problem. This article explains how to separate scene planning from image rendering, represent objects and relationships as structured constraints, handle negative conditions and readable text, and evaluate outputs with repeatable validation rather than relying on visual impression alone.
Image generation is moving from “looks like” to “satisfies the requirements.” The main challenge is not style, but whether the specified constraints are actually followed. When users provide subjects, attributes, relative positions, quantities, negative conditions, and text that must be readable, a model may treat the words as texture prompts instead of logical requirements. The result can be incorrect attribute assignments, reversed spatial relationships, negative instructions becoming positive elements, or text rendered as unreadable strokes.
A more reliable workflow is to break the task into verifiable scene planning and then pass the plan to a rendering process that specializes in lighting, texture, and visual appearance.
1. Where Generation Fails
Diffusion-based generation is effective at progressively forming textures, lighting, and overall visual style from noise. Its training objective primarily concerns whether the pixel distribution looks plausible, not whether every logical relationship in a sentence is satisfied. As a result, it may generate a girl, a raincoat, a bench, a dog, and an apple, but still fail to satisfy all of the following requirements at once:
- The girl is on the left.
- The dog is on the right.
- The bench is blue.
- The dog is not eating the apple.
Common errors generally fall into four categories:
- Attribute binding failures: A color, material, or action is assigned to a neighboring object.
- Spatial relationship failures: Left and right, front and back, containment, or occlusion are ignored or reversed.
- Negative and quantity failures: Instructions such as “do not include” or “include only one” are treated as ordinary co-occurrence terms.
- Text failures: Letters or Chinese characters are rendered as decorative patterns, with unstable stroke counts and structures.
These problems cannot be reliably solved by simply sampling more images. Additional sampling may improve the odds of success, but it does not guarantee that the constraints have been modeled. In production, visual quality and constraint satisfaction should be treated as separate acceptance criteria.
2. Plan First, Render Second
A more reliable architecture uses an autoregressive planning module to produce a structured scene plan, followed by a diffusion-based rendering module that fills in the pixels. The planning module does not need to determine the final lighting or texture. Its role is to convert natural language into an executable blueprint, including:
- Canvas regions: left, center, and right, or foreground, midground, and background;
- Object inventory: each object’s category, quantity, attributes, and actions;
- Relationship list: which object is to the left of another, which object occludes another, and which objects must not touch;
- Text inventory: the exact string, language, approximate position, and whether the text must be legible;
- Negative constraints: objects or states that must not appear.
The rendering module then generates the image according to the scene plan and the requested style. This prevents instruction understanding and visual rendering from being forced into the same denoising process. If the plan is wrong, it can be rewritten. If the rendering is wrong, a local redraw can be attempted while keeping the plan fixed.
This separation also improves auditability. By storing the planning result, a system can explain why the final image deviated from the original prompt.
A conceptual workflow is shown below. It describes control logic rather than a specific vendor API:
prompt -> parse_constraints()-> scene_plan {objects, layout, negatives, glyphs}-> validate(scene_plan) # quantity, conflicts, bounds-> render(scene_plan, style)-> check(image, scene_plan) # relationships and text sampling
Validation does not need to provide perfect automated judgment. At a minimum, it should check for duplicate or conflicting object names, negative items that also appear in positive descriptions, empty or excessively long text strings, and layouts that place two primary subjects in the same region.
These checks can be implemented with ordinary program logic and usually cost far less than repeatedly generating images without diagnosis.
3. Writing Executable Instructions
Prompts should be organized as object–attribute–relationship descriptions rather than long chains of adjectives. For example, a mixed description can be decomposed as follows:
- Subject A: A girl in a red raincoat, positioned on the left side of the image and sitting down;
- Subject B: One dog on the right side, with no apple in its mouth and no overlap with the girl;
- Background: A blue bench positioned behind and between the two subjects;
- Prohibited elements: Do not include a second dog or a person holding an apple.
Negative conditions should be written as independent constraints instead of being buried in a long sentence. Quantities should use explicit numbers rather than vague terms such as “some” or “several.” Spatial relationships should refer to identifiable objects instead of relying only on words such as “next to.”
When possible, an upstream system should produce structured fields first, with a template assembling the final prompt. Free-form language can provide style guidance, but it should not be the only source of constraints.
4. Treat Image Text as a Separate Problem
Posters, interface mockups, and infographics require accurate text. When text is treated only as image texture, a model may imitate strokes that look like writing without preserving the identity of each character.
A more reliable approach is to:
- Store the target string as a symbol sequence during planning rather than embedding it only in a style description;
- Apply stronger layout constraints to the text area during rendering, or reserve a clean background panel;
- For titles, prices, warnings, and other text that must be exact, overlay vector or text layers during post-processing while the image model generates only the background and decoration;
- Validate the result character by character rather than judging whether it looks approximately correct.
Long sentences, small font sizes, strong perspective, and complex typefaces all increase the failure rate. If a product requires legal copy or prices to be error-free, correctness should not be entrusted entirely to a generative model. The generated image should serve as a background, while the text is handled by a typesetting engine. This is a capability boundary, not something that prompting alone can completely overcome.
5. Evaluate More Than Visual Appeal
Before deployment, create a small, fixed regression set. Each test case should focus on one type of constraint rather than testing style, quantity, negation, and long text simultaneously.
Track at least the following metrics:
- Constraint satisfaction rate: Position, quantity, negative conditions, and attribute binding;
- Text accuracy: Measured through edit distance or manual, character-by-character verification;
- Stability: Whether repeated rendering from the same plan produces consistent failure patterns;
- Latency and retry count: Track planning failures and rendering failures separately.
Visual quality scores can still be useful, but they should not replace constraint scores. Otherwise, a model may optimize for images that look better while continuing to produce incorrect relationships.
Failed examples should be fed back into the planning templates. If a particular sentence pattern is frequently missed, the parser should explicitly decompose it instead of adding more synonyms to the prompt.
6. Engineering Trade-Offs
A hybrid workflow adds the cost of an additional planning step and introduces two possible failure points. Simple posters and loosely constrained illustrations may still be suitable for direct rendering. Technical illustrations, brand visuals, and materials containing prices or warnings benefit more from planning before rendering.
The scene plan can also be reviewed by a person before generation. Confirming the plan in advance can significantly reduce wasted image-generation attempts.
When deploying in the cloud, planning text, reference images, and generated images should be stored separately. Text and structured scene plans can be stored in object storage or a database, while generated images can be associated with tasks through task IDs. Asynchronous jobs can use a queue to smooth traffic spikes.
Logs and monitoring should record whether constraints passed and how long each stage took. User prompts should not be scattered as unencrypted plaintext across long-term debugging logs.
Reference images and prompts may contain business-sensitive information, so the generation service should use the same identity and audit controls as other business systems. Secrets should never be placed in prompts. When sharing results externally, export the finished image or document instead of exposing intermediate planning logs.
7. Practical Conclusions
The root cause of inaccurate complex instruction following is that a pixel-generation model is not necessarily required to form a verifiable scene logic before rendering. Separating autoregressive planning from diffusion rendering provides a clearer control structure.
By representing positions, quantities, negative conditions, and text as structured constraints—and validating them independently—image generation can move from trial-and-error demonstrations toward a repeatable production workflow.
Exact text, strong negative constraints, and precise counting should still receive human or programmatic verification. The model can propose the image, but the surrounding system must demonstrate that the image meets the specification.
