Text-to-Image Generation for Complex Prompts: Plan First for Spatial Relations, Negation, and Text Rendering
This article explains why text-to-image models fail on complex prompts involving attribute binding, spatial relations, counting, negation, and rendered text. It presents a two-stage plan-then-render architecture with a validated intermediate representation, then details engineering strategies for region constraints, negation rewriting and post-processing, attribute binding, and text rendering. It concludes with a five-stage pipeline design.
I. Problem Background: Why Complex Instructions Fail
When text-to-image models handle complex instructions, failures are concentrated in five semantic categories: attribute binding, spatial relations, counting, negation, and text in the image. Consider the prompt: “A girl in a red raincoat sits on the left side of a blue bench, and on the right side there is a dog that is not biting an apple.” If red, girl, blue, bench, dog, and apple all appear in the image, the overall result may look acceptable. However, the ownership, position, and logical relations among these elements are not constrained item by item. As a result, common outcomes include colors in the wrong place, swapped positions, or ignored negation conditions.
The following table lists typical manifestations of each failure type and priority handling strategies.
| Failure type | Typical manifestation | Priority strategy |
|---|---|---|
| Attribute binding | Red drifts onto the bench | Bind attributes to a unique object id using modifier structure |
| Spatial relations | Girl and dog are left-right swapped | Use normalized region constraints and layout conditions |
| Counting | Three dogs requested, only two drawn | List instances individually in the object list and assign each instance an independent id |
| Negation | Dog bites the apple | Rewrite as a positive state description and verify with detection after generation |
| Text rendering | Glyphs look like text but are unreadable | Overlay programmatic typography and run OCR validation |
The root cause is that the model treats the prompt as a set of keywords that must appear together, rather than as structured constraints that must be satisfied item by item. Diffusion models generate images through multi-step denoising and excel at local texture and overall style. Spatial relations and logical negation require explicit compositional reasoning, which does not fully match the statistical nature of the denoising process. Simply increasing training data or scaling up the model usually does not eliminate these errors, so structured design is needed at the system level.
II. Architecture: Plan First, Then Render
A common approach splits generation into two stages. In the first stage, a language understanding module parses the instruction and outputs a structured scene plan. In the second stage, an image generation module renders the image according to the plan. The two stages are connected through an intermediate representation.
The intermediate representation usually contains five types of fields:
- Canvas information: size, aspect ratio, overall style
- Object list: each object’s id, name, attributes (color, material, state), and region
- Spatial relations: relative position, occlusion, contact relations
- Constraints: content that must appear and content that must not appear
- Text content: strings that need to be rendered and their positions
The following is an example. The apple’s position is consistent with the scene and is placed on the ground to the right of the bench.
{"canvas": {"ratio": "4:3", "style": "realistic photography"},"objects": [{"id": "girl", "name": "girl", "attrs": ["wearing a red raincoat", "sitting"], "region": "left"},{"id": "bench", "name": "bench", "attrs": ["blue"], "region": "center"},{"id": "dog", "name": "dog", "attrs": ["standing", "mouth closed"], "region": "right"},{"id": "apple", "name": "apple", "attrs": ["fallen on the ground to the right of the bench", "not touching the dog"], "region": "right_back"}],"relations": [{"subject": "girl", "relation": "sits on", "object": "bench"},{"subject": "girl", "relation": "is to the left of", "object": "dog"},{"subject": "apple", "relation": "is on the ground to the right of", "object": "bench"},{"subject": "apple", "relation": "is away from", "object": "dog"}],"negative": ["the dog has an apple in its mouth"]}
Before the planning output enters the rendering stage, it must pass the following validation. If any item fails, the process must return to the planning stage and regenerate; errors must not be passed into rendering.
- id is unique. The same id cannot appear twice in objects.
- region can only be one of left, center, or right, optionally with a _back suffix to indicate the background layer.
- The subject and object in relations must reference ids that already exist in objects.
- Left and right relations use the viewer’s perspective. Every “left” or “right” relation in relations must specify its reference object.
- Objects mentioned in negative must exist in objects, and object attributes must not conflict with the negation condition. For example, if the negation condition is “the dog has an apple in its mouth” and the dog’s attributes say “holding an apple in its mouth,” the two conflict.
- Each color, material, and state attribute belongs to only one object.
The advantage of this design is that each constraint can be checked independently. The disadvantage is that the planning stage adds latency and inference cost, and the plan itself can also be wrong. Therefore, the intermediate representation must be checkable and must not be passed between modules as a black box.
III. Handling Spatial Relations and Negation
3.1 Converting Spatial Relations into Region Constraints
Expressions such as “left,” “upper right,” and “foreground” are all relative concepts. In engineering, they can be mapped to normalized regions on the canvas. The coordinate origin is at the top-left corner, and both x and y are in [0, 1].
Coordinate convention: x, y ∈ [0, 1], origin at the top-left cornerleft -> x ∈ [0.0, 0.4)center -> x ∈ [0.3, 0.7)right -> x ∈ [0.6, 1.0]
The left, center, and right regions overlap, with intervals between 0.3 and 0.4 and between 0.6 and 0.7. Without ownership rules, the same object could be assigned to two regions. The following rules resolve conflicts.
- Ownership is determined by the object’s center point. An object’s bounding box may cross region boundaries, but only the center point determines which region it belongs to. For example, if the girl’s center point is at x=0.25, she belongs only to left; if the bench’s center point is at x=0.5, it belongs to center.
- When the center point falls in an overlap area, use the region explicitly declared in the planning stage. If it is not declared, assign it to the region whose centerline is closer to the center point. If the distances are equal, take the first in the order left, center, right.
- Objects in the same region are sorted by x from small to large, and adjacent objects’ bounding boxes must leave at least 5% of the canvas width between them. If this cannot be satisfied, shrink non-subject objects first. If conflicts remain, move one object into the _back layer and distinguish them by depth.
- Pixel coordinates are calculated as round(x × W), where W is the canvas width. Regions are treated as right-open intervals to avoid the same pixel belonging to two regions at once. The rightmost interval [0.6, 1.0] includes the right boundary.
- If relations contain explicit left or right relations, order objects in topological order. When there are no explicit relations, sort by id lexicographically to ensure the same input produces the same layout, which makes reproduction easier.
Region division does not need to be pixel-precise. For most scenes, coarse left-center-right and top-middle-bottom divisions can already significantly reduce the probability of position errors.
3.2 Rewriting and Post-Processing Negation
Negative prompts have some effect on “removing something,” but they struggle to precisely express “an object does not enter a certain relation with another object.” In many generation workflows, “the dog does not bite the apple” is interpreted as both a dog and an apple appearing in the image, while the negation is weakened.
A more reliable approach is to rewrite negation as positive state descriptions:
- “does not eat the apple” becomes “the dog’s mouth is closed, and the apple is away from the dog”
- “not a red dog” becomes “the dog’s fur is brown”
- “no text in the image” becomes “the background is a natural scene without text,” combined with post-processing to ensure the result
The value of positive descriptions is that they can be checked. Whether the model has completed the state “mouth closed” can be judged through keypoints or detection results, while “does not bite” is difficult to verify directly.
For constraints that cannot be fully rewritten as positive descriptions, post-processing is needed after generation. The steps are as follows.
- Detection: use object detection or text-region detection to locate prohibited content. “No text in the image” requires text-region detection; “the dog has an apple in its mouth” requires detecting whether the dog’s mouth and the apple overlap.
- Judgment: when prohibited content is detected, record its bounding box. Results with confidence below a threshold (for example, 0.5, to be calibrated on a validation set) are treated as not detected.
- Local inpainting: generate a mask only for the bounding box expanded by about 10%, and use “background without this content” as the prompt for local inpainting. The mask should be as small as possible to avoid changing unrelated areas.
- Programmatic overlay: if the prohibited content is text, directly fill the mask region with background color or texture instead of letting the model generate text.
- Failure judgment and retry: re-detect after local inpainting. If it is still detected, retry at most two more times, changing the random seed each time. After three failures, mark it for manual review and do not retry indefinitely.
Local inpainting can produce seams and style deviation at mask edges. After each modification, compare the structural similarity of the area outside the mask with the original image. If the difference exceeds a threshold (to be calibrated per sample), the modification range is too large, and the mask should be shrunk or the repair should be abandoned.
3.3 Attribute Binding
A common attribute binding problem is that color or material drifts onto other objects. The countermeasure is to make each attribute belong to only one object and use explicit modifier structure, for example, “the girl wearing a red raincoat,” rather than placing “red” alone at the beginning of the sentence. The clearer the modifier structure, the more stable the attribute ownership.
During planning, the following checklist can be used for self-checking:
- Does each color, material, or state word appear in only one object’s attrs?
- Are attributes written in the modifier position of the object name, rather than scattered elsewhere in the sentence?
- Do two objects have the same attribute description? For example, if two coats are both described as “red,” add an id or position description for each instance.
A counterexample is “a red coat hangs on the back of the chair, and another blue coat is placed on the floor.” This is clear by itself, but if it is written as “the red coat and the blue coat are both on the chair,” the model may assign both colors to the same coat.
IV. Engineering Strategies for Text Rendering
The difficulty of text rendering in image generation models is that characters are discrete symbols. They require correct stroke structure, character spacing, and glyph shape, while conventional training objectives focus more on visual similarity. Directly asking the model to draw a title often produces only text-like texture.
This can be improved at three levels.
- Model layer: choose a text encoding method that is sensitive to character structure so that the text condition carries glyph information. This capability requires specialized training; developers usually obtain it by selecting a model that already supports text rendering, rather than implementing it themselves. Different models vary significantly, and during selection it is advisable to evaluate the dimensions in the following table using your own test set.
| Evaluation dimension | Check method |
|---|---|
| Chinese support | Use a test set composed of common and rare characters and measure recognition rate |
| Glyph structure | Test characters with many strokes to see whether missing strokes or merging occur |
| Font size and multiple lines | Measure first-pass rate at different font sizes and line counts |
| Rotation | Test recognition results at 0°, 90°, and 15° rotation |
| Copyright and license | Check model license terms and confirm commercial scope and font sources |
Pipeline layer: remove text that must be presented precisely from the generation task. The image model handles background, composition, and atmosphere, while text is overlaid through programmatic typesetting. Font, size, and alignment are controlled by code, making the result stable and reproducible. If the text is a fixed title or price, this approach should be preferred.
Validation layer: perform text recognition on the generated result and compare the recognized result with the target string. The following pseudocode gives the decision logic.
function validate_text(image, target_text, retry_count):result = ocr(image) // returns text and overall confidenceif result.confidence < CONF_MIN:return REVIEW, "OCR confidence too low"recognized = normalize(result.text)expected = normalize(target_text)dist = edit_distance(recognized, expected)ratio = dist / max(len(recognized), len(expected), 1)if dist == 0 or (len(expected) >= 4 and ratio <= 0.05):return PASSif retry_count < 2 and ratio <= 0.2:return REPAIR_LOCAL, dist // local mask then programmatic reflowif retry_count < 2:return REFLOW, dist // reflow the entire block or regenerate the base imagereturn REVIEW, "retry limit exhausted"
The normalize function converts full-width characters to half-width, removes whitespace, and unifies case. Punctuation is kept by default because punctuation may be part of the content. Edit distance is used to judge the degree of error. When the string length is less than 4 characters, if even one character differs, it cannot be judged as passing and should enter the repair flow.
Validation thresholds need to be calibrated with samples; the values above are starting points. Validation must also handle the following cases.
- Chinese: the OCR model must support Simplified Chinese. Mixed traditional and simplified text and rare characters should be measured separately, not mixed with overall metrics.
- Artistic text and handwriting: OCR confidence is generally low. Such text should go directly to manual review, or standard font typesetting should be used instead.
- Rotated text: perform direction detection first, rotate the text region to horizontal, and then recognize it. If the direction cannot be detected, treat it as a failure.
- Low contrast: perform contrast enhancement or binarization on the text region before recognition. If confidence after enhancement is still below CONF_MIN (for example, 0.6, to be calibrated), treat recognition as unreliable.
- Misjudgment: similar-looking Chinese characters (for example, characters that differ by only one stroke) may be misrecognized by OCR and judged as passing. This risk must be controlled through sampling and manual review.
The retry limit is: the same text can be locally repaired at most 2 times and fully reflowed at most 2 times. If the limit is exceeded, or if a manual review condition listed in the next section is hit, the case is transferred to manual review.
Manual review triggers include: text contains artistic text, brand names, prices, legal or medical content; OCR confidence is below the threshold and retries are exhausted; the same request fails three consecutive rounds; the user explicitly requests review.
V. Complete Pipeline Design
A practical pipeline includes five stages.
