Handling Complex Instructions in Text-to-Image Models: Planning, Text Rendering, and Constraint Control
Text-to-image models often struggle with complex prompts involving spatial relationships, attribute binding, negative conditions, and text rendering due to their inherent diffusion-based architecture. This article explores engineering approaches to solve these issues, including a two-stage "plan then render" workflow, treating text as discrete symbols, and handling negative conditions with structured fields. It also provides a comprehensive evaluation framework and practical deployment guidelines to ensure high-quality, constraint-compliant image generation.
1. The Root Cause: Why Complex Instructions Fail
Common failures in text-to-image models center on four types of constraints:
- Spatial relationships: Left, right, top, bottom, front, back, and inclusion.
- Attribute binding: Which color, material, or quantity belongs to which object.
- Negative conditions: An object that should “not appear” or an action that should “not occur.”
- Text content: Titles, slogans, and numbers that require verbatim accuracy.
Diffusion models start from noise and progressively denoise, with the entire image driven by a unified set of semantic conditions. Once the text encoder compresses the prompt into vectors, attributes like “red,” “girl,” and “dog” can easily cross-contaminate, leading to high variance across multiple generations of the same prompt. This randomness is inherent to the model architecture and cannot be fundamentally solved merely by increasing the prompt length.
2. Approach 1: Plan First, Render Later
Splitting the generation process into two stages makes it easier to control the outcome:
- Planning stage: Parse the instruction into a structured scene description, explicitly defining objects, positions, attributes, relationships, and exclusions.
- Rendering stage: Generate pixels according to the structured description and verify against the plan.
Here is an example of a structured scene description:
{"canvas": {"ratio": "3:2"},"objects": [{"id": "girl", "desc": "girl in a red raincoat", "position": "left"},{"id": "bench", "desc": "blue bench", "position": "center"},{"id": "dog", "desc": "a dog, no food in its mouth", "position": "right"}],"negative": ["apple in contact with dog"]}
The benefits of this approach are that relationships can be verified, descriptions can be reused, and structures can be manually edited. The trade-off is the addition of a planning inference step, which increases latency, and the quality of the planning caps the upper limit of the final result. Therefore, the structured format requires clearly predefined field meanings.
Post-rendering, detection results can be used to verify spatial order:
def check_horizontal_order(spec_objects, detected_centers):"""spec_objects: List of objects from the planning result, position is left / center / rightdetected_centers: {id: horizontal center coordinate} from the detection module, 0 is leftmost, 1 is rightmostReturns a list of object ids that fail the constraint"""rank = {"left": 0, "center": 1, "right": 2}ordered = sorted(spec_objects, key=lambda o: rank[o["position"]])failed = []for a, b in zip(ordered, ordered[1:]):if detected_centers[a["id"]] >= detected_centers[b["id"]]:failed.append(a["id"])return failed
Upon verification failure, the system can resample or perform local inpainting only on the failed regions, rather than regenerating the entire image from scratch.
3. Approach 2: Treat Text as Symbols
If the model only learns glyph shapes as textures, ensuring correct stroke order and character structure is difficult. Three common countermeasures exist:
- Character-level representation: Introduce character or glyph information during the text encoding phase so the model understands how each character is composed of strokes.
- Specify text content: Pass the required text as a separate string rather than mixing it into the natural language description.
- Post-render recognition and comparison: Use Optical Character Recognition (OCR) to read the generated result and compare it verbatim with the expected text.
The comparison process can be implemented as follows:
def normalize(text):return "".join(text.split()).lower()def verify_text(rendered_image, expected_text, ocr):recognized = ocr(rendered_image)return normalize(recognized) == normalize(expected_text)
For complex scripts (such as CJK), additional checks for font library coverage, font consistency, and size proportions are necessary. If title-like text fails recognition, it should trigger a direct resample rather than relying on manual correction for each image.
4. Approach 3: Engineering Handling of Negative Conditions
Instructions like “no apples” are the most error-prone. The reason is that the negative word itself activates the negated concept; if “apple” appears in the text, an apple may appear in the image. Therefore, positive descriptions alone cannot fully express negative conditions.
In practice, the following methods can be combined:
- Negative prompts: List undesired elements. This reduces the probability of violations but cannot guarantee zero occurrences.
- Post-detection: Use object detection or segmentation models to find forbidden objects. Discard the result and resample if a match is found.
- Independent fields: Write negative conditions as a separate field in the planning structure, ensuring they are not treated as positive descriptions during rendering.
The third method is the most maintainable, as negative rules can be added or removed independently, much like test cases.
5. How to Evaluate Image Generation Quality
To determine if an image generation solution has truly improved, a fixed test set and quantifiable metrics are required. It is recommended to cover the following dimensions:
- Spatial relationship accuracy
- Attribute binding accuracy
- Negative condition violation rate (lower is better)
- Text correctness (tracked at both character and string levels)
- Counting accuracy
- Generation time and resource cost per image
Due to the randomness of generation, the same prompt should be sampled multiple times, reporting the pass rate rather than looking at a single result. Below is a simple evaluation framework:
from dataclasses import dataclassfrom typing import Callable@dataclassclass Case:prompt: strchecker: Callable[[object], bool] # Takes an image as input, returns whether it passeddef run_suite(generate, cases, n_samples=4):report = []for case in cases:passed = 0for _ in range(n_samples):image = generate(case.prompt)passed += int(case.checker(image))report.append({"prompt": case.prompt, "pass_rate": passed / n_samples})return report
The test set should cover simple, medium, and complex difficulty levels, and regularly incorporate failed samples from production to prevent the evaluation set from diverging from real-world requirements.
6. Engineering Implementation Suggestions
- Tiered invocation: Generate simple assets directly; route complex constraints through the “plan + verify” pipeline to control average latency.
- Cache planning results: Identical structured descriptions can be reused, which also facilitates comparing the effects of different model versions.
- Manual review nodes: Content involving brand names, prices, or legal disclaimers must undergo manual review.
- Canary release: Test new solutions on a small percentage of traffic to compare pass rates and user feedback before a full rollout.
- Presentation scenarios: Generate per-page content from a structured outline first, then export to common office presentation formats to ensure the results remain editable.
Additionally, pay attention to copyright and data compliance during implementation. Training data sources, the usage scope of generated content, and the privacy protection of user-uploaded assets should be clearly defined during the solution design phase.
