Text-to-Image Generation for Complex Instructions: Why You Should Plan First and Render Later

    Complex prompts with multiple objects, attributes, spatial relations, negations, and on-image text are still hard for text-to-image models. This article explains why those prompts fail, how a two-stage "plan, then render" pipeline addresses them, how to render legible text as symbols rather than texture, where to add validation checkpoints in production, how to benchmark structured instruction following, and when the added complexity is actually worth it.

    1. Why Complex Instructions Fail

    The early problems of text-to-image models concentrated on semantic alignment. When a prompt describes several objects along with their attributes and spatial relations, the typical errors are:

    • Attribute misbinding: a color is assigned to the wrong object. The dog turns red, or the raincoat takes on the color of the bench.
    • Lost spatial relations: relative positions such as “on the left” or “on the right” are not reliably mapped onto regions of the frame.
    • Negation ignored: a constraint like “does not eat the apple” is dropped, and an apple still appears in the image.

    The root cause is that the text encoder outputs a high-dimensional continuous vector, and objects, attributes, positions, and negations are all blended into the same set of vectors. Diffusion denoises from noise step by step, and at each step it depends on the text condition only indirectly. There is no mechanism that explicitly guarantees that structural constraints such as “left,” “right,” and “not” are satisfied.

    2. Two-Stage Generation: Plan First, Then Render

    A common industry practice is to split generation into two steps. In the first step, a model with stronger language understanding parses the instruction into a structured scene description that spells out the objects, their attributes, spatial relations, and constraints. In the second step, an image generator paints the pixels according to that description.

    The benefit of this division of labor is that understanding and drawing each handle the part they are good at. The output of the planning stage is an intermediate representation that can be inspected, so validation, logging, and correction become practical engineering tasks. The costs are higher end-to-end latency and the need to keep the interface between the two modules stable.

    A scene description can use an illustrative format like this one. The fields are for explanation only and are not a fixed specification:

    1. {
    2. "canvas": {"width": 1024, "height": 1024},
    3. "objects": [
    4. {"id": "girl", "type": "person", "attributes": ["red raincoat"], "region": "left"},
    5. {"id": "bench", "type": "furniture", "attributes": ["blue"], "region": "center background"},
    6. {"id": "dog", "type": "animal", "attributes": ["has not eaten an apple"], "region": "right"}
    7. ],
    8. "constraints": [
    9. {"type": "negation", "target": "dog", "rule": "No apple appears at the dog's mouth"}
    10. ]
    11. }

    The plan should satisfy three requirements:

    1. Each object is bound to its own attributes, so attributes cannot drift between objects.
    2. Spatial relations are expressed with region labels or canvas coordinates, for example left, right, or middle ground.
    3. Negative conditions are converted into checkable rules instead of being left in natural language.

    3. Text Rendering: Treat Text as Symbols

    Producing legible text inside an image is a separate difficulty. Traditional diffusion models tend to learn text as texture, which produces broken strokes and malformed glyphs. The problem is especially visible with Chinese, because Hanzi are structurally far more complex than Latin letters.

    Three approaches are common in engineering practice:

    • Glyph conditioning: encode the text that must appear as a separate additional condition, so the renderer can explicitly reference the character structure.
    • Post-generation verification: run OCR on the generated image, compare the recognized text against the target character by character, and repaint local regions where they differ.
    • Layout constraints: restrict text length, font-size regions, and typography style to reduce the freedom of the generator.

    In practice, pass the text to be displayed in a dedicated field rather than mixing it into the descriptive sentence. For evaluation, measure deviation with character-level edit distance. Test mixed Chinese and English separately, because the two writing systems differ greatly in morphological complexity; looking at only one of them can mask problems in the other.

    4. Engineering Checkpoints for Production

    When you integrate this kind of model into a product, checkpoints can be placed at three positions:

    1. Input side: cap the instruction length, filter sensitive content, and confirm that the plan passes schema validation.
    2. Output side: run constraint checks on the generated image, covering object counts, dominant colors, relative positions, and text content.
    3. Process side: set a maximum retry count and a fallback strategy. When everything fails, return the best available result together with the checks that did not pass. Never fail silently.

    The pseudocode below shows this flow. Function names are illustrative:

    1. def generate_image(prompt, planner, renderer, checker, max_retry=3):
    2. plan = planner.parse(prompt)
    3. if not validate_schema(plan):
    4. raise ValueError("Plan does not match the schema")
    5. image, report = None, None
    6. for _ in range(max_retry):
    7. image = renderer.render(plan)
    8. report = checker.check(image, plan)
    9. if report.passed:
    10. return image, report
    11. plan = planner.refine(plan, report.failed_items)
    12. return image, report

    Plans can also be cached. When the same or a very similar instruction appears again, reuse the scene description and only re-render, which cuts down calls to the language model. The cache key should include both the instruction text and the version of the planning module, so an upgrade does not read back a stale plan.

    5. Evaluation

    Evaluation should not stop at whether the image looks good. It helps to build a prompt set that covers these dimensions:

    • Positional relations: left/right, top/bottom, front/back, containment.
    • Attribute binding: the correspondence between colors, materials, and counts and the objects they belong to.
    • Negative constraints: “does not contain X,” “no Y action.”
    • Text rendering: Chinese, English, digits, and mixed content.

    Report the pass rate for each dimension separately. When comparing model versions, fix the random seed so results are reproducible. Automated checks cover rules that can be quantified; human review covers the plausibility and aesthetic questions that automatic rules struggle to measure. Only the combination tells you whether a change produced a real improvement.

    6. Choosing a Pipeline

    Whether to adopt the two-stage approach depends on business needs.

    Good fits include e-commerce hero images, promotional posters, assets that require precise composition, and images that carry fixed copy. These scenarios demand high instruction adherence and text accuracy, so the inspectability of a two-stage design delivers real value.

    Poor fits include purely stylized creative work, latency-sensitive real-time interaction, and cases that only need a rough mood image. For these, single-stage generation is usually simpler and cheaper.

    When deciding, weigh four factors: instruction complexity, how much text appears in the image, the latency budget, and the long-term maintainability of the system.

    7. Conclusion

    Reliable generation from complex instructions comes down to separating understanding from drawing and making the intermediate result inspectable. Text rendering needs dedicated handling, and evaluation needs to cover structured capabilities. In production, validation, retry, and caching determine whether the approach holds up over time.