Planning and Rendering for Complex-Instruction Image Generation

    This article explains why image generation models often misplace attributes, swap spatial relations, ignore negations, and garble text when a prompt contains many constraints. It describes a two-stage approach in which a planning module converts the instruction into a structured scene specification and a renderer generates the image under that specification. It also covers explicit handling of relations and negations, text validation, assertion-based evaluation, and production considerations.

    Why Complex Prompts Look Understood but Render Wrong

    Sending a natural-language prompt to an image generation model may look like adding a few modifiers. In practice, it imposes several kinds of constraints at once: composition, attributes, quantity, negation, and readable text. Typical failures are not poor color choices. The constraints break apart. A color lands on the wrong object, left and right are swapped, an explicitly excluded object appears anyway, and text in the image becomes unreadable texture. For developers, these errors are harder to fix in post-processing than a style the user dislikes, because they break the instruction itself.

    Traditional diffusion-based generation is good at recovering texture, lighting, and materials from noise. Its training objective is mainly to produce images that look realistic or match a specified style. Words and pixels are usually linked through cross-modal alignment, which creates statistical associations. The model learns to recognize common objects and colors, but it does not guarantee that these elements combine syntactically into a coherent scene. Spatial words, negation, and precise glyphs are often weak signals in the training distribution. “Left” is not always labeled consistently in the data, and “do not include something” appears far less often as a hard constraint. As a result, the model tends to treat the prompt as a bag of keywords rather than a specification that must be satisfied item by item.

    Write the Scene as an Executable Specification Before Rendering

    A safer engineering split separates deciding what the image should contain from rendering it well. The first stage is handled by a language or autoregressive module with sequence modeling capability. It reads the full instruction and outputs a structured scene plan instead of sampling pixels right away. The plan should include at least a list of subjects, the attributes bound to each subject, relative positions, occlusion and layering, elements that must appear or must not appear, and any text that must be readable, including its language and approximate layout. The second stage uses a diffusion or similar rendering module to generate the image from this plan. Reference conditions constrain the composition and the glyph regions.

    This split is not about chaining two models and hoping for the best. It separates error types. The planning stage handles logical consistency: who is on the left, which attribute belongs to which subject, and what a negated clause actually excludes. The rendering stage handles visual quality: lighting, materials, perspective, and overall style. If the plan is vague, even a strong renderer can only beautify a wrong scene. If the plan is clear but the renderer lacks region conditions, text and small objects can still drift to the wrong positions. For this reason, the interface should pass structured conditions. Rewriting the plan as a longer prompt is not enough.

    Make Relations, Negations, and Attribute Bindings Explicit

    Spatial relations should not stay in natural-language terms such as “on the left” or “next to.” In practice, you can normalize the scene to a coordinate system and assign each object an approximate bounding box, a layer order, and an orientation. Then declare alignment relations. For example, the main subject is on the left, the companion object is on the right, and the two do not overlap. Quantities should be enumerated rather than described with vague words such as “some.” Attribute bindings should be written as object-attribute pairs, so that colors, clothing, and materials do not leak between entities.

    Negated clauses need to be compiled into an exclusion list. Relying on words like “not” or “don’t” in the prompt is not enough. For example, “not holding something” should be stated in the plan as: that object appears neither as a held item nor as a background prop. The rendering conditions should then lower the weight of the corresponding concept or apply a region mask. Appending negative words to the end of the prompt usually cannot offset strong co-occurrences in the training data. For relations such as containment, support, and occlusion, write them as checkable predicates: A is on top of B, or C is partially occluded by D. If the planning module cannot determine a relation, it should return a clarifying question or lower its confidence. It should not silently insert the most common composition.

    The following example shows a vendor-neutral intermediate representation of a planning output:

    1. scene:
    2. objects:
    3. - id: person
    4. attrs: [red_coat]
    5. box: [0.05, 0.2, 0.45, 0.9]
    6. - id: animal
    7. attrs: [no_apple]
    8. box: [0.55, 0.45, 0.9, 0.85]
    9. relations:
    10. - left_of(person, animal)
    11. - sitting_on(person, bench)
    12. text:
    13. - content: "周末市集"
    14. region: [0.2, 0.05, 0.8, 0.18]
    15. lang: zh

    The renderer consumes only the geometry, attributes, and text regions in this specification. The business side keeps the mapping between the original instruction and the specification. This makes it possible to tell whether a failure came from misunderstanding the instruction or from a rendering error.

    Text Is a Symbol, Not a Clothing Pattern

    Text failures in images often happen because the model treats characters as local texture. Strokes may resemble a character, but the topology is wrong. With multiple characters, they can merge, mirror, or appear and disappear. To make text on posters, interface mockups, or packaging concept images usable, the characters must be constrained as discrete symbols.

    In engineering, three layers of safeguards are usually needed. First, the planning stage locks the string, the language, the number of lines, and the reading order. It also prevents the rendering stage from freely creating copy. Second, reserve a separate region for text, away from high-detail textures and strong highlights. Estimate font size and weight separately for Chinese, English, and digits, because stroke density differs at the same pixel height. Third, use glyph conditions, region masks, or post-render validation to restore symbol-level correctness. Validation can detect the text in the image and compare it with the target string. If they do not match, repaint only the text region instead of resampling the whole image. This preserves a composition that already passes and avoids repeated full-image generation, which amplifies randomness.

    Do not treat “strokes that look like characters” as a passing standard. Acceptance should distinguish three states: exact match, missing or extra characters, and similar-looking characters with the wrong meaning. Only the first state can go into design delivery. The other two must go back to the planning stage or the text layer. They should not be fixed by manually guessing on the image.

    Evaluate More Than One Good-Looking Sample

    The root of trial-and-error results is the lack of repeatable constraint evaluation. Break the prompt into assertions that can be judged automatically or semi-automatically. Check whether an object exists, whether attributes are bound correctly, whether left-right and front-back relations hold, whether negated items are absent, whether the text matches the target string, and whether quantities match. Score each assertion separately, then look at the combined success rate. Reporting only “overall aesthetics” hides systematic relation errors.

    The test set should cover combinations that interfere with each other: multiple same-colored objects, crossed color and position, double negation, long Chinese titles, small-size annotations, and rare but plausible spatial layouts. Run regression tests with fixed random seeds for each instruction. Compare the specification satisfaction rate rather than a single lucky image. Keep the intermediate plan for failed samples. This shows whether the plan itself was wrong or the plan was correct but the rendering deviated. In the first case, revise the prompt compilation and relation templates. In the second case, strengthen the region conditions, increase the relevant steps, or narrow the repaint scope.

    Workflow, Cost, and Limits in Production

    A maintainable pipeline usually runs in this order: instruction normalization, plan generation, specification validation, conditioned rendering, text and relation review, and local repainting when needed. The original image, the specification, and the scores are then written to object storage. Logs and monitoring track the share of each failure type. Planning and rendering can run as two inference services so that each scales independently based on its queue. Put the heavy work on the rendering side. The planning side is a better candidate for caching, because structured results for identical instructions are often reusable. A container platform handles version isolation, so that an upgraded planning template does not break older rendering conditions.

    On cost, do not assume that one more round of planning always improves accuracy. If an instruction has only a single subject and a style word, direct rendering is cheaper. The hybrid flow adds the most value for multi-object scenes, strong spatial relations, negation constraints, and text that must be readable. Use rules to route requests first. Enable planning only when the system detects coordinate words, negation words, quoted copy, or more than a set number of entities. Manual review should happen at the specification layer and the text validation layer. Designers should not have to redraw images from scratch.

    The limits should also be acknowledged. An intermediate specification cannot express every aesthetic intent, and overly rigid bounding boxes can make compositions stiff. In open scenes, physical plausibility and fine details such as hands can still go wrong, independent of relation correctness. Planning models can also misread ambiguous sentences. In that case, the ambiguity should be exposed rather than resolved by inventing a single interpretation. External commitments should be phrased as improving the reproducible satisfaction rate of complex constraints, not as a guarantee that any long instruction will be drawn correctly on the first try.

    For business teams, the practical steps are these. Upgrade prompt engineering to specification engineering, and build object, relation, and layout templates for high-frequency scenarios. Change text delivery from “generate, then retouch” to “lock the region and validate automatically.” Replace subjective spot checks with an assertion suite. This turns image generation from chance-based sampling into a production process that can be located, regression-tested, and iterated.