From Gacha to Control: The Evolution of Semantic Understanding in Image Generation Models
This article traces how image generation models moved from unpredictable "gacha"-style output to controllable, deliverable results. It breaks the problem into three layers—semantic planning, text rendering, and structured delivery—and explains why single-stage diffusion models struggle with attribute binding, spatial relations, and negation. It then presents a "cognition-then-rendering" architecture, structured JSON blueprints, an iterative plan–render–critique–re-plan loop, region-level conditioning, and glyph-level text modeling. Finally, it covers moving from images to editable documents, along with evaluation, cost, deployment, and risk considerations.
Introduction: From “Gacha” to Deliverable
Every time a large model releases a new version, the first thing users notice is often the quality of the images, charts, and documents it produces. A version bump may look like a trivial change to a number, but behind it is often a shift in the underlying technical paradigm.
Image generation is one of the most visible areas where this change shows up. When earlier models were given complex descriptions involving multiple objects and attributes, their behavior was closer to “gacha”—running the same prompt a few times could produce a different, and sometimes wrong, result each time. This instability was no accident. Mainstream text-to-image models are built around diffusion denoising: they generate pixels directly from text conditions without any explicit understanding of the scene’s semantics. They can capture holistic features such as “style” and “color tone,” but they struggle to reliably handle structured requirements such as “who is where,” “which attribute belongs to which object,” and “which element should not appear.”
This article breaks the problem into three layers:
- Semantic planning: understanding and decomposing complex instructions;
- Text rendering: making text inside the image legible and usable;
- Structured delivery: getting the output into editable, publishable office documents.
It then offers engineering practices and evaluation methods for each.
1. Why Complex Scenes Go Wrong
Three Common Failure Modes
- Attribute mismatch: asked for “a girl in a red raincoat” and a “blue bench,” but the red ended up on the bench.
- Broken spatial relations: asked for a person sitting on the left side of a bench, but the person ended up standing in front of it.
- Failed negation: asked for “a dog that is not holding an apple,” but the model drew the apple in the dog’s mouth.
Why These Failures Happen
- Limits of the modeling approach: diffusion models use cross-attention to implicitly associate words in the text with generated regions. This binding is soft and global, with no explicit constraint on which word modifies which object.
- Bias in training data: “red raincoat” co-occurs far more often in the corpus than “red bench.” The more complex the condition, the more easily generation falls back to a high-frequency prior.
- The inherent weakness of negation: generative models aim to fit the data distribution, and samples expressing “the absence of something” barely exist in the corpus, so negation conditions are hard to encode correctly.
- Distorted evaluation: if you only look at overall image-text similarity, an image with “the right color but the wrong layout” can still score highly, and the model’s flaws get hidden by the average.
2. The Overall Approach: Plan First, Then Render
To improve stability in complex scenes, the core idea is to decouple “understanding” from “drawing”: the system first builds a structured scene intent, then hands it to a rendering module for execution.
A common implementation is a two-stage “cognition–rendering” architecture:
- Cognition layer: usually an autoregressive language model. Processing input word by word, it is naturally suited to instruction parsing, long-context understanding, and multi-step reasoning. It converts natural language into a structured blueprint, including the object list, attributes, spatial layout, negation constraints, and style and aspect ratio.
- Rendering layer: usually a diffusion-based model, responsible for lighting, materials, brushwork, and details. It generates the image conditioned on the blueprint, focusing only on “how to draw it well” rather than guessing “what relation is being drawn.”
1. Comparing the Two Paradigms
| Capability | Single-stage diffusion | Cognition planning + diffusion rendering |
|---|---|---|
| Complex instruction following | Text encoded directly; long sentences lose information | Parsed into a structured blueprint first; constraints are traceable |
| Entity–attribute binding | Implicit attention alignment; easy to mix up | Attributes explicitly attached to entity slots; hard to drift |
| Spatial relations | No explicit layout constraint | Layout generated first, then used as a rendering condition |
| Negation | Hard to encode; easily fails | Converted into forbidden objects or negative conditions |
| In-image text | Mostly texture-style generation; prone to garbled characters | Can incorporate glyph-level modeling and text-region control |
| Compute cost | One generation pass produces the image | Adds an inference stage; cost controllable via scheduling |
2. Intermediate Representation: A “Lingua Franca” Between the Two Models
The cognition and rendering layers are typically models trained and optimized separately. Without constraints, the cognition layer outputs plain natural language, and the rendering layer has to parse the semantics again, diluting the planning result with noise.
The intermediate representation should therefore be structured and verifiable. A common approach is to have the cognition layer output a JSON blueprint. It serves both as a rendering condition and as something that can be automatically validated independently of the image.
Example:
{"canvas": { "width": 1024, "height": 1024 },"entities": [{"id": "e1","category": "girl","attributes": { "clothing_color": "red", "clothing_type": "raincoat" },"relation": { "spatial": "left_of", "reference": "e2" }},{ "id": "e2", "category": "bench", "attributes": { "color": "blue" } },{ "id": "e3", "category": "dog" }],"forbidden": [{ "entity": "e3", "relation": "holding", "target": "apple" }],"style": "flat_illustration"}
In this example, red is attached only to e1’s slot and blue belongs only to e2; the relationship between e3 and the apple is explicitly defined as forbidden. The cognition layer thus fixes “which sentence constrains which object,” producing a structured intermediate representation that can be reviewed.
3. Iterative Repair: Turning Failure into an Observable Signal
A single planning round does not always succeed on the first try. A common engineering practice is to add a “plan–render–critique–re-plan” loop:
def generate_with_planning(prompt, max_attempts=3):plan = cognition_model.parse(prompt) # Cognition layer: natural language -> structured blueprintfor attempt in range(max_attempts):image = renderer.generate(plan) # Rendering layer: generate image conditioned on the blueprint# A vision-language model acts as a "judge": check whether objects and attributes match the requirementsreport = critic.check(image, plan)if report.passed:return image# Feed the failure reason back to the cognition layer in structured form, adjusting only the relevant constraints before regenerating# For example: red detected on the bench -> strengthen e1's color bindingplan = cognition_model.revise(plan, report.issues)return image
The value of the repair mechanism is that it turns “this time it drew wrong” from a random failure into a process that is observable, diagnosable, and retryable.
3. The Rendering Layer: Making Control Actually Reach the Pixels
1. From Condition to Layout
With text conditions alone, the rendering model has little control over the position, size, and occlusion of elements. A common engineering approach is to derive a low-resolution layout map from the blueprint: each object corresponds to a placeholder region and a category label. The rendering model takes this as an additional condition, first fixing the skeleton of the scene and then refining details.
The layout constraint can be either hard or soft:
- Hard constraint: each object must appear fully within its designated region;
- Soft constraint: objects should stay near their designated region but are allowed to adjust slightly based on actual content.
Given the inherent uncertainty of generated content, scenarios with hard requirements such as counting and direction are better served by hard constraints or “validate after generation.”
2. Region-Level Condition Injection
A global text condition makes different objects share the same set of semantic vectors, which easily leads to “cross-contamination.” Region-level control works like this:
- Map each object’s semantic vector one-to-one to its layout region;
- Have each region receive only the semantic information of its own object, keeping weak connections to other regions.
This maintains consistency among “attribute–object–region” throughout the denoising process.
3. Phased Generation
Generating a high-resolution image directly from noise in one pass is computationally expensive and prone to detail distortion. A common approach is to first generate the overall composition at a lower resolution, then increase the resolution and refine details stage by stage. This keeps the overall layout stable while achieving sharper detail locally.
4. The Cost-Effectiveness of Multi-Round Regeneration
Multi-round regeneration can significantly improve results in complex scenes but adds compute overhead. A practical approach is to tier by difficulty:
- Simple prompts use a single round;
- Complex prompts enable planning and critique;
- Only samples that still fail go into regeneration.
This concentrates compute where it is genuinely needed.
4. Text Rendering: From “Texture” to “Glyph”
1. Why Diffusion Models Draw Text Badly
In conventional image-text paired data, text in images mostly exists as image content and rarely comes with per-character annotations. Diffusion models can only learn text as a kind of high-frequency texture, with no understanding that “a character is made of strokes arranged in a specific structure.” The result is missing or extra strokes, strokes that merge together, and even characters that look like text but are actually illegible. Because Chinese glyphs have dense strokes, the problem is even more pronounced.
2. Approaches to Glyph-Level Modeling
- Stronger text encoding: use a text encoder with character- or subword-level understanding so the model “reads” the text’s semantics rather than just its appearance.
- Glyph conditions: build a glyph-level condition specifically for the text to be rendered—for example, describing stroke structure per character or providing glyph features—so that generation references the “glyph” rather than a “character token.”
- Region masks: use text-region masks to constrain where and over what range text appears, reducing interference with the background texture.
- Training-data enhancement: introduce synthetic data with precise text annotations and high-quality design assets to improve glyph consistency during training.
3. Boundaries That Remain
Even when a model can draw legible Chinese slogans, some boundaries remain:
- Long text and precise typography: multi-line layouts with multiple font sizes and exact alignment are hard for a model to guarantee entirely;
- Mixed Chinese–English text: the glyph features of different writing systems differ substantially;
- Small font sizes and low resolution: details are easily compressed and blurred.
For deliverables meant to be shared externally, a safer approach is “AI-generated visual assets + an overlaid vector text layer.” This preserves the design quality while ensuring text is accurate and editable.
5. From Image to Document: Structured Delivery of Generation Capabilities
For image generation to truly land, the output is often not a single image but entry into structured carriers such as office documents and presentations.
1. A General Pipeline
outline = planner.plan(topic, slide_count=8) # Outline: title, key points, and image intent for each pageslides = []for page in outline:# The image module generates visual assets; text always exists as text objectsvisual = image_engine.generate(page["visual_prompt"])slides.append({"title": page["title"],"bullets": page["bullets"],"image": visual,"layout": "title_left_image_right", # Layout is decided by the template engine})deck = template_engine.render(slides) # Apply layout templates to keep fonts and color schemes consistent
2. Key Design Points
- Separation of visuals and text: images are embedded as objects while text remains editable text objects. This preserves the visual expressiveness of generated images while avoiding the uneditable, unsearchable “whole-page screenshot” problem.
- Templated layouts: unified layout templates control fonts, color schemes, and spacing to ensure visual consistency across pages.
- Intent-driven imagery: the planning module generates an image intent for each page based on its content, and the image module then generates accordingly, avoiding a disconnect between images and content.
- Exportable and reversible: the output is structured objects, so after export to a common document format, users can still edit them in office software.
6. Engineering in Practice: Evaluation, Cost, and Deployment
1. Evaluation System
- Intent understanding: whether object count, attributes, spatial relations, and negation conditions are executed correctly;
- Text rendering: character accuracy, spelling accuracy, and legibility;
- Aesthetic quality: composition, lighting, and color harmony;
- Structural usability: whether document layouts are clean and text is editable.
It is advisable to build both an automated evaluation set and a human spot-check mechanism. Automated evaluation supports fast regression, while human spot-checks catch problems that automated metrics struggle to capture.
2. Cost and Latency
- Tiered scheduling: choose a lightweight or full pipeline based on prompt complexity;
- Cache reuse: hit the cache for identical prompts and reuse layouts for identical templates;
- Batch inference: use batch inference in offline scenarios to improve GPU utilization;
- Quantization and distillation: use quantized models and distilled lightweight models within an acceptable accuracy range.
3. Cloud Deployment
Cloud deployment can combine general capabilities:
- Elastic compute: container platforms scale on demand to handle traffic peaks;
- Object storage: store generated images and intermediate artifacts;
- Message queues: smooth out peaks and support asynchronous generation tasks;
- Logging and monitoring: track the latency, failure rate, and retry count of each generation.
For example, on Baidu AI Cloud, developers can combine general capabilities such as containers, object storage, message queues, and logging and monitoring to build the infrastructure layer for a generation service, scaling resources flexibly according to business needs.
4. Iteration Loop
- Collect failure samples (attribute mismatch, garbled text, layout drift);
- Categorize them as intent-understanding, rendering-quality, or layout issues;
- Feed them back to the cognition layer, the rendering layer, and the template engine respectively;
- Validate fixes with a regression test set.
7. Risks and Considerations
- Compositional generalization: once the number of objects exceeds a certain scale, constraint conflicts and occlusion problems increase significantly; for highly complex scenes, consider staged generation or human intervention.
- Precise information: content involving exact numbers, dates, or formulas should be rendered from structured data rather than relying on model generation.
- Content safety: review both prompts and outputs for compliance, label generated content, and establish a human-review channel.
- Copyright and privacy: pay attention to copyright and portrait-rights risks in training and generated assets, and avoid generating content that involves real people or protected styles.
Conclusion
From “gacha” to “control,” the evolution of image generation models is essentially a shift from “pixel fitting” to “semantic understanding.” The cognition layer decomposes complex instructions into a structured blueprint, the rendering layer turns the blueprint into a high-quality image, and text rendering and document structuring make the output truly usable, editable, and deliverable. It is the coordination of all three that represents the key path for landing today’s image generation technology in enterprise scenarios.
