Training Multimodal Image Generation on Domestic AI Accelerators: A Full-Pipeline Practice
This article examines what it takes to move multimodal image generation from single-shot inference to a complete training pipeline on domestic AI accelerators. It covers hybrid autoregressive-diffusion architectures for text rendering, non-standard resolution training, system-level bottlenecks in graph dispatch and communication, data engineering across pretraining and alignment stages, and a practical checklist for evaluating whether a domestic-compute training loop is truly closed and reproducible.
Why Full-Pipeline Training on Domestic Compute Is Harder Than Inference
Public discussion of image generation models often focuses on output quality: whether posters, educational illustrations, and cover images are clear, and whether text is readable. For engineering teams, a more informative question is whether the entire workflow—data preprocessing, large-scale pretraining, supervised fine-tuning, and preference alignment or reinforcement learning—can run on the same set of domestic accelerators and training frameworks, rather than migrating only the inference stage.
Inference migration mainly involves operator coverage and peak memory. Full-pipeline training also faces data throughput, optimizer states, gradient synchronization, checkpointing, and failure recovery. If any part still depends on external accelerators, the training loop is not closed. Therefore, when evaluating image generation models on domestic compute, the first question should be whether the pipeline is closed; leaderboard scores come second.
Task Decomposition: Understanding and Writing Are Different Problems
Complex visual text generation and long-text rendering expose the weaknesses of pure diffusion models: global semantics and layout can be roughly correct, while local glyphs may have missing strokes, wrong characters, or disordered reading order. A more reliable industry approach is to separate these capabilities.
- An autoregressive module handles semantic understanding and global composition: topic, hierarchy, whitespace, and reading order.
- A diffusion decoder handles pixel-level synthesis and works with glyph or text-conditioning encoders to constrain local rendering.
The former is closer to “understanding the prompt and layout”; the latter is closer to “writing the characters correctly.” A hybrid architecture costs two sets of parameters, two sets of losses, and more complex alignment. The benefit is significantly better controllability in text-dense scenarios such as posters, infographics, and covers than an end-to-end single diffusion model.
A schematic training objective can be written as:
L = L_ar(layout, semantics) + λ * L_diff(pixels | layout, glyph)
λ needs to be adjusted according to the data distribution. Samples with a high proportion of text should increase the glyph-related loss; otherwise, the model will prioritize texture optimization at the expense of readability.
Text Rendering: Conditions Must Reach the Decoder, Not Just the Prompt
Failure modes for Chinese posters are consistent: the sentence in the prompt is “understood,” but the text in the image does not match. The root cause is usually not an insufficient vocabulary size, but dilution of the text condition during diffusion.
Practical approaches include:
- An independent glyph encoder that turns the target string into conditions aligned with spatial positions, rather than relying only on global text embeddings.
- Injecting that condition at multiple denoising timesteps, instead of giving a single prompt at the initial noise.
- Preserving real typography in training data: multiple lines, mixed layouts, small font sizes, outlines, and background contrast, rather than only short-title examples.
Evaluation should cover both complex visual text and long-text sets. Reporting only aesthetic scores can hide failures in character generation. In open-source communities, text-specific benchmarks often reflect this capability more accurately than general aesthetic leaderboards.
Non-Standard Resolutions: Tokenizers and Positional Encoding Must Change Together
Traditional image generation locks outputs to a small number of aspect ratios to keep patch grids, positional encoding, and batch shapes stable. Posters and social covers often require non-standard sizes such as 1240×1600. If cropping or padding is applied only at inference, text and subjects may be cut off or large invalid regions may remain.
A cleaner path is:
- Sample multiple widths and heights within a target pixel range during training, rather than sampling only squares.
- Improve image tokenization or patch partitioning so the grid can cover any aspect ratio within that range.
- Support variable length and width in positional encoding, avoiding forcing non-standard sizes into a fixed sinusoidal table.
The maximum resolution is still constrained by memory and attention complexity. When claiming “any aspect ratio,” teams should also specify the side-length range covered by training and the inference bucketing strategy; otherwise, batching efficiency can drop sharply.
Training Systems: Bottlenecks Are Often in Dispatch and Communication, Not Nominal Compute
Running multimodal training on domestic accelerators typically involves three categories of optimization, all of which should be described independently of any specific vendor.
Dispatch path. If a dynamic graph is recompiled and dispatched at every step, the host side becomes a bottleneck. A multi-stage pipeline that overlaps “preparing the next step’s graph” with “executing the current step” is needed to keep accelerators fully utilized.
Communication-computation overlap. Gradient synchronization in data parallelism and activation exchange in tensor parallelism should overlap with matrix multiplication. Multi-stream parallelism assumes that the collective communication library and compute streams can truly run concurrently, rather than merely enabling multiple streams while execution remains serial.
Operator fusion. Optimizer updates (such as adaptive methods with momentum and weight decay), exponential moving averages, normalization, and several elementwise operations are suitable for fusion into affinity kernels to reduce memory traffic. Fusion must be reproducible: numerical deviations can directly cause instability in later fine-tuning.
These optimizations serve the same goal: letting preprocessing, pretraining, fine-tuning, and preference optimization run on the same runtime, rather than switching clusters at every stage.
Data and Stages: Without Data Engineering, Architectural Advantages Disappear
The full pipeline should include at least:
- Preprocessing: resolution normalization, text and layout annotation, low-quality image and garbled-text filtering, and deduplication.
- Pretraining: first learning general composition and texture, then increasing the proportion of text-dense samples.
- Supervised fine-tuning: aligning layout and wording using posters, covers, and infographics filtered manually or by rules.
- Preference or reinforcement learning: rewards should include text accuracy, layout constraints, and basic aesthetics, rather than optimizing only for a “premium look.”
If rewards cannot be verified, the model may learn flashy backgrounds while continuing to write incorrect text. Text tasks are better suited to automatically checkable metrics—edit distance, recognition results within detected boxes—as part of the reward.
How to Judge Whether a Solution Is Worth Investing In
Without relying on any single product claim, use the following checklist:
- Does data preprocessing through alignment training complete on the same accelerator family, and can checkpoints resume training in place?
- Are text-specific and long-text evaluations public and reproducible, rather than limited to curated samples?
- Are non-standard resolutions part of the training distribution, rather than an inference-time cropping trick?
- Do fused operators and pipelines have numerical alignment reports, not just throughput numbers?
- Are open-source weights, training configurations, and inference instructions sufficient for third parties to reproduce the main conclusions?
For cloud deployment, object storage holds datasets and checkpoints, container platforms manage multi-machine jobs, and logging and monitoring alerts help locate slow nodes and communication jitter. These are general capabilities; they cannot replace the model’s own conditioning design and data quality.
Conclusion
The real progress in multimodal image generation on domestic chips is a closed training loop and controllable text rendering, not another impressive sample image. Autoregressive modules handle semantics and composition, while diffusion decoders with glyph conditions handle correct writing. Non-standard resolutions depend on joint changes to tokenization and positional encoding. On the system side, teams need to eliminate dispatch bottlenecks, overlap communication with computation, and fuse memory-intensive operators. Decisions should be based on pipeline closure, text evaluation, resolution coverage, and reproducibility—not on release messaging.
