Training and Selecting Multimodal Image Generation Models on Domestic AI Accelerators
This article explains how to train and select multimodal image generation models on domestic AI accelerators. It covers defining the full training pipeline before choosing an architecture, the hybrid autoregressive-plus-diffusion design, accurate Chinese text rendering, flexible resolution output, and the engineering practices that matter most on domestic hardware—framework adaptation, operator fusion, parallel scheduling, logging, and checkpoint resume. It also provides tabular acceptance metrics and a selection decision tree to help teams decide whether domestic compute fits their needs, along with an FAQ on common failures and troubleshooting.
1. Training Image Generation Models on Domestic Compute: Define the “Full-Pipeline Loop” First
Multimodal image generation models are moving from “can draw” to “draws accurately.” If your team wants to train an image generation model on domestic AI accelerators, the first step is not to pick a model architecture. It is to define the scope of the “full-pipeline loop”: can data cleaning, image-text pair construction, tokenizer design, pretraining, supervised fine-tuning, preference optimization or reinforcement learning, and evaluation and inference deployment all run stably on the same stack of domestic hardware, drivers, communication libraries, and training framework?
Here, “full pipeline” does not mean just porting inference—it means adapting the entire training chain. It tests not whether a single operator is fast, but the combined capability of framework adaptation, operator ecosystem, parallelism strategy, fault recovery, and long-run training stability. For developers, if a key operator, communication backend, or debugging tool is missing, the problem is often amplified at scale, showing up as throughput drops, abnormal loss, node dropouts, or failed checkpoint resume.
Before starting, define three types of goals:
- Business goal: e-commerce main images, posters, social media covers, or general creative generation?
- Quality goal: does it require correct Chinese text, long-text stability, and reusable layouts?
- Engineering goal: are single-card throughput, memory ceiling, thousand-card scaling efficiency, and fault recovery time acceptable?
Without clear acceptance criteria, it is not advisable to jump straight into large-scale training.
2. Hybrid Architecture: Autoregressive for Understanding and Layout, Diffusion Decoder for Pixel Restoration
Today’s high-quality image generation models increasingly adopt a hybrid paradigm of “autoregressive + diffusion decoder.” The autoregressive backbone typically handles semantic understanding, instruction parsing, and global layout planning; the diffusion decoder then restores pixel-level detail and generates texture under semantic and layout constraints.
You can think of inference as two stages:
# Pseudocode: hybrid architecture generation flow (illustrative only, not a real API)def generate(prompt, target_size):# Stage 1: autoregressive backbone understands semantics and plans layoutsemantics = ar_backbone.reason(prompt) # understand intentlayout = ar_backbone.plan_layout(semantics, target_size)# Stage 2: diffusion decoder restores pixels under constraintsimage = diffusion_decoder.decode(condition=layout,glyph_hint=glyph_encoder.encode(prompt), # text rendering guidancesize=target_size,)return image
The advantage of this division of labor is “cognitive generation”: the model first understands the request and plans the image, then generates it step by step. It is better suited to complex instruction following, multi-object layout, Chinese posters, and local editing.
However, the hybrid architecture is not always better in every scenario. It is important to define the applicable boundaries:
- Suitable scenarios: complex instructions, text layout, multi-subject composition, and business cases that need strong layout control.
- Use with caution: low-latency generation, limited compute budget, simple texture generation, or when a single-tower small model already meets the need.
- Prerequisites: high-quality image-text pairs, clear layout supervision, a stable text encoding module, and sufficient training scale.
Regarding “whether it is more stable,” the statement should be conditional: in complex instruction-following and text-layout scenarios, if training data coverage is sufficient, evaluation tasks match production scenarios, and model scale matches the compute configuration, the hybrid architecture is often more controllable. But if data is insufficient or evaluation criteria are inconsistent, the hybrid architecture is not necessarily better than a traditional single-tower diffusion model.
3. Chinese Text Rendering: From “Can Write” to “Writes Correctly and Stably”
Generating landscapes or portraits is relatively mature, but accurately writing Chinese text within an image remains a hard task for image generation models. The difficulties concentrate in three areas:
- Complex glyph structure: Chinese characters have many strokes and dense components; even slight deviation in the diffusion process can produce missing strokes, missing parts, or blurry characters.
- Long text tends to break down: as text gets longer, repetition, missing characters, character bleed, and wrong sentence breaks appear.
- Semantic-pixel coupling: text must be correct in both content and glyph shape, and a single module struggles to guarantee both.
A common solution is to introduce a dedicated text or glyph encoder into the diffusion decoder that converts the target text into a strong constraint and steers the text region during decoding. You can think of this as giving the model a “standard copybook” to write under constraint, rather than generating purely from scratch.
Avoid absolute conclusions. Whether something “leads” should depend on evaluation conditions. A more rigorous statement is: after specifying the model version, test set, resolution buckets, decoding parameters, and scoring criteria, some open-source models with glyph-guidance mechanisms may perform well in public evaluations of Chinese visual text generation and long-text rendering. Without this information, one should not generally claim “clearly leading.”
Build an error-diagnosis table for text rendering:
| Error type | Common symptom | Possible cause | Detection method |
|---|---|---|---|
| Missing strokes | Incomplete glyph, missing components | Insufficient glyph supervision, too few diffusion steps | Template comparison, OCR recognition |
| Wrong characters | Similar or incorrect characters | Text encoding errors, weak semantic constraints | OCR character-by-character comparison |
| Missing characters | Several characters missing in long text | Context truncation, insufficient attention | Text length check |
| Character bleed | Repeated or misaligned adjacent characters | Unstable positional encoding, layout planning failure | Layout detection, manual spot checks |
| Wrong position | Text appears in wrong region | Layout planning deviation, insufficient conditional control | Region hit evaluation |
| Blurry text | Text region blurry and unreadable | Insufficient resolution, weak decoder detail capability | Sharpness scoring, manual review |
For production, use both automated evaluation and manual spot checks. Automated evaluation can cover OCR accuracy, character completeness rate, and position hit rate; manual evaluation is better suited to judging font style, visual comfort, and business usability.
4. Flexible Resolution Output: From Fixed Tiers to Business-Specified Aspect Ratios
Traditional image generation models usually support only fixed resolution tiers, such as 4:3, 16:9, and 1:1. But real business needs often require more flexible sizes, such as e-commerce main images, social media covers, posters, and detail-page banners. Such scenarios should not be simply generalized as “arbitrary aspect ratios.” A more accurate statement is: support for business-specified common aspect ratios and a specified resolution range.
For example, the business may require support for the following sizes:
- 1024×1024: general image generation
- 1240×1600: social media vertical cover
- 1080×1350: e-commerce content image
- 1920×1080: banner or video cover
- 2048×2048: high-resolution poster candidate
To achieve this, the key is not just changing parameters at inference time, but designing resolution buckets, positional encoding schemes, and memory strategy during training.
| Approach | Training cost | Memory pressure | Layout-break risk | Applicable scenario |
|---|---|---|---|---|
| Fixed resolution | Low | Low | Low | Single output type |
| Bucketed training | Medium | Medium | Medium | Limited common aspect ratios and business sizes |
| Native multi-resolution | High | High | Relatively high | Posters, complex layouts, high-resolution needs |
Engineering strategies you can adopt:
- Training resolution buckets: divide common sizes into several buckets, such as 1:1, 4:3, 3:4, 16:9, 9:16, with different resolution tiers.
- Positional encoding adaptation: use 2D RoPE, 2D sinusoidal positional encoding, bucket-ID encoding, or dynamic scaling to reduce the model’s strong dependence on a fixed grid.
- Memory budget control: higher image resolution significantly increases the token count, so evaluate activations, attention overhead, and gradient checkpointing accordingly.
- Layout-break detection: add checks such as edge truncation, distorted subject proportions, repeated patterns, local blur, and text overflow to acceptance.
For the business side, the value of flexible resolution is not just “being able to produce an image,” but reducing secondary cropping, redrawing, and layout rework. If the model can directly output an image that fits the placement, overall content production efficiency improves noticeably.
5. Training Engineering on Domestic Compute: From Principles to an Executable Checklist
When training multimodal image generation models on domestic chips, the engineering side is often more critical than the algorithm side. I recommend teams build an executable checklist before training that covers framework versions, chip adaptation, operator optimization, parallel configuration, logging metrics, and checkpoint resume acceptance.
1. Framework and Ecosystem Adaptation
Confirm that the training framework, domestic adaptation version, compiler, driver, and communication library form a stable version matrix. Lock the image, version numbers, and dependencies to avoid environment inconsistency across nodes.
| Check item | Recommendation | Acceptance evidence |
|---|---|---|
| Framework version | Mainstream open-source framework or domestic adapted version | Image digest, version lock file |
| Chip adaptation version | Chip model, driver, SDK, communication library | Environment matrix and compatibility report |
| Data pipeline | Data preprocessing, caching, sampling, filtering | Sampling consistency check |
| Training suite | Whether pretraining, SFT, RL flows are connected end to end | Small-model end-to-end run record |
2. Operator Fusion Optimization
High-frequency operators are the focus of throughput optimization. Evaluate at least modules such as AdamW, EMA, RMSNorm, LayerNorm, GELU, CrossEntropy, and attention fusion. The goal is not simply to reduce the number of operators, but to reduce memory access overhead, kernel launch counts, and communication wait time.
| Operator/module | Optimization goal | Measurement |
|---|---|---|
| AdamW | Reduce memory access during parameter update | Step time, samples/s |
| EMA | Reduce weight synchronization overhead | Extra time ratio |
| RMSNorm | Reduce reduction and memory access | Kernel count, single-card throughput |
| Attention | Reduce memory and compute waste | tokens/s, peak memory |
| CrossEntropy | Reduce large-vocabulary compute overhead | Training throughput, peak memory |
When measuring, distinguish three levels: single card, single machine multi-card, and multi-machine multi-card. Only optimizations that reproduce across different parallel scales are suitable for production training.
3. Parallelism and Pipeline Scheduling
In large-scale training, the parallelism strategy directly affects scaling efficiency. Focus on:
- Data parallelism: control global batch size and gradient synchronization frequency.
- Tensor parallelism: split the large model backbone to reduce per-card memory pressure.
- Pipeline parallelism: partition by layer to reduce per-card load.
- Sequence parallelism or long-context optimization: suitable for high-resolution scenarios with many visual tokens.
- Multi-stream parallelism: overlap compute streams and communication streams to reduce communication wait.
- Dynamic graph multi-level pipeline dispatch: relieve instruction dispatch bottlenecks.
Record the actual parallel parameters of each training run, including parallelism dimensions, gradient accumulation, activation checkpointing, communication backend, and pipeline partitioning. Do not just record “parallelism was used”—record “why it was partitioned this way” and “how metrics changed after switching.”
4. Logging Metrics and Stability Monitoring
Training logs should not contain only loss. Include at least:
| Metric category | Key metrics | Purpose |
|---|---|---|
| Training progress | step, epoch, tokens seen | Determine whether training is progressing normally |
| Quality metrics | loss, grad_norm, lr | Detect divergence, gradient explosion, or abnormal learning rate |
| Throughput metrics | samples/s, tokens/s, step time | Assess compute utilization and bottlenecks |
| Communication metrics | allreduce time, communication ratio | Determine whether communication is a bottleneck |
| Stability metrics | nan/inf, OOM, timeout, node restart | Assess long-run training risk |
| Resource metrics | peak memory, CPU/GPU utilization | Determine whether there is room for optimization |
5. Checkpoint Resume and Fault Isolation
Large-scale training cannot only consider “whether it can run”; it must also consider “whether it can recover quickly after breaking.” Checkpoint resume acceptance items should include:
- Whether optimizer state is fully saved.
- Whether EMA weights are saved.
- Whether the learning rate scheduler can resume.
- Whether the data loading offset can resume, avoiding duplicate or missing samples.
- Whether the loss curve is continuous after resume.
- Whether evaluation results match expectations after resume.
- Whether random seeds and key states are controllable.
- Whether faulty nodes can be automatically isolated and rescheduled.
If loss jumps significantly after checkpoint resume or evaluation metrics fluctuate widely, it means the training state was not truly recovered—not simply “continued training.”
6. Acceptance Metrics and Selection Recommendations
When training multimodal image generation models on domestic compute, the final measure is whether it passes business acceptance. Do not rely solely on “public evaluations” as the only basis; instead, build a tabular set of acceptance metrics.
| Acceptance dimension | Example metric | Recommended criteria |
|---|---|---|
| Text accuracy | OCR character match rate, wrong-character rate | Specified test set, specified resolution |
| Glyph completeness | Component completeness rate, missing-stroke rate | Template comparison or manual scoring |
| Instruction following | Multi-object hit rate, attribute accuracy | Rule-based judgment or manual review |
| Resolution consistency | Edge truncation rate, proportion distortion rate | Compare output size against placement |
| Throughput efficiency | samples/s, tokens/s | Fixed batch and parallel configuration |
| Stability | Task restart count, nan/inf count | Long-run training duration |
| Training cost | Per-step cost, total cost estimate | Combined compute and labor |
Selection Decision Tree
Judge step by step with these questions:
Is there a supply chain constraint?
- Yes: prioritize domestic hardware and adapted frameworks.
- No: comprehensively compare the ecosystem maturity of domestic and overseas solutions.
Is high-quality Chinese text rendering required?
- Yes: first validate glyph encoding, text-region constraints, and OCR evaluation.
- No: use a lighter generation pipeline.
Is multi-ratio or high-resolution output needed?
- Yes: design resolution buckets, positional encoding, and memory budget in advance.
- No: a fixed-resolution approach is usually cheaper.
How large is the data scale?
- Small: prioritize fine-tuning, distillation, or lightweight adaptation.
- Large: need more complete pretraining, parallelism, and fault-tolerance capabilities.
Is the launch timeline tight?
- Yes: first run the minimum viable loop, then scale up gradually.
- No: reserve time for operator optimization, evaluation systems, and stability building.
7. FAQ
1. What are common failures when training image generation models on domestic chips?
Common failures include missing operators, communication timeouts, memory overflow, nan/inf in loss, node dropouts, failed checkpoint resume, and data loading offset errors. When troubleshooting, first confirm whether the minimum viable loop is reproducible, then check parallel configuration, communication libraries, operator fusion, and random seeds.
2. What usually causes resolution layout breaks?
Common causes include insufficient training resolution buckets, ill-suited positional encoding, memory shortage leading to degradation, limited attention range, and uneven data size distribution. First validate with fixed resolution, then gradually introduce bucketed training.
3. How to troubleshoot poor Chinese text rendering?
Troubleshoot stage by stage following “text encoding → layout planning → glyph guidance → diffusion decoding → post-processing.” First confirm whether the text is correctly encoded, then check whether the text region is accurately planned, and finally evaluate glyph completeness and OCR accuracy.
4. Must you pretrain from scratch?
Not necessarily. If business data is limited, prioritize using an open-source base, supervised fine-tuning, glyph module enhancement, or a distillation approach. Only when data scale, compute resources, and the evaluation system are all mature is large-scale pretraining recommended.
5. How to judge whether domestic compute is ready for production?
Run three tests: an end-to-end minimum viable loop, a long-run training stability test, and a business acceptance test. Only when training is recoverable, metrics are reproducible, and generated results meet business placement requirements is it ready for formal launch.
8. Summary
Training multimodal image generation models on domestic compute is not about simply replacing hardware, but about building a reproducible, verifiable, and scalable engineering system. The autoregressive model handles understanding and layout, the diffusion decoder handles pixel restoration, glyph guidance improves text controllability, resolution buckets and positional encoding enhance output flexibility, and operator fusion, parallel scheduling, and checkpoint resume determine whether training can run stably.
For developers, what truly needs to be mastered is not a single point technique, but the complete methodology from data, architecture, training, evaluation, to launch. Defining boundaries, building checklists, and letting metrics speak is a more reliable path to building multimodal image generation capability in contexts that prioritize technological self-reliance and supply-chain controllability.
