Training Multimodal Image Generation Models on Domestic AI Chips: Engineering Practices and Optimization
A practical guide to training multimodal image-generation models end to end on domestic AI accelerators. It covers the hybrid "autoregressive backbone + diffusion decoder" architecture, Chinese text rendering, arbitrary-aspect-ratio generation, and system-level optimization across the framework, scheduling, parallelism, and operator layers, along with a developer checklist and an FAQ for avoiding migration pitfalls.
In recent years, the competition in multimodal image-generation models has been shifting from pursuing visual appeal alone toward semantic understanding, text rendering, business-specific size adaptation, and engineering stability. At the same time, another important change has emerged: more and more teams are trying to run the entire training pipeline on domestic compute platforms, rather than adapting only at the inference stage.
This article focuses on training multimodal image-generation models on domestic compute and discusses architecture choices, Chinese text rendering, arbitrary-aspect-ratio image generation, training optimization on domestic chips, and the boundary conditions to watch in real deployments. One caveat first: whether a fully domestic training pipeline is suitable for a project depends on whether the model weights, training framework, chip model, communication library, parallelism strategy, and data scale are available and verifiable. For closed models, models that expose inference APIs only, or projects where training dependencies cannot be confirmed, it is not advisable to assume that a full-pipeline migration can be completed.
1. What Is a Fully Domestic Training Pipeline
A “fully domestic training pipeline” usually means that all stages—data cleaning, preprocessing, pretraining, supervised fine-tuning (SFT), reinforcement learning (RL), and so on—are completed on domestic AI accelerator chips and their accompanying software stack, rather than only migrating at the inference stage. The difficulty of such tasks lies in the fact that pretraining is a long-running, large-scale, tightly coupled distributed workload with high demands on operator coverage, communication efficiency, numerical stability, and fault recovery. Any missing operator, precision deviation, or communication anomaly can cause training to stall, convergence to go wrong, or results to become irreproducible.
When discussing results, therefore, avoid the blanket claim that a solution is “on par with mainstream approaches.” A more rigorous statement is: in existing public cases or strictly aligned measurements, if model scale, training data, evaluation tasks, comparison baselines, and measurement methods are consistent, a fully domestic training pipeline may have comparable feasibility; if these conditions are missing, one should not directly infer that the results are equivalent.
2. Hybrid Architecture: Autoregressive Backbone and Diffusion Decoder
A multimodal image-generation approach that has drawn considerable attention recently typically adopts a hybrid architecture of “autoregressive backbone + diffusion decoder.” This design can be summarized as division of labor.
The autoregressive backbone handles semantic understanding and global planning. It receives text prompts, parses out the objects, relationships, layout, and semantic conditions to be generated, and is well suited to complex instructions, long-range dependencies, and “understand first, then draw” tasks.
The diffusion decoder handles image generation. Conditioned on the semantic output of the autoregressive backbone, it performs iterative denoising in pixel space or latent space to ultimately produce an image. For text regions in the image, a glyph encoding module is usually introduced to inject character-structure information into the decoder, improving the stability of text rendering.
The following is illustrative pseudocode. Note that it is not directly runnable and is only meant to explain the data flow. For actual use, replace the specific model interfaces, input/output formats, and dependency components:
# Hybrid architecture forward-pass sketch; not directly runnabledef forward(image_model, prompt, resolution):# 1) Autoregressive backbone: understand semantics, plan the composition# Replace image_model.ar_backbone.encode with the actual tokenization,# encoding, or conditioning-extraction interfacesemantic_tokens = image_model.ar_backbone.encode(prompt)# 2) Glyph encoding: provide character-structure features for text regions# Actual inputs may include character sequences, glyph images,# OCR/layout information, or character-position conditionsglyph_features = image_model.glyph_encoder(semantic_tokens)# 3) Condition fusion: combine semantic and glyph conditions# Actual fusion (concatenation, cross-attention, or other)# depends on the model architectureconditions = fuse(semantic_tokens, glyph_features, resolution)# 4) Diffusion decoder: sample an image under the guidance of the conditions# Replace diffusion_decoder.sample with the actual sampler and pass in# step count, noise schedule, guidance strength, and other parametersimage = image_model.diffusion_decoder.sample(conditions)return image
This pseudocode depends on components such as a tensor-computing framework, a text encoder, a glyph encoder, a condition-fusion module, and a diffusion sampler, as well as inputs like the prompt, resolution, random seed, and sampling configuration. For real implementation, specify the inputs and outputs of each interface, the device-placement scheme, and the numerical precision mode, so as to avoid describing the “hybrid architecture” only at a conceptual level.
3. Chinese Text Rendering: From Drawing to Writing Correctly
Common failures when generative models draw text include broken strokes, garbled glyphs, Chinese characters stitched into meaningless radicals, and incorrect line breaks in long text. Chinese text is especially difficult because the character set is vast and the structures are complex, including left-right, top-bottom, and enclosing structures. Compared with Latin letters, Chinese glyphs are more sensitive to stroke order, component position, and spatial proportion.
Common ways to improve Chinese text rendering include introducing glyph encoding, injecting precise character structure into the decoder as an additional condition, and optimizing layout and line-breaking strategies for long-text rendering. The role of glyph encoding is not to let the model “guess strokes by probability” but to give the model a clearer character constraint during generation.
For evaluation, it is advisable not to look only at whether a single image “looks like text,” but to establish more granular checks, such as character recognition rate, stroke recall, glyph integrity, multi-line consistency in long text, typo rate, and readability. For real business scenarios such as posters, covers, and product-detail pages, also evaluate whether text regions are disturbed by background texture and whether multi-line text remains clear after scaling.
4. Arbitrary-Aspect-Ratio Generation: Tokenizer and Positional Encoding Set the Boundary
Traditional image-generation models usually preset a few resolutions, such as 1:1, 4:3, and 16:9. But in real business, e-commerce detail pages, social-media covers, and vertical posters often require non-standard sizes. If a model can output only fixed aspect ratios, cropping, padding, or broken composition is likely to occur.
Supporting more flexible output ratios usually depends on the Tokenizer, positional encoding, and condition-injection strategy. Ideally, the model should be able to understand different width-height combinations within its supported resolution range, without being forced to align to a fixed grid.
But the boundary must be emphasized here: whether arbitrary aspect ratios are supported depends on the model design, Tokenizer, positional encoding, GPU-memory capacity, throughput target, and aspect-ratio constraints. “Arbitrary aspect ratio” does not mean that stable generation is possible at all sizes and for all content. In general, it should be used within the resolution range the model supports. For example, some models claim to cover a wide range of sizes, but the practically usable range must be judged in combination with GPU memory and the number of sampling steps.
When selecting a model, it is advisable to check the following: Does it natively support the target aspect ratio? What is the maximum pixel count? Is there dedicated encoding for text regions? Is GPU-memory usage acceptable at the target resolution? Does throughput meet business needs? Do extreme aspect ratios break the composition or text structure? Only when all these questions have clear answers can “arbitrary aspect ratio” be brought into a production environment.
5. Engineering Optimization for a Fully Domestic Training Pipeline
Training a multimodal image-generation model from scratch on domestic compute cannot rely on a single tuning trick; it requires systematic optimization across four layers: framework, scheduling, parallelism, and operators. At the same time, any environment must be recordable and verifiable, including the framework name and version, chip model, driver and firmware versions, communication library, parallelism degree, precision mode, and checkpoint strategy.
5.1 Framework Layer: Building an End-to-End Training Suite
It is advisable to build an end-to-end pipeline on an actually available domestic deep-learning framework or an adapted version, covering data preprocessing, pretraining, SFT, and RL. The specific framework may come from a domestic vendor’s in-house framework or a vendor-maintained PyTorch adaptation. Whichever is chosen, record verifiable environment information, such as framework version, chip model, driver version, communication-library version, parallelism strategy, and mixed-precision configuration.
A unified data format and training entry point reduce the maintenance cost of multi-stage training and make it easier to reproduce experiments across different clusters.
5.2 Scheduling Layer: Attention to Dynamic Graphs and Pipelined Dispatch
In large-scale training, if single-card instruction dispatch becomes a bottleneck, compute may sit idle. Multi-level pipelined dispatch with dynamic graphs can unfold the control flow level by level along the pipeline, easing dispatch overhead and improving per-unit compute utilization. When evaluating, observe single-card utilization, pipeline bubbles, instruction-wait time, and communication blocking, rather than looking only at theoretical peak compute.
5.3 Parallelism Layer: Overlapping Communication and Computation
In distributed training, gradient synchronization, parameter updates, or cross-card communication may interrupt computation. Multi-stream parallelism is a common approach: separate the communication stream from the compute stream and overlap communication with computation through event synchronization. The following is likewise illustrative and not directly runnable:
# Communication-computation overlap sketch; not directly runnablewith compute_stream, comm_stream:comm_stream.wait_event(compute_done)# Replace with the asynchronous communication interface# provided by the actual communication library or frameworklaunch_async(all_reduce, grads, stream=comm_stream)forward_backward(batch, stream=compute_stream)
In real engineering, confirm that the communication library, stream objects, event objects, gradient aggregation method, and framework interfaces all match. Poorly designed communication dependencies can lead to incorrect waiting, synchronization failures, or unstable performance.
5.4 Operator Layer: Chip-Affine and Fused Operators
For the computing characteristics of domestic chips, chip-affine fused operators can be used—for example, fusing AdamW updates, EMA, RMSNorm, and other operations into fewer kernel calls to reduce GPU-memory reads/writes and kernel-launch overhead. In long pretraining, such optimizations are especially important because small overheads are amplified over thousands of iterations.
However, fused operators must be judged not by performance alone but also by numerical stability. It is advisable to run precision-alignment tests before formal training, including single-layer outputs, loss curves, gradient distributions, and long-run convergence trends. If the precision deviation is unacceptable, fall back to a verified implementation rather than blindly pursuing throughput.
6. Developer Recommendations and Checklist
For teams that want to train and deploy multimodal image-generation capabilities on domestic compute, the following practical path can be referenced.
First, prioritize verifying operator completeness. Before formal pretraining, run an end-to-end smoke test on a small-scale task to expose missing operators, communication anomalies, precision deviations, or checkpoint-recovery failures as early as possible.
Second, use general-purpose infrastructure to reduce operational overhead. Training data can be stored in object storage, training jobs can be managed through container-orchestration platforms, data pipelines can be decoupled through message queues, and training metrics can be observed through logging services and monitoring alerts. The emphasis here is on general-purpose infrastructure types, not tied to any single vendor—for example, object storage, container orchestration, logging services, and monitoring/alerting.
Third, prioritize scenario-specific fine-tuning over blind retraining. Most business cases can perform SFT or lightweight alignment on top of a pretrained base, obtaining a text-rendering style, output style, and composition ability better suited to the business at lower cost.
Fourth, include text rendering and size adaptation in the acceptance criteria. These two directly affect the usability of the finished product, and it is advisable to focus on verifying them during selection, fine-tuning, and before launch.
Fifth, pay attention to long-run training risks. Long-cycle training must consider checkpoint recovery, data backup, permission isolation, GPU-memory-insufficiency fallback, and cost budgeting. It is advisable to include at least: saving checkpoints periodically; saving key random seeds and data versions; monitoring loss, throughput, card utilization, and convergence anomalies; setting up failure alerts for training jobs; and preparing a GPU-memory fallback strategy for extreme-resolution tasks.
7. FAQ: How to Avoid Pitfalls When Migrating to Domestic Compute
Q: How do you avoid pitfalls when migrating to domestic compute?
A: Run a small-scale end-to-end smoke test first, then scale up gradually. Focus on checking operator coverage, communication library, checkpoint, numerical precision, logging/monitoring, and fault-recovery capability; do not directly start long-duration pretraining.
Q: What if operators are missing?
A: First look for equivalent operators or official fallback implementations; if a custom one is required, run single-operator precision tests and end-to-end comparison tests. Simply falling back an operator to a generic implementation is not enough, as this can lead to excessively low throughput or unstable training.
Q: How do you debug the SFT and RL stages?
A: In the SFT stage, first check data quality, prompt templates, label formats, and the loss curve; if overfitting occurs, adjust the learning rate, sample distribution, or regularization strategy. In the RL stage, focus on checking the reward function, sample efficiency, abnormal rewards, policy drift, and training stability, and keep a fixed random seed and a comparison baseline.
Q: How do you choose a suitable domestic compute platform?
A: It is advisable to comprehensively evaluate chip model, framework version, communication library, operator coverage, long-run stability, monitoring capability, checkpoint recovery, and documentation completeness, rather than looking only at single-card compute metrics.
Conclusion
From hybrid architectures to fully domestic training pipelines, the engineering competition in multimodal image generation has entered a stage that places greater emphasis on stability, reproducibility, and business usability. For developers and enterprises, what is truly valuable is not abstract concepts but the ability to complete the full loop from training and fine-tuning to acceptance under well-defined hardware, framework, and data conditions. Understanding the division of architecture, controlling capability boundaries, and completing environment verification and risk control are the keys to more robustly bringing high-performance generative AI applications to autonomous compute.
