Training Multimodal Image Generation Models on Homegrown AI Accelerators: Key Techniques

    This article examines what it takes to train multimodal image generation models end to end on homegrown AI accelerators developed in China. It covers why hardware portability matters on both the supply-chain and technical fronts, what a complete training pipeline (data preprocessing, pretraining, SFT, RL alignment) really involves, the hybrid autoregressive-plus-diffusion-decoder architecture that combines semantic understanding with pixel-level generation, how to achieve accurate Chinese text rendering and native arbitrary-aspect-ratio output, training engineering optimizations such as pipelined dispatch, communication-compute overlap and fused operators, and the cloud infrastructure required for deployment. It closes with four transferable lessons for teams targeting hardware-diverse training stacks.

    1. Why “Homegrown Accelerators + Multimodal Image Generation” Matters

    Over the past few years, training multimodal image generation models has relied heavily on a small number of mature accelerator cards and their surrounding software ecosystems. As supply conditions have become less predictable, more and more teams are treating “the ability to train from scratch across multiple hardware backends” as a hard capability to build. This matters on two levels:

    • Supply chain: Reduce dependence on a single hardware ecosystem and avoid bottlenecks at critical stages.
    • Technical: It genuinely tests the portability of deep learning frameworks, operator libraries, communication libraries, and parallelism strategies. Training code that only runs on one specific hardware setup tends to hide a large number of implicit assumptions; only when the full pipeline runs end to end on new hardware do these problems surface and get fixed.

    For this reason, work such as “the first multimodal model trained end to end on homegrown accelerators” is essentially a stress test of the robustness of the entire technology stack, not just a model release.

    2. An End-to-End Pipeline: More Than a Single Forward Pass

    A common misconception is to equate “the model can do inference on new hardware” with “the model can be trained on it.” What is genuinely difficult is a complete closed-loop pipeline: data preprocessing → large-scale pretraining → supervised fine-tuning (SFT) → reinforcement learning alignment, with all four stages running on the same hardware and framework.

    • Data preprocessing: Cleaning and deduplicating image-text pairs, resolution normalization, image-text alignment verification, and preparing target text annotations for text rendering. This stage determines what the model “has seen” and sets the ceiling for downstream capability.
    • Pretraining: Learning world knowledge and visual-language correspondence from massive image-text corpora, building foundational generation ability.
    • SFT (supervised fine-tuning): Aligning the model to task formats with high-quality, well-structured instruction data so it learns to “understand” complex natural language descriptions.
    • Reinforcement learning alignment: Using preference data or reward signals (such as DPO-style methods or reward models) to improve visual aesthetics, instruction following, and text accuracy.

    The value of running the entire pipeline on a homogeneous setup is avoiding precision drift and engineering fragmentation. If pretraining runs on hardware A and fine-tuning on hardware B, differences in numerical precision, operator implementations, and random number generation can quietly introduce bias, and troubleshooting such issues is extremely costly.

    3. Core Architecture: A Hybrid of Autoregressive and Diffusion Decoder

    The key innovation of this class of models is combining two paradigms, each handling its own part of the job:

    • Autoregressive backbone (language-model-style structure): Handles semantic understanding, global composition, and prompt parsing, converting a natural language description into a structured semantic representation. It is good at “reading” — understanding intent, constraints, and scene layout.
    • Diffusion decoder (DiT-style structure): Takes the semantic representation as a condition and performs pixel-level denoising and generation. It is good at “drawing well” — turning abstract semantics into high-quality, detail-rich images.

    Why combine them? Pure diffusion models tend to be weak at semantic understanding of complex, long prompts and often “miss the point”; pure autoregressive per-token image generation is inefficient and limited in image quality. The hybrid architecture injects the reasoning capability of language models into image generation, playing to the strengths of each. This idea is also consistent with the industry direction of combining language model reasoning with visual generation, sometimes described as “cognitive generation.”

    An illustrative inference flow is shown below (for structural explanation only, not a specific API):

    1. # Illustrative: autoregressive semantic understanding + conditioned generation via a diffusion decoder
    2. semantic = ar_model.encode(prompt) # Backbone parses semantics and composition
    3. text_cond = glyph_encoder.encode(text) # Strong condition for text rendering (glyph/text)
    4. image = diffusion_decoder.sample( # Pixel-level denoising generation
    5. condition=semantic,
    6. extra_condition=text_cond,
    7. steps=N,
    8. resolution=(H, W), # Arbitrary resolution
    9. )

    4. Chinese Text Rendering: Solving the “Forgot the Character” Problem

    Text rendering remains one of the hardest problems in multimodal image generation, and Chinese is especially challenging. The reasons: the Chinese character set is huge, glyph shapes are complex, and strokes are dense. During pixel-level denoising, diffusion models easily blur characters into illegible blobs or write incorrect strokes — a problem commonly described as “forgetting the character while writing it.”

    The solution is to decouple “understanding” from “writing correctly”: the semantic backbone handles comprehension, while a dedicated glyph/text encoder injects the target text as a strong condition into the decoder, constraining the model to render the specified characters during generation. This capability is essential for scenarios where the image must contain accurate Chinese, such as marketing posters, social media covers, and information cards — if the text is wrong, the entire asset is essentially unusable.

    5. Non-Standard Resolutions and Arbitrary Aspect Ratios

    Traditional image generation models are typically bound to fixed resolutions and token grids (preset ratios such as 4:3 or 16:9). This is awkward in real business scenarios: posters, covers, and feed creatives come in all kinds of sizes, and generating at a standard size and cropping afterward often damages the composition or truncates text.

    The improvement direction is native support for arbitrary aspect ratios: by improving the tokenizer and positional encoding strategy, the model can accept non-standard resolutions within a size range (for example, from about 1024×1024 to 2048×2048) and output images directly at ratios such as 1240×1600. The engineering challenges mainly lie in three points:

    • Variable resolution requires positional encodings that can extrapolate and handle different token counts;
    • Memory usage and generation throughput vary non-linearly with resolution, requiring resource scheduling;
    • The aspect ratio distribution during training and inference must match; otherwise, an illusion arises in which the model appears to perform well only on standard ratios.

    6. Training Engineering Optimizations: Making Large Models Actually Trainable

    Completing large model training on hardware with relatively limited compute and a less mature ecosystem puts far more weight on engineering optimization than mature platforms do. Work can start at three levels:

    • Reducing dispatch bottlenecks: Adopt multi-level pipelined dispatch for dynamic graphs to ease the overhead of dispatching operators from host to device, keeping device-side utilization high.
    • Communication-computation overlap: Through multi-stream parallelism, let communication streams and computation streams overlap, hiding synchronization waits inside computation.
    • Fused operators: Use hardware-friendly fused operators such as AdamW + EMA, RMSNorm, normalization, and attention to reduce memory accesses and kernel launch overhead.

    These optimizations are especially critical on new hardware: the less mature the ecosystem, the more likely there are gaps in its operator and communication libraries, and the more the framework layer needs to “fill in” performance through fusion, pipelining, and overlap.

    7. Cloud Deployment and Engineering Considerations

    Bringing model training and inference into production relies on a complete set of supporting infrastructure. From a general capability perspective, this typically involves:

    • Object storage: Storing training data, checkpoints, and model weights;
    • Container platform: Elastic scheduling to support distributed training and inference services;
    • Message queue: Connecting data pipelines and asynchronous tasks;
    • Logging services and monitoring/alerting: Tracking training stability metrics such as loss, throughput, and memory usage to catch problems early.

    For example, the object storage, container platform, and monitoring/alerting capabilities offered by Baidu AI Cloud can provide baseline support for training scheduling and online inference of multimodal models. When choosing a platform, the focus should be on hardware backend compatibility, distributed training support, and observability, rather than on a single feature point.

    8. Summary and Outlook

    Overall, four transferable lessons emerge from training multimodal image generation models on homegrown accelerators:

    1. Hardware diversity is a trend. The ability to train end to end across multiple backends is a litmus test for the maturity of a technology stack.
    2. Hybrid architecture (autoregressive + diffusion) is becoming a mainstream paradigm that balances semantic understanding and image quality.
    3. Chinese text rendering and arbitrary resolution are key differentiators that determine whether a product can ship.
    4. Engineering optimization (pipelined dispatch, communication-compute overlap, operator fusion) determines whether a model can actually be trained with limited compute.

    These capabilities are not exclusive to any single vendor; they form a methodology that can be reused across hardware and frameworks. Understanding the trade-off logic behind them is more valuable than memorizing any specific product.