Training Multimodal Image Generation Models End-to-End on Domestic AI Accelerators

    This article explains the technical methodology for training multimodal image-generation models end to end on domestic AI accelerators. It covers the full pipeline—data preprocessing, pretraining, SFT, and RL—and examines framework adaptation, operator alignment, numerical stability, distributed parallelism, communication-compute overlap, and fused operators. It also discusses a hybrid autoregressive-plus-diffusion architecture, Chinese text rendering, arbitrary-resolution generation, and deployment options such as APIs, private deployment, and fine-tuning.

    1. Background: The Signal Toward More Autonomous Multimodal Generation

    The capability ceiling of image-generation models has long been determined by both compute supply and the maturity of training frameworks. Multimodal pretraining has historically concentrated on a small number of imported accelerators and their surrounding frameworks, creating a situation where models can be open-sourced, but compute remains difficult to control independently.

    A recent development in the domestic AI compute ecosystem is worth attention: an autoregressive multimodal image-generation model completed end-to-end training, from data preprocessing to large-scale pretraining, entirely on domestic accelerator clusters. Its weights and analysis materials were also open-sourced.

    The significance is not that a single run succeeded, but that it validated the proposition that domestic compute can support the complete training pipeline of a frontier multimodal model. Rather than evaluating a specific product, this article extracts a reusable technical methodology for teams working on multimodal R&D and compute deployment.

    2. Full-Pipeline Training on Domestic Compute: Challenges and Solutions

    When training large models on domestic chips, the biggest challenge is often not whether computation can be performed, but whether every stage of the training pipeline has been adapted. A complete training pipeline typically includes four stages: data preprocessing, large-scale pretraining, supervised fine-tuning (SFT), and reinforcement learning based on human feedback or rules. If any stage has missing operators, numerical deviations, or communication bottlenecks, the entire pipeline may struggle to converge stably. Full-pipeline enablement on domestic compute essentially means migrating and optimizing all four stages.

    2.1 Unified Training Suite and Framework Adaptation

    A practical engineering approach is to build a unified training suite that covers the full pipeline and to rely on a deep learning framework with automatic parallelism and graph operators for migration. Frameworks with native support for domestic hardware backends can reduce the need for large amounts of handwritten operator-bridging code. During migration, three types of problems usually need to be addressed:

    • Operator alignment: Map mainstream model components, including attention, normalization, activation, and loss functions, to equivalent operators that have been heavily optimized on the hardware backend.
    • Numerical consistency: Verify the numerical stability of forward and backward passes under mixed precision to avoid gradient explosion or NaNs.
    • Distributed parallelism: Make appropriate partitions across tensor parallelism, pipeline parallelism, and data parallelism so that single-device memory and cluster bandwidth are both fully utilized.

    2.2 Communication-Compute Overlap: Dynamic Pipelining and Multi-Stream Parallelism

    In large-scale training, communication often becomes the throughput bottleneck. Two classic optimizations are particularly important:

    • Dynamic multi-stage pipeline dispatch optimization: The execution graph during training can be split into multiple pipeline stages and dispatched sequentially. By reducing CPU-to-device dispatch waits, devices can stay as saturated as possible and pipeline bubbles can be minimized.
    • Multi-stream communication-compute overlap: Forward, backward, and gradient communication are assigned to different execution streams so that compute-intensive operators and communication-intensive operators overlap along the time axis. In the ideal case, communication time is hidden by computation, and end-to-end throughput increases significantly.

    The following illustrative pseudocode shows how multi-stream parallelism may be organized:

    1. # Illustrative: orchestration of communication-compute overlap (not real interfaces)
    2. compute_stream = device.Stream()
    3. comm_stream = device.Stream()
    4. for micro_batch in micro_batches:
    5. with compute_stream:
    6. loss = forward_backward(micro_batch) # Compute stream handles forward + backward
    7. with comm_stream:
    8. all_reduce_gradients() # Communication stream handles gradient synchronization
    9. # The two streams overlap in time; communication is hidden by computation
    10. compute_stream.synchronize()
    11. comm_stream.synchronize()

    2.3 Fused Operators for Domestic Chips

    Operator fusion is often the last mile of performance tuning. Combining multiple fine-grained operators into a single fused operator can reduce intermediate read/write traffic to memory and lower kernel launch overhead. Common fusion directions in practice include:

    • Optimizer fusion: Fuse AdamW’s momentum update and weight decay into a single operator to reduce element-wise memory access.
    • Exponential moving average (EMA): Integrate parameter smoothing into the training loop, obtaining smoother weights with low additional overhead and potentially improving generation quality.
    • Normalization and loss fusion: Fuse operators such as RMS Norm with surrounding computations to reduce intermediate tensors.

    A common prerequisite for these fused operators is hardware affinity: the operator implementation must match the memory hierarchy and compute-unit characteristics of the target chip. Only then can theoretical gains translate into measured throughput improvements.

    3. Hybrid Architecture: Autoregressive Understanding Plus Diffusion Rendering

    High-quality, controllable image generation, especially generation that includes text, is shifting from a single diffusion model toward a hybrid architecture. One approach that has been repeatedly validated is the combination of an autoregressive model and a diffusion decoder:

    • Autoregressive foundation (semantics and layout layer): A billion-scale autoregressive language model, such as a 9B model, serves as the foundation. It handles semantic understanding and global composition planning. It first translates text and visual intent into a structured intermediate representation, deciding what should appear in the image and how elements should be arranged.
    • Diffusion decoder (pixels and text layer): A DiT-based diffusion decoder, such as a 7B model, reconstructs the intermediate representation into high-quality pixels. A dedicated glyph encoder can also be used to handle glyph rendering, addressing issues such as missing strokes or garbled characters when generated text is involved.

    The intuition behind this division of labor is that the former is good at understanding requirements and planning the global layout, while the latter is good at precise reconstruction and rendering details. Connected in sequence, they combine the reasoning and planning ability of a language model with the pixel-reconstruction ability of a diffusion model. In industry, this idea is sometimes described as cognitive generation: the model first understands, then generates, rather than trying to produce results directly from noise.

    This direction is also consistent with the broader trend of introducing language-model reasoning into image generation: the model first “thinks clearly” in semantic space and then “paints” in pixel space, improving controllability in complex composition and mixed text-image scenarios.

    4. Text Rendering: A Key Requirement for Direct Chinese Poster Generation

    Text rendering is one of the hardest problems in multimodal image generation, and Chinese is especially difficult. Latin alphabets have a limited set of glyphs, while Chinese has a large number of commonly used characters and complex stroke structures, placing high demands on a model’s glyph priors. To achieve readable, direct Chinese output in scenarios such as posters and covers, three elements typically need to work together:

    1. Glyph encoder: A dedicated encoding path for text explicitly injects character-structure information into the decoder, preventing the model from learning only a vague prior that “text looks like text.”
    2. Long-text consistency: Long-text rendering evaluates whether the model can keep every character in a continuous passage correct, unshifted, and non-repetitive.
    3. Complex visual text generation: In dense text-image layouts, the model must produce natural layering, occlusion, and perspective relationships between text and background.

    In evaluation, the industry often uses two types of benchmarks: complex visual text generation and long-text rendering. Leading performance on these benchmarks in open-source models usually indicates that the text-rendering pipeline has become relatively mature. For developers, these capabilities directly determine whether a model can support production scenarios that require both images and text, such as poster design, social media covers, and infographics.

    5. Arbitrary Resolution: Native Support for Non-Standard Aspect Ratios

    Traditional image-generation models often output fixed preset aspect ratios, such as 4:3 or 16:9. When a non-standard size is required, for example a custom format such as 1240×1600, the only option may be cropping or stretching as post-processing, which can degrade composition and sharpness.

    By improving tokenization strategies and positional encoding schemes, a new generation of models can natively generate arbitrary aspect ratios over a relatively wide range, for example from 1024×1024 to 2048×2048. The engineering value includes:

    • Posters and portrait covers: Design work often requires non-integer aspect ratios, and native support avoids post-processing.
    • Multi-device adaptation: The same generation capability can produce assets for screens with different aspect ratios, improving batch-production efficiency.
    • Composition fidelity: Because the model sees diverse aspect-ratio distributions during training, it is less likely to produce broken layouts when extrapolating to new ratios.

    To implement arbitrary resolution, the tokenizer’s grid partitioning and the positional encoding’s extrapolation ability place higher demands on the system. The training data must also include sufficiently diverse resolutions and aspect ratios so that the model remains stable on previously unseen proportions.

    6. Cost, Open Source, and Engineering Deployment

    For enterprises and developers that want to validate quickly, three practical paths are worth comparing:

    • API access: Integrate rapidly through low per-image pricing, which is suitable for small- to medium-scale businesses that prioritize time to market.
    • Private deployment: Deploy open weights to self-owned or cloud-based container platforms, and combine them with common cloud capabilities such as object storage, message queues, logging, and monitoring alerts to build a controllable generation service.
    • Secondary fine-tuning: Use open foundations and fine-tune with proprietary data through SFT or LoRA to produce styles that fit brand tone.

    For cloud deployment, the generation service can be divided into a four-layer architecture: an access layer for authentication and rate limiting, a compute layer for inference containers with elastic scaling, a storage layer for object storage of result images, and an observability layer for logs and metrics monitoring. This helps balance elasticity, cost, and stability. These are general engineering patterns in the industry, not proprietary solutions from a specific vendor.

    7. Conclusion

    Focusing on the end-to-end training of multimodal image-generation models on domestic AI compute, four core conclusions can be drawn:

    1. The full pipeline can be made runnable: Domestic chips can support the complete pipeline from data preprocessing and pretraining to SFT and RL, provided that frameworks, operators, and parallelism strategies have been properly adapted.
    2. Hybrid architectures can be more effective: A division of labor in which an autoregressive foundation handles semantic planning and a diffusion decoder handles pixel and text rendering is a mainstream engineering form of cognitive generation.
    3. Text rendering is a differentiator: Direct Chinese long-text and complex visual-text generation capabilities directly determine usability in production scenarios such as posters and covers.
    4. Arbitrary resolution and open-source delivery reduce adoption costs: Native support for non-standard aspect ratios, combined with open-source and API-based delivery options, significantly lowers the barrier to deploying multimodal generation capabilities.

    For engineering teams, the real opportunity lies in turning these capabilities into reusable training and inference pipelines, fine-tuning them with proprietary data, and building stable, economical, and continuously evolvable multimodal generation services on top of domestic compute that is increasingly under self-control.