Full-Pipeline Training of Multimodal Image-Generation Models on Homegrown AI Accelerators: Architecture and Engineering Practices

    This article distills the technical practices for training a multimodal image-generation model from data preprocessing through pretraining, supervised fine-tuning, and reinforcement learning on homegrown AI accelerators. It examines why a hybrid autoregressive-plus-diffusion architecture suits image generation, how a unified training suite and system-level optimizations address dispatch, communication, operator, and reliability bottlenecks, and how tokenizer design supports non-standard image sizes and aspect ratios. It also covers Chinese text rendering, evaluation considerations, deployment, and cost planning.

    1. Background: The Compute Challenge of Multimodal Image Generation

    Multimodal image-generation models are moving beyond simply producing an image. They are expected to interpret complex semantics, render text correctly, and support real business image dimensions. This evolution places greater demands on training engineering. Parameter counts continue to rise; training data has expanded from image-text pairs to complex visual corpora with fine-grained text annotations; and the training pipeline now spans data preprocessing, large-scale pretraining, supervised fine-tuning, and reinforcement learning. Historically, this kind of end-to-end training was assumed to depend on a certain class of high-end accelerator cards and their accompanying software stack. As homegrown accelerator ecosystems mature, a practical engineering question has emerged: can the entire pipeline—from data cleaning to final alignment—run on homegrown chips?

    Teams have recently disclosed practices for completing full-process training of a multimodal image-generation model on clusters of homegrown accelerators and releasing the results as open source. This article does not focus on a specific product. Instead, it abstracts the practice into a general technical approach and examines three questions: why a hybrid architecture is a good fit for image-generation tasks; which engineering bottlenecks must be solved to run full-pipeline training on homegrown compute; and what technical mechanisms underlie non-standard resolution support and Chinese text rendering.

    2. Hybrid Architecture: Separate Semantic Understanding from Accurate Text Rendering

    2.1 Why Not Use a Single Diffusion Model?

    Traditional text-to-image models mostly use a diffusion architecture. A text encoder supplies conditional input, and a diffusion decoder gradually denoises to generate an image. This approach produces mature visual quality, but it often struggles with complex semantic understanding, long-text rendering, and precise spatial layout. The text encoder has limited capacity to compress long instructions semantically, and the diffusion process lacks an explicit global planning stage.

    2.2 Autoregression for Semantics and Composition; Diffusion for Pixel Refinement

    The practice uses a hybrid “autoregressive + diffusion decoder” architecture, splitting generation into two stages:

    • Autoregressive component: Handles semantic understanding and global composition. It unifies text instructions and visual tokens in a sequence-modeling framework, first determining what should appear in the image, roughly where it should be placed, and what the text content should be. This component has about 9B parameters and is trained entirely on homegrown accelerators.
    • Diffusion decoder component: Handles pixel-level refinement and text rendering. It receives the semantic plan produced by the autoregressive stage and, together with a text-encoding module, maps layout and glyph information into pixel space. This component uses a DiT structure with about 7B parameters.

    The engineering value of this division is that the autoregressive stage can use established sequence-modeling optimization techniques, while the diffusion stage concentrates on local detail generation. Because the two are decoupled, they can be tuned separately. Technically, combining a language model’s reasoning and planning abilities with image generation has become a common direction in the industry’s exploration of “cognitive generation.”

    2.3 Specialized Design for Text Rendering

    Chinese text rendering is one of the hardest problems for image-generation models. Chinese characters have complex glyphs and dense strokes, and poster scenarios require accurate text placement, appropriate font sizes, and no missing or broken strokes. This approach introduces a text-encoding module on the diffusion-decoder side. It injects both glyph features and semantic features of the text to be rendered into the decoding process, effectively providing an additional “glyph reference” during denoising. Unlike approaches that rely solely on implicit alignment from a text encoder, this method gives the model explicit structural constraints on text during pixel generation.

    3. Full-Pipeline Training on Homegrown Compute: Engineering Bottlenecks and Optimizations

    “Running the entire process on homegrown accelerators” is not a slogan. It means that all four stages—data preprocessing, pretraining, SFT, and RL—must run stably on the homegrown software stack. The following sections break down the key engineering optimizations.

    3.1 A Unified Training Suite for the End-to-End Workflow

    On top of a homegrown deep-learning framework, a self-developed training suite is needed to smooth over engineering differences between stages. A complete training suite typically covers:

    • Data preprocessing: image-text pair cleaning, resolution normalization, parsing of text-region annotations, tokenization, and packing.
    • Large-scale pretraining: joint or staged training of the autoregressive base model and the diffusion decoder.
    • Supervised fine-tuning: alignment on high-quality data for scenarios such as posters, illustrations, and covers.
    • Reinforcement learning: generation-quality optimization based on reward models or rule-based constraints.

    The purpose of a suite is to consolidate shared capabilities such as distributed strategies, mixed precision, checkpoint resumption, and logging and monitoring, so that each stage does not have to reinvent the same infrastructure.

    3.2 Dispatch Bottlenecks and Communication-Compute Overlap

    In large-scale training, host-to-device dispatch efficiency often becomes a hidden bottleneck. This practice uses dynamic-graph multi-level pipeline dispatch to eliminate dispatch blocking, allowing data movement and computation to form a pipeline. It also uses multi-stream parallelism so that communication operations and compute operations can overlap. For long-sequence training of autoregressive models, this optimization is especially important for maintaining accelerator utilization.

    3.3 Operator-Level Affinity Optimizations

    Getting performance from homegrown accelerators requires operator-level adaptation. Common techniques include:

    • Fusing elementwise operations in the AdamW optimizer with EMA updates to reduce memory accesses.
    • Using fused operators for normalization layers to reduce kernel launch overhead.
    • Applying custom layouts for the contiguous memory-access patterns in attention computation.

    These optimizations do not change the mathematical equivalence of the model, but they can significantly improve training throughput. Operator fusion must be checked for numerical consistency to avoid precision differences that affect convergence.

    3.4 Stability in Distributed Training

    Full-pipeline training has a long cycle, so fault recovery matters more than single-step performance. In practice, the system needs asynchronous checkpoint saving, automatic rescheduling of failed tasks, data loaders that support resumable reading, and separate storage for training state and logs. For the RL stage, it is also necessary to manage the pipeline handoff between generated samples and training samples so that waiting on the generation side does not slow down the training side.

    4. Non-Standard Resolutions: From Preset Ratios to Arbitrary Ratios

    4.1 Limitations of Conventional Approaches

    Most image-generation models are trained at preset ratios such as 4:3 and 16:9, and inference supports only a limited set of sizes. Posters, e-commerce hero images, and social-media covers often require non-standard dimensions such as 1240×1600. Directly resizing such images can break the composition, while cropping can lose content.

    4.2 Tokenizer Strategy Redesign

    This approach modifies the tokenizer strategy so that the model natively supports image generation across a resolution range from 1024×1024 to 2048×2048 with arbitrary aspect ratios. The core idea is not simple interpolation but decoupling image tokenization from resolution. The image is divided into a variable-size token grid, so sequence length adapts to the aspect ratio. At the same time, training data mixes samples with multiple ratios so that the model sees enough non-standard layouts.

    4.3 Value for Business Scenarios

    Native support for non-standard resolutions directly reduces post-processing costs. In poster generation, for example, designers do not need to generate a square image first and then manually expand it; the model can generate directly at the target size, and text regions and the main composition are planned at the final ratio during generation. This is especially useful for batch generation of social-media assets.

    5. Text Rendering Capability and Evaluation Perspective

    Chinese text rendering evaluation usually focuses on two areas: complex visual text generation and long-text rendering. The former tests whether a model can render text correctly under complex backgrounds, multiple fonts, and multiple font sizes. The latter tests whether long passages contain missing characters, misalignment, or repetition. This approach performs strongly among open-source models on these two types of evaluation, indicating that the text-encoding module in the hybrid architecture and the autoregressive layout planning work effectively together.

    From an engineering perspective, there is no single silver bullet for improving text rendering. It generally requires three coordinated elements: training data with enough text-annotated samples; a text-encoding module that provides explicit glyph constraints; and reproducible automated metrics in evaluation, rather than relying only on subjective human judgment.

    6. Deployment and Cost Considerations

    The model is released as open source and also offers an online inference interface with per-image billing; the cost per image is relatively low. For developers, open source means local deployment and fine-tuning on demand. For business users, per-image billing suits scenarios with highly variable traffic. One caveat is that actual deployment must match accelerator memory with the inference batch size. At 2048×2048 resolution, the diffusion decoder’s memory footprint is significantly higher than at 1024×1024, so resources should be planned according to the peak size required by the business.

    7. Conclusion

    The hard part of full-pipeline training for multimodal image-generation models on homegrown compute is not replacing individual operators. It is coordinating architecture design, training engineering, data strategy, and evaluation. The hybrid architecture separates semantic planning from pixel generation, providing engineering handles for text rendering and non-standard resolutions. The training suite, pipeline dispatch, multi-stream parallelism, and fused operators address throughput and stability on homegrown accelerators. For teams preparing to conduct multimodal training on homegrown compute, a practical path is to start by validating feasibility through data preprocessing and tokenizer strategy, then gradually extend to pretraining and alignment. Migrating the entire pipeline at once increases debugging pressure and should be avoided.