Training Multimodal Text-to-Image Models on Domestic AI Chips: A Full-Pipeline Technical Practice
Competition in image generation has shifted from simply producing pictures to accurately understanding instructions, rendering complex layouts, and generating legible text—especially in scenarios like posters, infographics, and marketing pages. This article examines a complete multimodal image generation training pipeline executed on domestic (Chinese) AI chip infrastructure, covering the hybrid autoregressive-plus-diffusion architecture, the challenges of Chinese text rendering, arbitrary-resolution generation, engineering optimizations for distributed training, the four-stage training workflow, evaluation methodology, and practical recommendations for developers building similar capabilities on non-NVIDIA hardware stacks.
In recent years, the competitive focus of image generation models has gradually shifted from “whether an image can be generated at all” to “whether the model can accurately understand requirements, reliably produce complex compositions, and correctly render text within images.” In scenarios such as Chinese-language posters, long-form infographics, product promotion pages, and social media covers, text must be not only semantically correct but also consistent with typeface structure, layout position, font size hierarchy, and overall visual style. These tasks place much higher demands on a model’s language understanding, spatial planning, and image decoding capabilities.
Against this backdrop, a multimodal image generation training pipeline completed in a domestic chip environment offers strong engineering reference value. Its significance lies not merely in releasing an image generation model, but in validating that the full workflow—from data preprocessing and large-scale pretraining to supervised fine-tuning and reinforcement-based optimization—can form a closed loop on domestic computing infrastructure.
1. Why Full-Pipeline Training on Domestic Chips Matters
Historically, model training has depended on specific hardware ecosystems, with tight coupling among training frameworks, operator libraries, communication components, and scheduling mechanisms. For large multimodal models, inference adaptation alone is not difficult; what is genuinely complex is the stability and efficiency of the training stage.
A complete training pipeline typically includes:
- Raw data cleaning and format conversion
- Image-text pairing and quality filtering
- Encoding of text, images, and layout information
- Large-scale pretraining
- Instruction fine-tuning
- Preference optimization or reinforcement training
- Evaluation, regression testing, and model release
Any of these stages can be constrained by hardware operator support, memory capacity, communication efficiency, and framework maturity. Therefore, achieving end-to-end training on domestic chips demonstrates that the relevant software-hardware co-design covers a fairly complete engineering chain, and provides valuable experience for subsequent model migration and scaled deployment.
2. A Hybrid Architecture: Autoregressive Modeling Plus Diffusion Decoding
This class of model combines “autoregressive semantic modeling with diffusion-based image decoding,” splitting a complex image generation task into two collaborating stages.
The first stage handles understanding. Autoregressive models excel at processing long text instructions, contextual relationships, object attributes, and scene layouts. For example, when a user requests a Chinese poster containing a headline, a subtitle, multiple visual elements, and a specific style, the model must first determine the semantic relationships among these elements, then plan their positions and hierarchy within the composition.
The second stage handles generation. The diffusion decoder progressively produces a high-quality image based on the semantic representation formed in the previous stage. It is better suited to handling texture, lighting, edges, fine details, and overall style, giving the final output stronger visual quality.
This division of labor can be summarized as:
- The autoregressive module is responsible for “understanding the request” and “planning the composition”
- The diffusion module is responsible for “rendering visual details” and “producing the final image”
A traditional single architecture must simultaneously handle both language reasoning and pixel generation, which often creates tension between semantic understanding and image detail. The hybrid architecture decouples these modules so that each component plays to its strengths.
3. Why Chinese Text Rendering Is a Hard Problem
Text generation has long been a weak point for image models. The reason is that text is not ordinary texture—it is a visual symbol system with strict discrete structure. If a character is missing a stroke, gains an extra stroke, or has its components shifted, the meaning can change entirely.
English uses a relatively small character set, whereas Chinese involves a vast number of glyph shapes, radicals, and complex structures. The model must do more than generate text-like textures; it must ensure:
- Character content matches the prompt
- Glyph structures are complete and correct
- Text direction and ordering are correct
- Line spacing and character spacing are reasonable
- Headlines, body text, and annotations form a clear hierarchy
- Text maintains sufficient contrast against the background
To address this, the model can introduce a dedicated text encoding module in the generation pipeline, converting text content into structured representations that are better suited for visual decoding. At the same time, training data must cover scenarios such as posters, tables, menus, information cards, and long-text images, with dedicated evaluation of text recognition accuracy, layout consistency, and local sharpness.
The core of this optimization is not simply adding more training images, but teaching the model “what text is, where it should be placed, and how it should harmonize with the overall visual style.”
4. The Practical Value of Non-Standard Resolutions
Many image generation systems support only a few fixed sizes—square, landscape widescreen, or portrait ratios. But real-world design dimensions are rarely so regular. E-commerce detail pages, mobile covers, event posters, flyers, and long-form infographics all use varying aspect ratios.
If a model can only output fixed sizes, users must first generate an image and then adapt it through cropping, padding, or stretching. These post-processing steps commonly cause:
- Key text being cropped out
- The main subject shifting away from its intended position
- Distorted aspect ratios
- Whitespace areas that fail design requirements
- Detail loss from repeated rescaling
Supporting arbitrary aspect ratios must be designed in from the image encoding stage. Common approaches include improving the image tokenization strategy, adopting dynamic grid representations, and making positional encodings adaptable to different width and height dimensions. Training must also cover a variety of ratios and layouts so the model does not merely memorize a few fixed canvases.
Note that “supporting non-standard resolutions” does not mean all sizes achieve identical quality. Actual output is still affected by memory capacity, sampling steps, text length, image complexity, and the training distribution. In production, therefore, teams should establish a size whitelist, quality checks, and retry mechanisms for failed generations.
5. Engineering Optimizations for Training on Domestic Chips
Completing large-scale training requires more than peak hardware compute—it demands coordinated optimization of data flow, computation flow, and communication flow.
First, data preprocessing. Image-text data typically requires cleaning, deduplication, resolution filtering, text quality checks, and label organization. Running all preprocessing on the training host can leave compute devices idle waiting for data. A better approach is a parallel data pipeline that completes decoding, augmentation, and batch assembly in advance.
Second, pipeline scheduling. Dynamic graph execution can introduce frequent dispatch and synchronization overhead. Multi-stage pipelining, operator aggregation, and cache reuse can reduce the number of task submissions. These optimizations are especially important for long text sequences and high-resolution images.
Third, communication-computation overlap. In distributed training, if gradient synchronization, parameter broadcast, and compute tasks run strictly serially, device utilization drops significantly. Multi-stream parallelism allows some communication operations to proceed concurrently with forward and backward computation, reducing wait time.
At the operator level, fusion tailored to hardware characteristics is also necessary. Optimizer updates, normalization, exponential moving averages, and certain tensor transformations can all be fused to reduce intermediate tensor reads and writes. For large model training, memory access is often as critical as raw compute.
6. Breaking Down the Full Training Workflow
A fairly complete training workflow can be divided into four stages.
Stage one: data preparation. Establish a unified data format that associates images, text prompts, layout information, and quality labels. For images containing text, OCR recognition results should be added for later supervision and evaluation.
Stage two: pretraining. The model learns general visual concepts, linguistic semantics, and cross-modal correspondences from large-scale image-text data. The priorities in this stage are stability, throughput, and data coverage.
Stage three: supervised fine-tuning. Using high-quality instruction data, the model learns concrete tasks—for example, creating a poster from a description, editing localized text, adjusting aspect ratio, or maintaining subject consistency. The accuracy of fine-tuning data typically matters more than its quantity.
Stage four: preference optimization. Based on human annotations or automated evaluation results, joint optimization can be applied to text correctness, layout reasonableness, visual aesthetics, and instruction following. However, preference metrics must be carefully calibrated to prevent the model from chasing visual appeal at the expense of text accuracy or content fidelity.
7. Evaluation Cannot Rely on Aesthetics Alone
For general image generation, visual quality is an important metric, but for Chinese posters and complex layouts, a more granular evaluation system is needed.
At minimum, coverage should include:
- Text recognition accuracy
- Long-text completeness
- Font and glyph readability
- Layout consistency with the prompt
- Subject count and attribute correctness
- Compositional stability at non-standard resolutions
- Detail sharpness and visual coherence
Automated evaluation can use OCR, image-text similarity scoring, and object detection, but these cannot fully replace human review. For commercial design scenarios, safety review, copyright risk screening, and sensitive content filtering should also be added.
8. Practical Recommendations for Developers
Enterprises looking to build similar capabilities should start with small-scale validation rather than pursuing an oversized model from day one. A recommended path:
- Choose a well-defined scenario first, such as Chinese promotional graphics or information cards
- Build a high-quality, traceable image-text dataset
- Validate text understanding, layout planning, and image generation capabilities separately
- Perform operator coverage analysis and performance profiling on the target domestic chips
- Prioritize optimizing the data pipeline, communication efficiency, and memory usage
- Use targeted evaluation to continuously identify text errors and layout deviations
- Only then scale up the model size and business scope
Overall, the value of training multimodal image generation models on domestic chips lies in establishing a replicable technical route: a language model handles cognition and planning, a diffusion module handles visual rendering, a dedicated text encoding mechanism improves Chinese text rendering quality, and software-hardware co-optimization supports large-scale training. For developers, what is truly worth borrowing is not any single model, but the complete engineering methodology spanning data, architecture, training, and evaluation.
