Domestic Chip-Based Training for Multimodal Image Generation Models: Technical Architecture and Engineering Practice
This article examines an end-to-end approach to training multimodal image generation models on domestic AI chips. It explains the hybrid architecture that combines autoregressive language understanding with diffusion-based image generation, the use of glyph encoding for more accurate Chinese text rendering, support for non-standard image sizes, and system-level optimizations across data processing, communication, operators, and memory access. It also outlines a complete workflow from dataset construction and pretraining to fine-tuning, quality optimization, and production deployment.
In recent years, competition in image generation has shifted from simply producing images to understanding user requirements accurately, rendering complex layouts consistently, and generating visual content that can be used directly.
This is especially important for commercial posters, infographics, and social media covers. Users expect not only an attractive image, but also accurate titles, descriptions, button text, and layout structures.
Against this backdrop, multimodal image generation models trained entirely on domestic AI chips have attracted attention. Their value lies not only in image quality, but also in connecting data preprocessing, large-scale pretraining, supervised fine-tuning, and reinforcement-based optimization. This provides a practical reference for building an autonomous and controllable generative AI training system.
1. Why End-to-End Training on Domestic Chips Matters
Training a large multimodal model usually involves several stages. Data cleaning requires large-scale file access, image-text pairing, and quality filtering. Pretraining involves high-throughput matrix computation and cross-device communication. Supervised fine-tuning processes large volumes of structured instructions, while reinforcement-based optimization repeatedly performs inference, scoring, and parameter updates.
These stages place different demands on the computing platform:
- Data processing depends more on storage access, data orchestration, and task scheduling.
- Pretraining depends more on computational throughput, memory capacity, and communication efficiency.
- Fine-tuning places greater emphasis on training stability and resource utilization.
- Reinforcement-based optimization requires efficient switching between inference and training.
If only some stages run on domestic chips, critical parts of the workflow may still depend on other computing platforms. Achieving end-to-end compatibility—from data preprocessing and large-scale pretraining to fine-tuning and optimization—requires coordinated changes to the training framework, operator implementations, communication mechanisms, scheduling strategies, and engineering toolchain.
The broader industry significance is that model capability is no longer determined by algorithms alone. Hardware adaptation, software-stack maturity, and engineering execution have also become core competitive factors.
2. A Hybrid Architecture: Language Understanding for Planning and Diffusion for Rendering
This type of model uses a hybrid architecture that combines an autoregressive model with a diffusion decoder. The two subsystems have clearly defined responsibilities.
2.1 The Autoregressive Module: Understanding Requirements and Planning the Layout
The autoregressive module is mainly responsible for text understanding, semantic reasoning, and high-level composition planning.
For example, when a user requests a vertical promotional poster containing a main title, subtitle, product highlights, and footer text, the model needs to identify:
- The canvas orientation and approximate aspect ratio
- The hierarchy between the primary and secondary titles
- The text content and relative importance of each element
- The subject, background style, and preferred color direction
- The spatial relationship between text and graphic elements
This is not simple keyword matching. The model must convert a natural-language request into an executable visual layout plan. It first needs to understand what should be drawn and then determine where each element should be placed.
2.2 The Diffusion Decoder: Generating Details and Rendering the Image
The diffusion decoder converts high-level semantic and layout information into a pixel-level image. It generates the image through a gradual denoising process and is responsible for details such as texture, lighting, edges, and local structures.
For ordinary images, diffusion models can often generate landscapes, people, and objects effectively. Text rendering is more difficult because characters have strict structural requirements. A single incorrect stroke, distorted glyph, or reordered character can make the text unreadable.
As a result, relying solely on the visual diffusion process can produce images that appear to contain text but cannot actually be read.
2.3 Glyph Encoding: Improving Chinese Text Generation
To make text generation more stable, the model can introduce a dedicated glyph encoding module. This module encodes character structure, stroke relationships, and text-position information before providing the resulting conditions to the decoder.
The general process can be summarized as follows:
- Identify the text that the user wants to generate.
- Convert the text into a structured character representation.
- Extract glyph and spatial-layout features.
- Inject text conditions throughout the diffusion process.
- Reduce character errors, omissions, and distortions through training and post-processing.
This design gives the model both the ability to understand text and the ability to render it visually. Language understanding handles the former, while visual decoding and glyph constraints handle the latter. Their combination can significantly improve the usability of Chinese posters and infographics.
3. Key Challenges in Chinese Poster Generation
Chinese text generation exposes model weaknesses more readily than ordinary image generation for several reasons.
First, Chinese contains a large number of characters with complex glyph structures. Visually similar characters may differ by only one radical or a small number of strokes. If the model learns only the overall visual shape, it may generate an approximate but incorrect character.
Second, poster text is rarely limited to a single line. A typical design may include titles, labels, descriptions, and decorative phrases, each with different font sizes, colors, orientations, and alignment rules. The model must handle both textual correctness and layout quality.
Third, text often overlaps with graphics. A title may be placed over a person, product, or textured background. Without spatial constraints, characters may be obscured or become visually connected to the edges of the main subject.
Therefore, evaluating Chinese text generation should not focus solely on whether an image looks attractive. It should also consider:
- Text recognition accuracy
- Completeness of long text
- Correct character order
- Hierarchy between titles and body text
- The degree of distortion in text regions
- Consistency between visual style and semantic content
Evaluation tasks involving complex visual text generation and long-text rendering are designed to measure the model’s overall performance in these realistic scenarios.
4. The Engineering Value of Non-Standard Resolution Support
Many image generation systems support only a small number of fixed sizes, such as square images, wide-screen layouts, or vertical poster formats. This simplifies training and batch processing but does not fully match real content-production requirements.
Actual business scenarios involve a wider range of canvas sizes:
- E-commerce campaigns may require long images or narrow advertisements.
- Mobile covers often use vertical layouts.
- Feed content may require specific pixel dimensions.
- Display screens and printed materials typically use different aspect ratios.
- Social media templates may require dynamic cropping based on the layout.
Supporting arbitrary aspect ratios is not simply a matter of resizing images to a fixed shape. The tokenizer, positional encoding, latent-space mapping, and decoding process must maintain a consistent representation of image dimensions.
One possible approach is to improve the image tokenization strategy so that the model can represent visual grids with different widths and heights within a certain range. During generation, the system creates a visual-token layout based on the target dimensions, and the decoder then generates the image. This can reduce stretching, cropping, and subject displacement, allowing the result to better match the requested layout.
However, support for non-standard resolutions does not mean that every size will achieve the same quality. Larger images generally require more computation and memory. Extremely wide or tall aspect ratios may also make subject placement less stable. In practice, systems should define a reasonable range of supported dimensions and perform quality checks on generated results.
5. System Optimization for Training on Domestic Chips
Stable large-model training on domestic computing platforms usually requires coordinated optimization across the framework, scheduler, communication layer, and operators.
5.1 Data Processing and the Training Pipeline
Data preprocessing should not be completely separated from training. Instead, the system should establish a unified data pipeline that covers:
- Image-text data cleaning
- Duplicate sample removal
- Image quality filtering
- Text-image matching checks
- Resolution and aspect-ratio statistics
- Harmful-content and low-quality sample filtering
- Training-data sharding and caching
If data preparation cannot keep pace with the compute devices, the training cluster will spend time waiting, reducing chip utilization. Data loading, prefetching, decoding, and batching therefore need to be designed for parallel execution.
5.2 Dynamic Graphs and Multi-Level Pipelining
Task dispatch, operator execution, and device-to-device communication can all become bottlenecks during training. A multi-level pipeline based on dynamic graphs can separate data preparation, computation orchestration, and device execution, reducing idle time caused by continuous task dispatch.
Multi-stream parallelism can also overlap communication with computation. For example, while some devices perform matrix calculations, other data can be used for gradient synchronization or parameter preparation. With carefully designed dependencies, this approach can reduce communication stalls and improve overall throughput.
5.3 Operator Fusion and Memory-Access Optimization
Large-model training repeatedly performs operations such as normalization, optimizer updates, exponential moving averages, and gradient processing. If each operation generates separate memory reads and writes, memory pressure increases.
Operator fusion combines consecutive operations into a single execution flow, reducing intermediate-tensor materialization and repeated memory access. Common optimization directions include:
- Fusing normalization and scaling operations
- Combining optimizer updates with weight correction
- Reducing the creation of temporary tensors
- Optimizing type conversion under mixed-precision training
- Reducing unnecessary copies in communication buffers
These optimizations may not change the model architecture directly, but they can have a significant impact on training efficiency and stability.
6. From Training to Application: A Complete Workflow
A reusable multimodal image generation training workflow generally includes the following stages.
Stage 1: Dataset Construction
Collect images, descriptions, layout annotations, and text-region information, then establish a consistent data format. For poster data, the dataset can also record text-box coordinates, font-size hierarchy, and foreground-background relationships.
Stage 2: Foundation Pretraining
Use large-scale image-text data to teach the model visual concepts, text semantics, and composition relationships. The goal at this stage is to build general capabilities rather than optimize for a single application.
Stage 3: Supervised Fine-Tuning
Train the model on high-quality instruction data so that it can handle more complex user requirements, including multi-constraint layouts, specified text content, style transfer, and size control.
Stage 4: Preference and Quality Optimization
Use human or automated evaluation data to further improve visual appeal, text accuracy, layout quality, and instruction following. For text-related tasks, readability should receive greater weight in the overall evaluation score.
Stage 5: Inference Deployment
Deployment must account for memory consumption, concurrent requests, timeout handling, failure retries, and result caching. Enterprise applications should also include content-safety review, logging, and generation-result tracking.
7. Practical Considerations for Production
First, text accuracy should not rely solely on visual spot checks. A text-recognition model can be used for automated evaluation, with higher weights assigned to titles, key selling points, and numerical information.
Second, non-standard image sizes increase inference costs. Size policies should therefore be designed around business requirements. When necessary, systems can use staged generation, local repainting, or post-processing layout.
Third, generated content may still contain factual errors, abnormal brand marks, or sensitive material. Production systems need an appropriate review workflow and should not treat generated images as final deliverables without verification.
Fourth, adapting to domestic chips is not a one-time task. As model scale increases, communication topology, memory management, operator coverage, and compiler optimization will all require continued iteration.
Overall, the combination of autoregressive understanding, diffusion-based generation, glyph constraints, and end-to-end training on domestic chips represents an important technical direction for multimodal image generation. It brings language reasoning, visual composition, text rendering, and computing-platform adaptation into one system. This can improve the practicality of complex poster-generation scenarios while helping move generative AI from algorithmic prototypes toward complete engineering systems.
For developers, the most valuable lesson is not the name of any particular model. It is the systematic methodology behind the approach: data, architecture, operators, training workflows, and application-level evaluation must form a closed loop. Only then can a system continuously deliver generation capabilities that are stable, controllable, and suitable for production use.
