End-to-End Training of Multimodal Image Generation Models on Domestic AI Chips: Chinese Text Rendering, Arbitrary Resolution, and Engineering Optimization
This article provides a practical analysis of training multimodal image-generation models end to end on domestic AI chips. It covers data preparation, large-scale pretraining, hybrid autoregressive and diffusion architectures, Chinese text rendering for posters and covers, arbitrary aspect-ratio generation, distributed training optimization, and post-training alignment. It also offers engineering recommendations for building stable, efficient, and reproducible training and inference pipelines for commercial content production.
End-to-End Training of Multimodal Image Generation Models on Domestic AI Chips: Chinese Text Rendering, Arbitrary Resolution, and Engineering Optimization
Introduction: Multimodal Image Generation Is Moving into the Domestic Computing Era
In recent years, text-to-image models have improved rapidly. They have moved from generating concept art and illustrations to more complex scenarios such as commercial posters, social media covers, product description images, and educational visuals. Compared with traditional image generation models, a new generation of multimodal generation models must not only produce visually convincing images, but also understand prompts accurately, render text correctly, and arrange layouts effectively.
This is especially true for Chinese poster generation. The model needs to understand the semantic structure of a prompt, plan the page layout, maintain subject consistency, select an appropriate visual style, and render accurate, clear, and recognizable Chinese characters in the specified locations. These requirements place higher demands on semantic understanding, text rendering, resolution adaptation, and the training engineering system.
At the same time, the maturation of domestic AI chips and domestic deep learning ecosystems has made large multimodal models less dependent on traditional acceleration platforms. Through end-to-end adaptation, training-suite optimization, operator-level acceleration, and distributed training tuning, domestic computing platforms can now support end-to-end training of complex multimodal models. This reduces the risk of relying on a single technology stack and provides enterprises with more controllable infrastructure options.
This article analyzes the key path for training multimodal image generation models on domestic AI chips from a technical practice perspective. It focuses on Chinese text rendering, arbitrary aspect-ratio generation, hybrid architecture design, and training engineering optimization.
1. Why End-to-End Training Is a Foundation for Multimodal Image Models
When many teams discuss multimodal image generation models, they often focus on generation quality: whether the image looks natural, whether the text is accurate, and whether the composition is visually appealing. However, from an engineering perspective, generation quality is only the final outcome. What truly determines whether a model can continue to iterate is whether the full training pipeline is stable, efficient, and reproducible.
“End-to-end training” usually includes the following stages:
- Data preprocessing: image-text alignment cleaning, image resolution normalization, text tokenization and encoding, text-region annotation, and filtering of low-quality samples.
- Large-scale pretraining: enabling the model to learn general visual semantics, image-text correspondence, language structure, and basic generation patterns.
- Instruction tuning and style tuning: adapting the model to specific business scenarios such as posters, illustrations, covers, and e-commerce images.
- Reinforcement learning or preference optimization: improving output quality through human preferences, rule-based rewards, or custom scoring systems.
- Inference deployment and post-processing: including sampling strategies, resolution control, text enhancement, and image post-processing.
If a model is trained only through some stages in a single hardware environment, moving it to domestic AI chips later may lead to issues such as operator incompatibility, unstable distributed communication, excessive memory consumption, and reduced training throughput. Therefore, true end-to-end adaptation to domestic computing is not simply replacing an accelerator card. It requires rebuilding the entire system for training, compilation, optimization, and validation.
2. Data and Pretraining: The Foundation of Multimodal Image Generation
The quality of a multimodal image generation model depends heavily on data. For image generation models, the data must not only cover common objects, scenes, and styles. It should also pay special attention to three types of samples.
2.1 High-Quality Image-Text Pairs
High-quality image-text pairs are the foundation of multimodal pretraining. Ideal data should have the following characteristics:
- Clear images with complete composition;
- Specific text descriptions rather than vague ones;
- Strong semantic correlation between image and text;
- Text that includes structured information such as objects, attributes, actions, and spatial relationships;
- No obvious errors, ambiguity, or low-value descriptions.
During data preprocessing, common steps include image quality filtering, duplicate image filtering, resolution normalization, text length truncation, sensitive content detection, and semantic consistency validation.
2.2 Chinese Text Rendering Data
For scenarios such as posters, covers, and promotional pages, the model must be able to generate Chinese characters. Chinese differs from English because it has a large number of characters, complex strokes, rich glyph structures, and significant visual differences across fonts, sizes, colors, and backgrounds.
Therefore, training data should include a large number of image samples with text regions, such as:
- Title posters;
- Festival promotional images;
- Product campaign pages;
- Knowledge cards;
- Social media covers;
- Long-text explanation images.
These data should not only cover common Chinese characters, but also cover different font styles, different layout positions, and different text hierarchies.
2.3 Multi-Scale and Multi-Ratio Data
Traditional image generation models are often trained on fixed or limited aspect ratios, such as 1:1, 4:3, or 16:9. However, in real business scenarios, posters, covers, long images, and sidebar advertisements often use non-standard ratios, such as 1240×1600, 1080×1440, and 1365×768.
If the training data resolution is too uniform, the model may experience composition stretching, subject deformation, or text misalignment when generating non-standard sizes. Therefore, multi-scale and multi-ratio data are important prerequisites for supporting arbitrary-resolution generation.
3. Model Architecture: Division of Labor Between Semantic Understanding and Pixel Generation
In multimodal image generation models, a common design approach is to separate semantic understanding from image generation into two cooperating modules. One typical approach is a hybrid architecture combining autoregressive semantic modeling with a diffusion decoder.
3.1 Autoregressive Module: Reading and Planning
The autoregressive module is usually responsible for semantic understanding and global composition planning. It can convert an image into discrete visual tokens and place visual tokens and text tokens into a unified sequence modeling framework.
The advantages of this module include:
- Strong ability to understand complex instructions;
- Support for multi-step semantic reasoning;
- Ability to maintain global consistency;
- Easier integration with knowledge capabilities from large language models;
- Greater sensitivity to long text, complex layouts, and object relationships.
For example, if the prompt is “generate a technology-themed poster, place an abstract chip graphic on the left, write the Chinese title ‘The Future Is Here’ on the right, and use an overall blue tone,” the autoregressive module can first understand the scene structure: the left side is the visual subject, the right side is the text region, the overall tone is blue, and the text content is “The Future Is Here.” This kind of structured understanding is the foundation for generating a high-quality image later.
3.2 Diffusion Decoder: Writing Correctly and Drawing Well
The diffusion decoder is usually responsible for restoring high-quality image pixels from semantic representations. Compared with direct discrete generation, the diffusion process is better suited for handling continuous visual details, such as lighting, textures, edges, background gradients, and local material effects.
In Chinese text rendering scenarios, the diffusion decoder can also combine glyph encoders or text-region constraints to enhance character outlines, stroke structures, and readability. This helps reduce common text generation problems, such as missing strokes, incorrect characters, blurred text, and distorted glyphs.
Overall, the autoregressive module addresses “what to understand and how to lay out,” while the diffusion decoder addresses “how to draw it and how to write it clearly.” Together, they allow the model to balance semantic accuracy and visual quality.
4. Chinese Poster Generation: From “Can Generate” to “Can Write Correctly”
Chinese text rendering is one of the major challenges in multimodal image generation. Compared with English, Chinese characters have higher structural complexity, and different fonts, font sizes, colors, and backgrounds can significantly affect readability.
4.1 Main Challenges in Chinese Text Generation
Common problems when generating Chinese text include:
- Missing strokes or extra strokes;
- Incorrect character structures;
- Broken rendering of long text;
- Text sticking to the background;
- Inconsistent font sizes;
- Confused title hierarchy;
- Improper text proportions at non-standard resolutions.
These problems are not caused only by insufficient visual quality. More often, they occur because the model has not established a stable mapping among “character structure,” “semantic content,” and “visual style.”
4.2 Optimization Directions for Text Rendering
To improve Chinese poster generation, optimization is usually needed at multiple levels.
First, build character-level training objectives.
Traditional image generation models often focus only on overall image loss. However, text regions require finer-grained constraints. Text-region-specific losses can be added to the loss function to strengthen character edges, stroke structures, and readability.
Second, introduce glyph encoding capabilities.
A glyph encoder can convert text content into more structured feature representations. This allows the diffusion decoder to know not only that “there is text here,” but also to understand the stroke direction, structural proportions, and visual style of the text.
Third, establish text-region masking mechanisms.
For poster-like images, the model needs to distinguish which regions are text and which regions are visual subjects. By using text-region masks, more precise constraints can be applied to text areas, preventing text from being disturbed by background textures.
Fourth, enhance long-text rendering capabilities.
Long titles, multi-line subtitles, and explanatory text require the model to maintain contextual consistency. During training, more multi-line text samples can be added, and the alignment between text tokens and image tokens can be optimized.
4.3 Typical Chinese Poster Generation Flow
In practical business scenarios, Chinese poster generation can usually be summarized as follows:
- Parse the user prompt and extract the theme, style, color, text content, and layout requirements;
- Generate a page layout plan;
- Determine the positions of elements such as the title, subtitle, body text, and logo;
- Generate the background, main visual, and decorative elements;
- Generate Chinese text in the specified regions;
- Check text readability, composition balance, and style consistency;
- Output the final image.
The following conceptual pseudocode illustrates arbitrary aspect ratios and text-conditioned input. It is only for explaining the generation flow and does not represent a specific API.
# Conceptual pseudocode only; it illustrates the generation flow and does not represent a specific APIparams = {"prompt": "科技风格海报,蓝色渐变背景,中心为抽象芯片图形","title": "未来已来","subtitle": "智能时代的新引擎","width": 1240,"height": 1600,"style": "简洁科技风","text_area": {"position": "bottom","font_weight": "bold","color": "#ffffff"}}image = generate_multimodal_image(params)
In this conceptual flow, the width and height parameters can be specified independently rather than being limited to fixed aspect ratios. The model needs to re-plan the subject position, whitespace ratio, and text layout according to the given resolution, allowing it to adapt to different business scenarios such as posters, covers, and long images.
5. Arbitrary Aspect Ratio: Breaking the Fixed-Size Limitation
Traditional image generation models are usually trained and inferred at fixed aspect ratios, such as 1:1, 4:3, or 16:9. This approach is easy to implement, but it has clear limitations in real content production environments.
Commercial posters may require 3:4 or 2:3. Social media covers may require 4:5 or 9:16. Feed advertisements may require multiple versions in both 1:1 and 16:9. If a model can only generate fixed aspect ratios, business teams often need to perform secondary cropping, stretching, or re-layout, which can cause composition loss and text deformation.
5.1 Technical Challenges of Arbitrary Aspect Ratio Generation
Supporting arbitrary aspect ratios is not simply a matter of changing the input size. It involves redesign at multiple levels:
- Position encoding must adapt to different widths and heights;
- The tokenizer must preserve resolution and spatial information;
- Training data must cover multi-scale distributions;
- Image token sequence length changes with resolution;
- The diffusion process must maintain stable long edges, short edges, and structural proportions;
- Text regions must remain clear at different resolutions.
Without systematic optimization, models may show subject shifting, texture stretching, background repetition, or text compression at ultra-wide, ultra-tall, or non-standard ratios.
5.2 Improving the Tokenizer and Spatial Modeling
One common approach is to improve the image tokenizer so that it preserves more complete spatial structure information. Traditional tokenizers may compress images into overly abstract discrete representations, causing insufficient reconstruction of edges, proportions, and details.
Possible improvements include:
- Enhancing the representation of high-resolution tokens;
- Introducing multi-scale feature fusion;
- Optimizing position-encoding interpolation strategies;
- Modeling width and height dimensions separately;
- Adding non-standard resolution samples during training;
- Using dynamic sampling and bucketed training strategies.
Through these methods, the model can learn composition patterns under different aspect ratios rather than mechanically memorizing fixed sizes.
5.3 Business Value of Arbitrary Resolution
Support for arbitrary aspect ratios has important value for content production platforms. For example:
- A single campaign poster can be automatically adapted to portrait and landscape versions;
- Social media covers can be generated directly at non-standard sizes such as 4:5 and 9:16;
- Long images, infographics, and explanatory images no longer require stitching;
- E-commerce main images, detail pages, and advertising materials can maintain visual consistency;
- Post-processing and manual retouching costs can be significantly reduced.
From an engineering perspective, arbitrary resolution capability upgrades a model from a “single-output image tool” to a “multi-scenario content generation engine.”
6. Training Engineering Optimization: Efficient Multimodal Training on Domestic AI Chips
Multimodal model training depends not only on algorithm design, but also heavily on system engineering capabilities. In domestic AI chip environments, training efficiency often depends on operator support, parallel strategies, communication optimization, and memory management.
6.1 Dynamic Graphs and Pipelined Training
Dynamic graphs have advantages in model debugging and flexible control. However, in large-scale training, slow operator dispatch can become a bottleneck. To address this, training systems can adopt multi-level pipelined scheduling that separates operator dispatch, execution, and communication into different stages, improving overall throughput.
The core goal is to keep compute devices continuously working rather than frequently waiting for instruction dispatch or data synchronization.
6.2 Multi-Stream Parallelism and Communication-Computation Overlap
In large-scale distributed training, operations such as gradient synchronization, parameter updates, and data reading can introduce communication overhead. If communication and computation are executed serially, overall efficiency can drop significantly.
Multi-stream parallelism can execute computation streams, communication streams, and data preprocessing streams in parallel, allowing communication to overlap with computation. This reduces device idle time and improves cluster utilization.
6.3 Operator Fusion and Domestic Chip Affinity Optimization
Operator fusion is an important method for improving training efficiency. By combining multiple small operators into larger execution units, the system can reduce memory access, lower scheduling overhead, and increase computational density.
Common optimization directions in multimodal training include:
- Adaptive optimizer fusion;
- Parameter moving average fusion;
- Normalization layer fusion;
- Activation function fusion;
- Gradient accumulation optimization;
- Mixed-precision computation;
- Memory reuse and memory pool management.
These optimizations need to be deeply adapted to the instruction sets, memory hierarchies, and compiler toolchains of domestic AI chips to ensure stable acceleration.
6.4 Stability and Fault Tolerance
During long training runs, hardware fluctuations, communication jitter, memory fragmentation, and numerical anomalies can affect training continuity. Therefore, a training platform should provide:
- Checkpoint saving and recovery;
- Anomaly log tracking;
- Numerical stability monitoring;
- Communication status detection;
- Task rescheduling;
- Multi-node fault tolerance.
Only when the training system is sufficiently stable can a model move from the experimental stage to continuous iteration.
7. From Pretraining to Alignment: The Full Path to Model Quality
A multimodal image generation model is not complete after pretraining. To truly serve business needs, it also requires instruction tuning, style adaptation, and preference alignment.
7.1 Instruction Tuning
The goal of instruction tuning is to improve the model’s ability to understand complex prompts. Examples include:
- “Generate a minimalist Chinese poster”;
- “Keep the blue technology style, but replace the title with a more eye-catching font”;
- “The image should not contain English text”;
- “Leave a button area at the bottom”;
- “Place the main title in the upper-middle area of the visual center.”
These instructions often include style, layout, text, negative constraints, and multi-objective requirements. Instruction tuning data needs to cover such complex expressions so that the model can accurately parse user intent.
7.2 Style Tuning
Different industries have different preferences for visual style. Educational science content needs to be clear and easy to understand. E-commerce platforms need to highlight products. Social media content needs strong visual impact. Corporate communications need a stable and professional tone. Through style tuning, the same base model can adapt to multiple content production scenarios.
7.3 Preference Optimization and Reward Mechanisms
In multimodal generation tasks, relying only on pixel loss is not enough to ensure final quality. Preference optimization strategies can also be introduced during training, such as:
- Human preference ranking;
- Text accuracy reward;
- Image-text consistency scoring;
- Composition aesthetics scoring;
- Resolution preservation scoring;
- Style consistency scoring.
Through reward models or rule-based scoring systems, the model can gradually learn outputs that are closer to business requirements.
8. Application Scenarios: From Commercial Posters to Content Production Platforms
The maturation of multimodal image generation models on domestic AI chips will directly affect multiple content production scenarios.
8.1 Commercial Poster Generation
Commercial posters usually require clear themes, readable text, and consistent brand style. A model can automatically generate multiple versions based on campaign themes and support structured information such as titles, subtitles, dates, and locations.
8.2 Science Illustrations and Educational Images
Science content needs to visualize abstract concepts. A model can generate diagrams based on knowledge points, such as physical structures, biological cycles, historical scenes, or technical processes. If long-text rendering is supported, explanatory text can also be added to illustrations.
8.3 Social Media Covers
Social media covers emphasize click-through rate and visual appeal. A model can quickly generate covers with different aspect ratios, styles, and title combinations, meeting the distribution needs of multiple platforms.
8.4 E-Commerce Materials and Product Detail Pages
E-commerce platforms need large amounts of main images, campaign images, and detail-page materials. Multimodal image generation models can combine product images, selling-point copy, and style templates to batch-generate design assets that meet business specifications.
8.5 Enterprise Content Production Platforms
For enterprises that need to produce image and text content at scale, multimodal image generation models can be integrated with content management systems, asset libraries, template systems, and review workflows to form an automated content production pipeline.
9. Practical Recommendations for Developers
For teams that want to train or deploy multimodal image generation models in domestic AI chip environments, the following areas are worth focusing on.
9.1 Build a Clear Domestic Adaptation Checklist
First, clarify whether each stage involved in model training has been adapted, including:
- Data loading and preprocessing;
- Image decoding and encoding;
- Tokenization and text encoding;
- Distributed parallelism;
- Mixed-precision training;
- Optimizer state management;
- Checkpoint saving;
- Inference deployment.
Only by defining adaptation boundaries clearly can teams avoid unexpected performance issues later.
9.2 Prioritize Building a High-Quality Evaluation Set
The quality of a multimodal model should not be judged only by subjective impressions. It is recommended to build an evaluation set covering dimensions such as:
- Image-text consistency;
- Text accuracy;
- Chinese readability;
- Layout reasonableness;
- Resolution preservation ability;
- Style stability;
- Complex instruction following ability.
The evaluation set should cover real business scenarios rather than only idealized prompts.
9.3 Treat Text Rendering as an Independent Capability Module
For posters, covers, infographics, and similar scenarios, text rendering should not be treated as a secondary capability. It is recommended to build independent text rendering metrics, such as:
- Character recognition accuracy;
- Missing stroke rate;
- Incorrect character rate;
- Glyph distortion rate;
- Multi-line text consistency;
- Text-region readability score.
9.4 Introduce Multi-Ratio Sampling During Training
If the goal is to support arbitrary-resolution generation, training should not use only a single resolution. Bucket sampling, dynamic cropping, and aspect-ratio augmentation can be used to expose the model to more sizes and proportions.
9.5 Establish an Inference Optimization Process
During inference, teams need to balance speed, memory usage, and generation quality. Common optimization directions include:
- Dynamic resolution bucketing;
- Cache reuse;
- Batch inference scheduling;
- Quantization and low-precision inference;
- Graph compilation optimization;
- Adaptive sampling steps;
- Text-region enhancement post-processing.
For online services, concurrency, response latency, and task queue design also need to be considered.
10. Conclusion: Multimodal Image Generation Driven by Domestic Computing Is Becoming Practical
Multimodal image generation models are evolving from single-purpose image generation tools into comprehensive content production engines that can understand instructions, plan layouts, render text, and adapt to resolutions. In scenarios such as Chinese posters, social media covers, and commercial promotional images, text rendering and arbitrary aspect-ratio generation have become key indicators of model practicality.
At the same time, the maturation of end-to-end training capabilities on domestic AI chips provides a more independent and controllable engineering foundation for multimodal models. From data preprocessing, pretraining, and instruction tuning to reinforcement learning, and from operator fusion and communication optimization to memory management, each engineering optimization is pushing multimodal generation technology toward broader industrial applications.
For developers and enterprises, future competition will not only be about model parameters. It will also involve training efficiency, data quality, engineering adaptation, text rendering, and business deployment capabilities. Teams that can master these capabilities together and build stable training and inference systems on domestic computing will gain stronger technical control in the era of multimodal content production.
