Engineering a Fully Trained Text-to-Image Model on Domestic AI Chips
This article examines the engineering challenges of training a multimodal text-to-image model entirely on domestically developed AI accelerators. It covers the full pipeline, including data preparation, model pretraining, distributed optimization, inference validation, non-standard resolutions, Chinese poster generation, system optimization, evaluation, and open-source release practices. The discussion highlights why full-process training is more demanding than inference-only deployment and what teams should consider when building reproducible and production-oriented AI training systems.
In recent years, generative AI has expanded from text understanding to image generation, visual question answering, and multimodal content creation. Compared with using an existing model, completing the entire process—from data cleaning to large-scale pretraining—on domestically developed AI chips introduces substantially greater engineering complexity. It also provides a more comprehensive way to validate the compatibility of domestic hardware and software stacks with AI training workloads.
A class of multimodal text-to-image models trained end to end in a domestic chip environment has recently been released. Its training pipeline covers data preprocessing, sample construction, model pretraining, distributed optimization, and inference validation, with core computation running on domestic AI accelerators.
The significance of this work is not limited to whether the model can generate images. More importantly, it explores how established large-model training methods can be migrated to a different computing architecture while addressing practical challenges such as Chinese typography, variable image sizes, and complex semantic understanding.
1. Why Full-Process Training Has Greater Engineering Value
Model development is usually divided into data preparation, pretraining, instruction tuning, evaluation, and deployment. If a model only performs inference on domestic AI chips, the main challenges involve operator compatibility and runtime performance. Full-process training requires the team to solve several additional problems.
1.1 Adapting the Data Processing Pipeline
Large-scale image-text datasets involve image loading, format conversion, resolution adjustment, text cleaning, label generation, and quality filtering. If data processing cannot keep pace with training throughput, the accelerators remain idle and overall resource utilization declines.
1.2 Adapting Training Operators and Compilation
Large-model training relies on matrix multiplication, normalization, attention, activation functions, gradient communication, and many other operators. Hardware architectures differ in operator support, memory layout, mixed-precision behavior, and compiler optimization. Training frameworks therefore require targeted adaptation.
1.3 Ensuring Stable Distributed Communication
As model and dataset sizes increase, a single device is often insufficient. Multiple devices must work together through parameter synchronization, gradient aggregation, checkpointing, and fault recovery. Each of these factors affects training efficiency and stability.
1.4 Maintaining Training Consistency
The same data, model architecture, and optimization strategy may produce small numerical differences on different hardware platforms. Without complete experiment records and consistent evaluation criteria, it is difficult to determine whether a problem originates from the data, code, hardware, or training strategy.
Full-process training is therefore not simply a hardware replacement. It is a systematic migration spanning data engineering, model engineering, software adaptation, and operations.
2. Core Components of a Text-to-Image Model
A complete text-to-image system typically includes a text encoder, an image generation module, and conditional control components.
The text encoder interprets natural-language prompts and converts them into semantic representations that the model can process. For Chinese use cases, the encoder must handle word boundaries, long text, numbers, punctuation, and specialized expressions effectively. Otherwise, generated text may be incomplete or semantically inaccurate.
The image generation module gradually constructs image content. Many systems generate images in a compressed visual representation space and then use a decoder to reconstruct the final image. This reduces computational cost, but it also creates challenges in detail recovery, text clarity, and small-object generation.
Conditional control components improve control over composition, style, layout, and local content. A user may not simply want “a product poster,” but may also require the title at the top, the main subject in the center, a specific background color, and several Chinese phrases to remain visible. The model must then understand semantics, spatial relationships, and visual hierarchy at the same time.
3. Data Preprocessing Is the Foundation of Training Quality
The quality of a text-to-image model depends heavily on its training data. Raw image-text data often contains duplicate images, low-resolution content, inaccurate descriptions, copyright risks, garbled text, and image-text mismatches. Such data cannot be used directly for training.
A comprehensive data preparation pipeline can include the following steps.
3.1 Image Quality Filtering
Images can be filtered according to dimensions, sharpness, file integrity, and content-safety policies. Severely compressed images, heavily blurred images, solid-color images, and duplicates reduce training efficiency.
3.2 Text Normalization
Descriptions should be deduplicated, cleaned of abnormal characters, normalized for punctuation, and filtered for invalid content. Chinese datasets also require attention to mixed simplified and traditional characters, irregular spacing, invisible characters, and encoding errors.
3.3 Image-Text Consistency Checking
Images and descriptions must have a basic semantic correspondence. A vision-language matching model can be used for initial filtering, followed by rule-based checks or manual sampling to reduce mismatched examples. If descriptions consistently disagree with their images, the model may learn incorrect associations.
3.4 Resolution and Aspect-Ratio Analysis
It is not always appropriate to crop every image to one fixed size. Real-world content includes landscape, portrait, square, and unusual aspect ratios. Before training, teams should analyze the aspect-ratio distribution and design an appropriate resolution-bucketing strategy.
3.5 Data Sharding and Caching
For large-scale training, processed samples should be converted into shard formats that support sequential access. A local cache or high-speed storage layer can reduce latency caused by frequent access to remote storage. Data reading, decoding, and augmentation should preferably run as a parallel pipeline.
4. Supporting Non-Standard Resolutions
Traditional text-to-image models often use a fixed resolution. This simplifies training and stabilizes memory usage, but limits practical applications. Posters, covers, long-form images, and product displays often require different aspect ratios. Support for non-standard resolutions is therefore important for improving usability.
A common approach is multi-resolution bucketing. Several training resolutions are defined in advance, and samples are assigned to buckets according to their aspect ratios. Images are then cropped or resized within each bucket. This preserves more of the original composition while remaining compatible with batch processing.
Several engineering issues require attention:
- Different resolutions require different amounts of memory, so batch sizes cannot be configured mechanically.
- Changes in height and width alter the dimensions of visual feature maps, so positional encoding must adapt accordingly.
- Extremely wide or narrow images may distort the proportions of the main subject.
- Inference should restrict extreme dimensions to prevent sudden memory growth.
- Evaluation should separately report results for square, landscape, portrait, and long-aspect-ratio images.
Non-standard resolution support does not mean that every possible size will produce the same quality. The model must balance training coverage, inference cost, and generation stability.
5. Special Challenges in Chinese Poster Generation
Chinese poster generation is a particularly difficult application because it combines image generation with typography and layout.
Text has a clear character structure: omitting a radical or stroke can make a character unreadable. Poster text also follows a hierarchy, including titles, subtitles, prices, dates, and button labels. In addition, text positions must satisfy spatial constraints involving the background and decorative elements. The model cannot merely generate textures that look vaguely like text.
Several approaches can improve Chinese poster generation:
- Add high-quality Chinese posters with accurate textual descriptions to the training data.
- Apply finer-grained annotation or augmentation to text regions.
- Use text representations that are better suited to modeling Chinese characters.
- Add text-legibility checks after generation.
- Establish separate evaluation metrics for titles, body text, and decorative text.
In practical applications, a hybrid approach can also be used: image generation handles the background, main subject, and overall style, while a dedicated layout module renders the text. This usually provides better text accuracy for commercial posters, although it makes the system pipeline more complex.
6. System Optimization Priorities on Domestic AI Chips
Training on domestic accelerators requires more than optimizing the speed of individual operators. Teams must evaluate the entire training loop.
6.1 Mixed Precision
Lower-precision computation can reduce memory consumption and improve throughput. However, it must be combined with loss scaling, higher precision for critical operators, and numerical monitoring to prevent gradient overflow or abnormal loss behavior.
6.2 Memory and Communication
Training large models consumes substantial memory for parameters, gradients, optimizer states, and activations. Gradient accumulation, activation recomputation, parameter sharding, and checkpoint strategies can help reduce peak memory usage.
For multi-device training, communication overhead must also be monitored. If communication accounts for too much of each iteration, overall efficiency will remain poor even when the accelerators provide strong computational performance. Parallel strategies should therefore be designed to minimize unnecessary synchronization.
6.3 Data Pipelines
In an efficient system, data prefetching, decoding, augmentation, and model computation should overlap. Multiprocess loading, hot-data caching, and asynchronous prefetching can reduce accelerator idle time.
6.4 Fault Recovery
Long-running training requires regular checkpointing, random-state recording, data-position tracking, and automatic recovery. If a device or task fails, training should be able to resume from the most recent valid state instead of repeating a large amount of computation.
7. Building a Reliable Evaluation System
Text-to-image models should not be evaluated solely through subjective visual inspection. A multi-dimensional evaluation system is recommended.
- Text alignment: Whether the generated content matches the subject, action, scene, and style specified in the prompt.
- Image quality: Sharpness, structural integrity, color harmony, and richness of detail.
- Text legibility: Whether Chinese characters are correct, complete, and clear.
- Layout quality: Whether poster elements are obstructed, overcrowded, or visibly misaligned.
- Resolution generalization: Whether quality remains stable across different aspect ratios.
- Safety and compliance: Whether the model generates inappropriate content, sensitive information, or clearly incorrect visual expressions.
Automated metrics are suitable for large-scale regression testing, while human review can identify problems involving complex compositions and textual semantics. Both should be used together. Prompts, random seeds, and output dimensions should remain fixed during evaluation so that different versions can be compared reliably.
8. Considerations for Open-Source Release
Open-sourcing a model involves more than uploading its weights. A release should provide complete usage guidance and reproducibility information, including the model’s intended use, applicable scenarios, data-processing principles, inference workflow, known limitations, license information, and safety requirements.
Without preprocessing code, configuration files, or evaluation examples, users will have difficulty reproducing published results. Enterprise users also need clear information about whether the model is suitable for commercial scenarios, whether generated content requires human review, and what compatibility differences may occur across hardware environments.
From an ecosystem perspective, the value of an open model lies in enabling developers to conduct further training, integrate applications, and improve results—not simply in providing a demonstration page. Platforms such as Baidu AI Cloud can be combined with general-purpose compute resources, object storage, container platforms, logging services, and monitoring and alerting capabilities to build complete training and deployment workflows. The specific architecture, however, should be designed according to data scale, compliance requirements, and cost objectives.
9. Conclusion
Fully training a text-to-image model on domestic AI chips demonstrates the combined capabilities of data engineering, algorithms, compilers, computing infrastructure, and operational engineering. The central challenge is not implementing a particular model architecture in isolation, but keeping the entire training system stable while achieving verifiable results for practical scenarios such as non-standard resolutions, Chinese poster generation, and complex semantic understanding.
Teams preparing similar projects should prioritize data quality governance and evaluation infrastructure before progressively addressing operator adaptation, distributed training, and inference optimization. Only by establishing an observable, recoverable, and reproducible engineering workflow can the combination of domestic computing infrastructure and generative models evolve from a one-time validation exercise into a sustainable production capability.
