The First Multimodal Image Generation Model Trained End to End on Domestic AI Chips
This article examines the architecture and training stack behind a multimodal image-generation model trained entirely on domestic Chinese AI hardware. It explains the model’s autoregressive-plus-diffusion design, Chinese text rendering, arbitrary-resolution generation, end-to-end training workflow, and optimization techniques for improving device utilization, communication efficiency, numerical stability, and production readiness.
Introduction: Multimodal Image Generation Enters a New Hardware Era
Two recent developments in AI infrastructure deserve to be viewed together.
The first concerns the supply of computing hardware. According to public reports, restrictions on the export of high-end GPUs to China by an international chip manufacturer may be easing, reducing some of the uncertainty surrounding long-term supply.
The second concerns demand for computing power. A Chinese AI company and a domestic computing-hardware vendor jointly released and open-sourced a multimodal image-generation model. The entire pipeline—from data preprocessing and large-scale pretraining to supervised fine-tuning (SFT) and reinforcement learning (RL)—was completed on a domestically produced training server. Based on publicly available information, it is the first multimodal generative model in the industry to complete full-process training on domestic chips.
The two developments may not have a direct causal relationship. Together, however, they point to a broader trend: hardware–software co-optimization between domestic computing platforms and foundation models is becoming more mature. Image generation, one of the most representative multimodal workloads, is moving from proof of concept toward large-scale production training.
Using this model as a case study, this article analyzes its architectural choices and training-system optimizations, and summarizes adaptation practices that other teams can reuse when deploying models on domestic AI accelerators.
Note: The model, hardware, and framework names discussed here are based on public information. The code samples are illustrative pseudocode intended to explain engineering concepts.
1. Why Image Generation Is a Stress Test for Domestic AI Hardware
Language models typically generate text autoregressively, one token at a time, with a relatively consistent workload. Image-generation models are more demanding because a complete training pipeline usually combines several very different computing patterns.
- Memory-bound workloads: Autoregressive branches decode tokens sequentially. Although the amount of computation per step is relatively limited, the model frequently reads the KV cache, making memory bandwidth a major bottleneck.
- Compute-bound workloads: A diffusion decoder iteratively denoises representations in latent space. Each step requires a full forward pass, placing significant pressure on matrix-multiplication units.
- Heterogeneous auxiliary operators: VAE encoding and decoding, glyph encoding, normalization, positional encoding, and related operations have diverse and sometimes fragmented execution patterns.
- Text-processing workloads: Large-scale image–text preprocessing includes image decoding, resizing, cropping, tokenization, and encoding. These CPU-heavy tasks can prevent accelerators from receiving data fast enough.
For a domestic accelerator, the challenge is not merely supporting one workload. The chip must handle all of these patterns at the same time. That requires broad operator coverage, mature compilation and graph optimization, and high-performance multi-device interconnects capable of handling frequent gradient synchronization.
The more important requirement is full-process training. Many adaptation projects stop after making inference work or training a small model. Completing the entire pipeline—from raw data to final checkpoints—on a domestic hardware and software stack demands a much higher level of engineering reliability. A small numerical inconsistency in a normalization operator or an improperly scheduled communication step can cause a long-running training job to drift silently or fail after substantial resources have been consumed.
For this reason, the significance of the first multimodal model trained end to end on domestic chips is not simply that it can run. It demonstrates that the model can be trained effectively. It provides a complete path that later teams do not need to rediscover from scratch.
2. Architecture: Autoregressive Understanding and Diffusion Rendering
2.1 A Hybrid Design for Bringing Language-Model Reasoning into Generation
The model uses a hybrid architecture that combines an autoregressive component with a diffusion decoder. The two modules have distinct responsibilities:
| Module | Publicly disclosed scale | Primary role | Analogy |
|---|---|---|---|
| Autoregressive component | 9B | Semantic understanding, instruction following, and global composition planning | Understands the request and plans the composition |
| Diffusion decoder | 7B DiT | Iterative latent-space denoising and visual-detail generation | Renders the final image |
The autoregressive component was trained entirely on domestic computing hardware. The diffusion decoder uses a 7B DiT architecture—Diffusion Transformer—and incorporates a dedicated Glyph Encoder to improve text rendering.
The design follows a broader direction also seen in recent multimodal-generation products from leading international vendors: combining the reasoning capability of language models with image generation. Zhipu has described this general approach as “cognitive generation.” Instead of directly transforming noise into an image, the system first determines what should be drawn and how it should be arranged, then decides how to render it.
The division of labor has a clear technical basis:
- The autoregressive branch handles global consistency. Generation proceeds as an ordered token sequence, allowing text, layout, and visual elements to be organized into a structured plan. This is well suited to instruction understanding and composition planning.
- The diffusion decoder handles high-fidelity details. Through iterative denoising, it gradually reconstructs a latent representation before decoding it into pixels with a VAE. This is advantageous for textures, lighting, and local visual details.
- Compared with purely autoregressive image generation, shifting the most expensive visual computation to the decoder means the full language model does not need to run for every image token, making inference cost and latency easier to control.
In short: the language model thinks through the image, while the diffusion model renders it accurately.
2.2 Chinese Text Rendering: From Approximation to Explicit Glyph Conditioning
Image-generation models often produce incomplete strokes, misplaced radicals, and visually similar but incorrect Chinese characters. Chinese text rendering is also one of the most direct indicators of whether an image-generation system is useful in practical applications. Chinese is particularly challenging because:
- The character set is large. Thousands of commonly used characters have structures far more complex than Latin letters.
- Characters contain dense strokes. At small sizes, strokes can merge, especially in structurally complex characters.
- Many characters look similar. Correct recognition may depend on very small differences in individual strokes.
- Text depends heavily on context. Long titles and multi-line layouts must preserve stylistic consistency across characters.
- High-quality training data is difficult to produce. Image–text alignment requires careful collection, annotation, and cleaning.
Pure diffusion models have difficulty generating structurally accurate text for three main reasons:
- They learn a direct mapping from textual descriptions to pixels without an explicit character-level representation.
- Text information can be diluted by the conditioning signal for the overall image.
- Dependencies between characters in a long sentence are difficult to preserve in latent space.
The hybrid architecture addresses this by separating what to write from how to render it:
- The autoregressive branch parses the prompt and determines the text content and approximate layout positions. It first works out the copy and composition.
- The Glyph Encoder converts those characters into glyph features and injects them as additional conditions into the diffusion decoder. The decoder effectively receives a reference form to reproduce.
This design is the key technical support behind the model’s reported leading open-source results on public evaluations for complex visual-text generation and long-text rendering.
Practical selection advice: If an application frequently generates Chinese slogans, product names, specification tables, or long headlines, evaluate whether the model has an explicit text-understanding, glyph-conditioning, and diffusion-generation path. Do not judge it solely by the overall attractiveness of sample images.
2.3 Inference Flow
Prompt│▼┌────────────────────┐ ┌────────────────┐│ Autoregressive │ Semantic│ Global layout ││ branch (9B) │────────▶│ plan ││ Intent and layout │ │ Copy and ││ understanding │ │ element positions│└────────────────────┘ └──────┬─────────┘│ Text content▼┌──────────────┐│ Glyph Encoder │ Glyph features└──────┬───────┘│ Condition injectionRandom noise ─────▶ ┌──────────────▼────────────────┐│ Diffusion decoder (7B DiT) ││ Iterative latent-space denoising│└──────────────┬─────────────────┘▼┌──────────────┐│ VAE decoding │└──────┬───────┘▼Target-size image at any ratio
3. Arbitrary Resolutions: Essential for Poster and Marketing Design
3.1 The Limitations of Preset Aspect Ratios
Many image-generation models support only a small number of fixed aspect ratios, such as 1:1, 4:3, or 16:9. Real-world applications are much more varied:
- Portrait posters commonly use 3:4 or 9:16.
- E-commerce detail pages may require images longer than 1:4.
- Social-media cover images may use a wide 2.35:1 format.
- Displays, roll-up banners, lightboxes, and other physical materials require many different dimensions.
When the generated resolution does not match the target format, users typically resort to cropping or stretching:
- Cropping can remove important information and may cut through text.
- Stretching can distort text and graphics.
- Repeated regeneration increases both cost and latency.
3.2 How Arbitrary-Resolution Generation Works
The model improves its tokenizer strategy to support arbitrary aspect ratios between 1024×1024 and 2048×2048. For example, it can directly generate a 1240×1600 image even though that size is neither square nor based on powers of two.
At a conceptual level, a diffusion model represents the canvas as a two-dimensional latent-feature grid. The grid’s height and width determine the shape of the canvas. Traditional implementations often assume a square latent grid, which limits the output to a small set of fixed ratios.
With an improved tokenizer, the number of rows and columns in the latent grid can scale independently according to the target canvas. Changing the width and height changes the grid shape itself. Combined with multi-ratio resolution buckets during training, this allows the model to learn stable priors for different aspect ratios. Width and height are no longer tied to a fixed relationship, making arbitrary ratios a natural consequence of the design.
# Illustrative pseudocode: width and height determine the resolution independentlydef generate(image_model, prompt: str, width: int, height: int,steps: int = 30, cfg_scale: float = 4.0):# 1. Autoregressive branch: understand semantics and plan compositionplan = image_model.ar_branch(prompt) # Layout, copy, element positions# 2. Glyph encoding: convert text to glyph conditionsglyph_cond = image_model.glyph_encoder(plan.text_segments)# 3. Tokenizer: construct a latent grid from the target width and heightlatent = image_model.tokenizer.init_noise(width, height)# 4. Diffusion decoder: denoise iteratively using layout and glyph conditionsfor t in image_model.diffusion_decoder.timesteps(steps):noise_pred = image_model.diffusion_decoder(latent, t, condition=plan.condition, glyph=glyph_cond)latent = scheduler.step(noise_pred, t, latent, cfg_scale)# 5. Decode the latent representation into pixelsreturn image_model.vae.decode(latent)
For developers, this means the application can pass the required width and height directly without building a separate post-processing pipeline for every format.
3.3 Engineering Considerations
- Performance increases with pixel count. Compute and memory usage are roughly proportional to the total number of pixels. A 2048×2048 image contains approximately four times as many pixels as a 1024×1024 image. Batch inference should therefore be planned by total pixel count rather than image count.
- Training must cover multiple aspect ratios. Training only on square images can produce unstable composition at extreme ratios. Resolution-bucket sampling across multiple aspect ratios improves reliability.
- Small outputs require separate validation. Whether images with a side shorter than 1024 pixels are usable should be determined through application-specific testing.
- Extreme panoramas require testing. Results are generally most reliable within the training range. For formats outside that range, including extreme ratios such as 1:5, test with small batches before production deployment.
4. End-to-End Training: From Data Processing to Reinforcement Learning
According to publicly disclosed information, the team built an end-to-end training suite on top of a deep-learning framework. It covers data preprocessing, pretraining, SFT, and RL, and runs entirely on Ascend Atlas 800T A2 systems.
From a distributed-training perspective, the suite treats the image-generation model in a way similar to a large language model. The workflow, parallelism strategies, and optimizer design can draw on established language-model training practices. The main differences are that the data pipeline includes image encoding and the objectives include pixel-level reconstruction and generation quality.
┌──────────────┐ ┌──────────────┐ ┌─────┐ ┌─────┐ ┌─────────────────┐│ Preprocessing │──▶│ Pretraining │──▶│ SFT │──▶│ RL │──▶│ Evaluation and ││ │ │ │ │ │ │ │ │ deployment │└──────────────┘ └──────────────┘ └─────┘ └─────┘ └─────────────────┘Multi-node data pipeline End-to-end training on a domestic cluster
| Stage | Primary objective | Data characteristics | Key engineering concerns |
|---|---|---|---|
| Data preprocessing | Convert raw material into high-quality training samples | Image–text pairs, rendered text, high-quality aesthetic images | Decode and encode throughput, deduplication, cleaning, and sampling ratios |
| Pretraining | Learn the general mapping between semantics, composition, and pixels | Large-scale image–text pairs and synthetic rendered data | Data, model, and pipeline parallelism; communication–computation overlap; checkpoint recovery |
| SFT | Align the model with instruction formats and common business scenarios | Instruction–image pairs and multi-turn dialogue data | Data mixing, learning-rate and step-count control, loss monitoring |
| RL | Optimize generation according to preference objectives | Human or model scores and image–text consistency rewards | Reward stability, sampling throughput, and reward hacking prevention |
Several details are particularly important:
- Data quality establishes the foundation. Chinese text-rendering ability depends heavily on the data. In addition to large-scale general image–text pairs, the model needs specialized samples containing specified copy, layout, and finished images. These should cover different fonts, sizes, backgrounds, and layout styles.
- Pretraining determines the upper bound. The autoregressive branch requires large-scale image–text data to learn semantic understanding and composition planning. The diffusion decoder needs high-quality samples that balance visual fidelity with text accuracy.
- SFT determines usability. Instruction tuning for posters, illustrations, covers, and other scenarios makes the model better aligned with real usage patterns.
- RL shapes preferences. Reward signals can optimize image–text consistency and text accuracy. The main engineering challenges are stable reward design and sufficient sampling throughput.
- Training cost should be managed hierarchically. Large-scale parallelism is appropriate for pretraining, while post-training should focus on precise alignment. This concentrates expensive compute where it provides the greatest value.
5. Training-System Optimization: Keeping Domestic Hardware Fully Utilized
For long-running training jobs, “it runs” is only the baseline. The important questions are whether the hardware is fully utilized and whether the training remains stable. The disclosed optimizations can be grouped into three categories.
5.1 Submission Bottlenecks: Multi-Stage Pipeline Dispatch
In dynamic-graph execution, the framework commonly processes operators one by one. The CPU parses the graph, selects operators, and dispatches work, while the accelerator executes it.
For large operators such as fully connected layers, matrix multiplication, and attention, device execution takes long enough that dispatch overhead is almost negligible. However, in models dominated by normalization and elementwise operations, an individual operator may complete in only tens of microseconds. If CPU dispatch cannot keep up, the device periodically waits for instructions.
Multi-stage pipeline dispatch separates the submission process into several stages:
- The front end constructs graphs and selects operators.
- The middle stage prepares execution, including memory allocation and constant processing.
- The back end submits work to the device queue.
Different threads can advance through these stages concurrently. CPU dispatch becomes decoupled from device execution, allowing dispatch latency to be hidden behind device computation.
The idea is similar to a database connection pool: instead of creating a new connection for every request, the system reuses prepared resources and removes repeated setup overhead.
5.2 Communication Bottlenecks: Overlapping Communication and Computation
In data-parallel training, gradients generated during backpropagation must be synchronized across devices. Communication can reach several gigabytes per step. If computation and communication are serialized, each step becomes:
compute → wait for communication → continue computing
A large amount of accelerator capacity is then spent waiting.
Multi-stream execution assigns separate streams to computation, communication, and data transfer. This allows communication to overlap with computation. Once the gradient for a layer becomes available during backpropagation, its reduction can begin while later layers continue their backward computation.
# Illustrative pseudocode: overlap backpropagation and gradient synchronizationcompute_stream = device.Stream("compute")comm_stream = device.Stream("comm")with compute_stream:loss = model(batch)with compute_stream:loss.backward() # Gradients are produced layer by layercomm_stream.wait_stream(compute_stream) # Wait for communication to be launchedwith comm_stream:all_reduce(model.grads) # Synchronize gradients across devicescompute_stream.wait_stream(comm_stream) # Wait for synchronized gradientswith compute_stream:optimizer.step() # Update parameters
Overlap is not free:
- Communication and computation share memory bandwidth, so more streams are not always better. The configuration must be tuned against the actual hardware topology.
- Dependencies between the computation graph and execution streams must be managed precisely. Otherwise, the system may read data before it is ready, creating subtle correctness bugs.
- At larger scales, gradient buckets can be scheduled together with backward computation so that computation hides communication. This type of overlap remains an important optimization area in its own right.
5.3 Operator Bottlenecks: Ascend-Oriented Fused Operators
When fine-grained operators run through generic implementations one at a time, every operator may need to read inputs, compute, and write results back to device memory. Each operation also incurs its own dispatch overhead.
A fused operator combines multiple logical operations into a single device execution. Intermediate data can remain in on-chip cache, reducing device-memory traffic and the number of dispatches. Public optimization information mentioned three categories:
| Fused operator | Combined logic | Benefit |
|---|---|---|
| AdamW EMA | Adaptive moment estimation, weight decay, and exponential moving average | Completes three parameter-update operations together with a single parameter read and write |
| COC, a curriculum-style contrastive algorithm | Negative-sample selection and contrastive-loss computation | Reduces intermediate tensors and performs sampling on the device |
| RMSNorm | Root-mean-square normalization and subsequent affine transformation | Reduces intermediate writes to device memory and dispatch overhead |
AdamW EMA illustrates the general approach:
- Training maintains both model weights and EMA weights. EMA smooths the weight trajectory and can reduce fluctuations near the end of training, so it is often used for final export.
- In a conventional implementation, adaptive moment updates, weight decay, and EMA updates run as separate operators. Model parameters may be read and written repeatedly.
- A fused implementation completes the entire update in one operator call, reading and writing the parameters once while reducing dispatch overhead.
# Illustrative pseudocode: conventional implementation versus fused implementation# Conventional: multiple operators and repeated device-memory accessm = beta1 * m + (1 - beta1) * gradv = beta2 * v + (1 - beta2) * grad * gradparam = param - lr * (m / (sqrt(v) + eps) + wd * param)ema = decay * ema + (1 - decay) * param# Fused: complete the update in a single operator calladamw_ema(param, grad, m, v, ema,lr=lr, beta1=0.9, beta2=0.999, eps=1e-8,wd=0.01, ema_decay=0.999)
6. A Reusable Engineering Method for Domestic-Accelerator Adaptation
This project provides a complete example from which other teams can derive a practical adaptation path.
1. Preserve correctness before pursuing speed
The first principle is numerical alignment.
- Run the same data and configuration for several hundred steps on both platforms.
- Compare outputs and gradients layer by layer, keeping relative error within an acceptable range.
- Pay particular attention to normalization, attention, and loss computation, where implementation details can strongly affect results.
- Performance optimization before numerical alignment only makes debugging more difficult.
2. Keep the device busy
First determine whether the bottleneck is dispatch-bound or communication-bound, then apply the corresponding optimization:
- For dispatch-bound workloads, prioritize multi-stage pipeline dispatch.
- For communication-bound workloads, prioritize computation–communication overlap.
- For memory-bound workloads, prioritize fused operators.
3. Reduce memory consumption
Use mixed precision, such as BF16 computation with FP32 master weights, activation recomputation, optimizer-state sharding, and tensor or pipeline parallelism. The goal is to find the configuration with the highest throughput and acceptable stability within the available device memory.
4. Make long-running training reliable
Long training jobs require operational safeguards:
- Periodic checkpointing and restart from checkpoints.
- Automatic isolation of failed devices and rescheduling of jobs.
- Centralized collection and visualization of training metrics.
- Elastic scheduling through container platforms, with logging, monitoring, and alerting for rapid diagnosis.
5. Build an evaluation loop
Visual-generation evaluation is inherently subjective, particularly for Chinese text rendering. Teams should create an automated evaluation set covering:
- Character accuracy and image–text consistency.
- Aesthetic quality and layout quality.
Run the evaluation set after every model iteration. This avoids the misleading conclusions that can result from judging a model only by a few sample images.
Teams that want to use similar capabilities without building a training cluster from scratch can combine compute scheduling, object storage, logging, and monitoring services from cloud platforms such as Baidu AI Cloud to construct a complete pipeline from data processing to model serving.
7. Applications and Deployment Recommendations
7.1 Typical Applications
- Commercial posters: Generate images directly from specified dimensions and copy, avoiding repeated generation, cropping, and manual text replacement.
- E-commerce materials: Produce product images, long-form detail-page graphics, and promotional banners that match store templates.
- Social-media covers: Generate landscape, square, and portrait versions from the same prompt.
- Educational illustrations and infographics: Render Chinese labels directly and reduce post-production text editing.
- Education and office work: Create illustrations for presentations, course materials, and handouts with readable Chinese characters.
7.2 Selection and Integration Advice
- Evaluate text rendering with real business copy. Build a test set from actual application content and measure character accuracy rather than relying only on public rankings.
- Test the required output sizes. Verify composition and text stability at every resolution commonly used by the business.
- Estimate resource costs. Calculate compute and memory requirements based on pixel count, denoising steps, and concurrency. Quantization or distillation may be considered when model compression is needed.
- Design a fallback path. For scenarios requiring very high text accuracy, combine generation with OCR validation and localized redrawing.
- Review licensing and compliance. Open-source weights may have different licenses that must be checked before commercial use. AI-generated content should also be labeled in accordance with applicable requirements.
- Track version changes. Public information indicates that an online experience has been launched on Zhipu’s platform and is expected to be integrated into one of its conversational products. The model’s capabilities are still evolving.
Conclusion
Architecturally, the combination of autoregressive understanding and diffusion decoding brings language-model reasoning into the image-generation process. Improvements in Chinese text rendering move the model beyond simply making attractive images toward producing useful business assets. Arbitrary-resolution generation directly addresses real requirements such as posters, covers, and e-commerce materials.
From an engineering perspective, multi-stage pipeline dispatch, multi-stream communication overlap, and Ascend-oriented fused operators show how the performance gaps of domestic AI hardware are being addressed step by step. More importantly, completing the full training pipeline creates a reusable path for future teams.
Changes in international GPU export policy may affect the short-term supply landscape. However, the milestone of training and open-sourcing a model entirely on domestic chips remains significant. As more hardware options become available, competition will increasingly return to the model itself. The capabilities that matter in production are straightforward: understanding the request, rendering text correctly, and generating the image at the required dimensions.
