Full-Process Training of Frontier Image Models on Domestic AI Compute Platforms

    While domestic AI compute platforms have been widely adopted for inference, training frontier models remains a high-barrier challenge. A recent full-process training practice for a new image generation model demonstrates that domestic compute, combined with its software stack, can successfully handle the entire pipeline from data preprocessing to model training and alignment. This article explores the technical complexities of image generation training, the integration of autoregressive and diffusion architectures, and the systemic requirements for achieving stable, efficient, and reproducible full-process training on emerging domestic compute infrastructures.

    Over the past period, domestic AI compute platforms have become relatively common in inference scenarios. However, high barriers remain in large-scale pre-training, complex multimodal modeling, and frontier model iteration. For image generation models, training requires massive compute resources and involves data cleaning, image-text alignment, distributed communication, mixed precision, model parallelism, training stability, and evaluation. Any weak link can cause unstable convergence or subpar results.

    Recently, a full-process training practice for a new generation image generation model on domestic compute drew attention. Its core value lies not merely in releasing a new model, but in proving that with domestic compute and a domestic software stack, a frontier model can complete the entire technical chain—from data preprocessing to training and alignment optimization. This marks a shift: domestic compute is no longer just for deploying pre-trained models but is participating in the entire process of model capability formation.

    1. Why Image Generation Training is a High-Difficulty Test

    Evaluating a compute platform’s training capabilities requires looking beyond single inference speed or theoretical peak compute. Training is a long-term, complex engineering task demanding high system stability.

    First, image generation models process massive image-text data. Raw data from web pages, public materials, or licensed datasets contains low-quality images, duplicates, incorrect descriptions, and copyright risks. Pre-training requires image decoding, size processing, text cleaning, quality filtering, semantic matching, and deduplication, placing high demands on storage bandwidth, data scheduling, and compute synergy.

    Second, these models often use multi-stage training: learning general visual representations, aligning image and text, and optimizing for specific scenarios like text rendering or complex compositions. Different stages have varying compute modes, mixing large-scale matrix operations with high-frequency data exchanges. Single hardware metrics cannot fully reflect training efficiency.

    Third, distributed training is highly sensitive to communication. As model and batch sizes scale, devices must frequently sync parameters, gradients, and intermediate states. Poor communication leads to compute waiting, lower utilization, and instability. The training platform must integrate compute, storage, network, and software compilation optimization.

    2. Cognitive Image Generation: From “Drawing” to “Understanding Before Drawing”

    Traditional image generation models excel at text-to-image tasks but struggle with complex instruction comprehension and accurate text rendering. For example, when asked to generate a poster containing formulas, flowcharts, or long text paragraphs, the model might get the overall style right but fail to ensure correct text content, reasonable layout, and clear logical relationships.

    The root cause is that image generation is not mere pixel prediction. The model must understand entities, actions, spatial relationships, quantities, causality, and visual styles, mapping them to the image space. For knowledge-heavy posters, educational illustrations, and complex scene designs, the model also requires world knowledge and reasoning.

    A representative technical route combines autoregressive modeling with diffusion-based image generation. The autoregressive part acts as a “planner” or “brain,” responsible for:

    • Parsing user instructions;
    • Identifying subjects and attributes in the image;
    • Establishing scene layout and spatial relationships;
    • Understanding text, numbers, and symbols;
    • Determining overall composition, style, and generation sequence.

    The diffusion part acts as the “drawer,” responsible for:

    • Restoring image details;
    • Generating natural textures and lighting;
    • Optimizing edges, strokes, and local structures;
    • Improving image clarity and visual quality;
    • Handling detail consistency in high-resolution outputs.

    By combining both, the model first forms a clear semantic and compositional plan before refining the visual layer. This architecture demonstrates stronger adaptability in complex instructions, knowledge-intensive content, and text rendering tasks.

    3. Why Text Rendering Remains a Bottleneck

    Text generation seems simple but is a high-difficulty problem in image models. Natural images tolerate local errors as long as the overall structure is acceptable. Text, however, has strict discrete constraints—one wrong stroke, a changed number, or a missing punctuation mark can alter the meaning.

    Text rendering involves at least three levels of capability:

    1. Semantic Correctness: The model must understand the requested content rather than just generating text-like textures.
    2. Spatial Layout: Titles, body text, labels, tables, and annotations require logical hierarchy, spacing, and direction.
    3. Visual Fidelity: Strokes, edges, thickness, and clarity must meet high standards, especially for small fonts and dense layouts.

    To improve this, training data must include structured visual materials like posters, tables, manuals, charts, and UI screenshots, not just plain images. Training objectives must balance image quality and text accuracy, moving beyond traditional visual similarity metrics.

    During evaluation, it is recommended to use both human and automated assessments. Automated evaluation checks text recognition accuracy, layout consistency, and attribute matching, while human evaluation judges aesthetics, semantic logic, and practical usability. Combining both prevents the model from over-optimizing for a single metric.

    4. Full-Process Training Requirements for Domestic Compute

    “Full-process” means forming a stable closed loop across the hardware and software system, typically including the following stages:

    1. Data Preprocessing

    This stage involves batch reading, format conversion, content filtering, image scaling, text tokenization, image-text matching, and sample packing. To improve efficiency, high-cost processing is often done offline, while sharded datasets, caching, and parallel data loading reduce training wait times.

    2. Pre-training

    Pre-training determines base capabilities and requires long runtimes and massive device coordination. Key focus areas include compute throughput, memory usage, gradient synchronization, checkpointing, and exception recovery.

    3. Supervised Fine-Tuning (SFT)

    SFT improves performance on specific tasks like complex instruction comprehension, text generation, and style control. The data scale is smaller but requires higher quality, needing finely designed instructions, answers, and quality labels.

    4. Preference Alignment and Reinforcement Optimization

    After acquiring basic generation capabilities, the model is further optimized using human preferences, rule-based feedback, or automated scores. This stage involves multi-round generation, scoring, filtering, and parameter updates, demanding high task scheduling and workflow orchestration.

    5. Evaluation and Deployment Verification

    Training completion does not mean the model is ready for use. It requires safety checks, stability tests, multi-resolution testing, and anomaly input testing. Additionally, validated operators, precision strategies, and parallel schemes must smoothly migrate to the inference environment.

    5. Key Issues for Domestic Software Stacks

    Hardware is just the foundation; the software ecosystem dictates resource utilization. To support frontier training, domestic software stacks must improve in several areas:

    Operator Coverage: Training involves numerous base operators like matrix multiplication, normalization, attention, sampling, and tensor transformations. Inadequate coverage causes frequent fallbacks, degrading performance.

    Compilation Optimization: Graph compilation, operator fusion, memory reuse, and auto-tuning reduce intermediate data transfers and redundant computation, lowering memory pressure and boosting utilization.

    Mixed Precision Support: Low-precision compute saves memory and boosts throughput but must be paired with loss scaling, high-precision retention for critical layers, and numerical monitoring to prevent gradient overflow or training divergence.

    Distributed Capabilities: Data, tensor, and pipeline parallelism must be combined based on model structure and device topology. Collective communication operations, fault recovery, and checkpoint resumes directly impact long-term training stability.

    Development Toolchains: Developers need tools to profile operator latency, memory usage, communication waits, and device utilization. Without comprehensive profiling tools, problem-solving relies on guesswork rather than systematic engineering.

    6. Implications for the Industry

    First, evaluation standards for domestic compute are upgrading. The focus shifts from whether it can run inference to whether it can support the full lifecycle from data to training and deployment.

    Second, it reduces reliance on a single tech ecosystem. When training frameworks, compilers, operator libraries, and hardware platforms work synergistically, enterprises gain flexible resource allocation and enhanced supply chain security.

    Third, it shifts R&D from “results-oriented” to “systems engineering-oriented.” A model’s advancement depends not just on benchmark scores but on training efficiency, iteration speed, data loops, fault recovery, and deployment costs. Only when the entire process is controllable can model capabilities translate into product value.

    However, this progress should be viewed rationally. A single successful full-process training does not mean all LLM types can migrate seamlessly. Different models have varying operator structures, parameter scales, data types, and communication modes, requiring continuous adaptation experience. Additionally, training costs, software maturity, and ecosystem compatibility will affect actual adoption speed.

    7. How Enterprises Should Evaluate

    For enterprises planning to use domestic compute for model training, an evaluation system can be built around the following dimensions:

    1. Task Adaptability: Verify support for the target model’s operators, parallel methods, and precision strategies.
    2. End-to-End Efficiency: Measure total time for data reading, compute, communication waits, and checkpointing, not just theoretical compute.
    3. Stability: Observe exception rates, checkpoint recovery capabilities, and result reproducibility over long runs.
    4. Migration Cost: Assess the effort required to modify existing code, data pipelines, monitoring systems, and deployment processes.
    5. Tool Maturity: Ensure the ability to quickly identify performance bottlenecks, memory issues, and numerical anomalies.
    6. Security and Compliance: Check training data sources, content filtering, output risks, and access control mechanisms.

    Overall, completing full-process training for frontier image models demonstrates a sustainable technological path: from hardware adaptation to software-hardware synergy, from single-point inference to complete training, and from “able to run” to “stable, efficient, and reproducible.” As data processing, pre-training, fine-tuning, alignment, and deployment form a closed loop, domestic compute is entering the core of LLM R&D. Future competition will shift from single chips or models to comprehensive capabilities in compute architecture, software tools, data engineering, and application ecosystems.