Training a SOTA Model End-to-End on Domestic AI Accelerators: What Full-Pipeline Training Really Takes

    A recent open-source image generation model was trained end-to-end on domestic AI accelerators and a domestic deep learning framework, spanning the full pipeline from data preprocessing to pre-training, fine-tuning, and reinforcement learning, without relying on the existing mainstream GPU software stack at any stage. This article sets brands aside and focuses on the engineering: which stages domestic compute must clear to become genuinely usable, the four stages that make up full-pipeline training, why training is far harder than inference, how to rigorously read a SOTA claim, a standard procedure for end-to-end migration with numerical and scaling-alignment checks, the infrastructure that cannot be overlooked, and the practical significance and common misconceptions for developers and enterprises evaluating a migration.

    Recently, an open-source next-generation image generation model announced that the base of its autoregressive architecture was trained entirely on domestic AI accelerators and a domestic deep learning framework, covering everything from data preprocessing to large-scale pre-training, fine-tuning, and reinforcement learning training, without using the existing mainstream GPU software stack at any stage. This news carries far more technical weight than a simple hardware swap.

    Over the past two years, the industry has held a persistent, consensus-level anxiety: domestic substitution on the inference side is largely settled, but the pre-training of frontier models still depends heavily on the existing GPU ecosystem. The reason is that training and inference differ by orders of magnitude in engineering difficulty.

    This article sets brands aside and talks only about engineering: which stages domestic compute must clear to become genuinely usable, what “full-pipeline training” actually includes, how a SOTA claim should be rigorously verified, and which metrics and risks developers and enterprises should watch when evaluating a migration.

    1. What “Full-Pipeline Training” Actually Means

    Many people interpret “training on domestic cards” as “just changing the device parameter in the training script,” which severely underestimates the training pipeline. A complete model production pipeline usually has four stages, each with a completely different workload profile:

    1. Data preprocessing and data engineering: image decoding, cropping and resizing, relabeling, tokenization, and packing into fixed-length samples. This stage is CPU-, memory-bandwidth-, and storage-throughput-intensive, and it is often the first bottleneck to appear at large scale.

    2. Pre-training: the most compute-intensive and longest-running stage. It sets the upper bound of the base model’s capability and places the harshest demands on parallelism strategy, numerical stability, and long-running stability.

    3. Supervised fine-tuning (SFT): instruction following and format alignment. The data volume is small but structure-sensitive, testing the efficiency of the data pipeline and sample packing.

    4. Reinforcement learning / preference alignment (RL/RLHF-style): a heterogeneous workload that alternates between sampling (inference) and training (forward/backward), with the highest scheduling complexity.

    The significance of emphasizing “full-pipeline construction from data preprocessing to large-scale training” lies in this: it is not a PoC on a single stage, but all four stages wired together and reproducible. This is also the key dividing line for judging the maturity of domestic compute. Inference only validates operators and low-precision support, while full-pipeline training validates the engineering completeness of the entire software stack.

    2. Why Training Is Harder Than Inference

    Inference is “one forward pass, aiming for low latency”; training is “forward + backward + parameter update + gradient synchronization, running continuously for weeks while pursuing convergence.” The difficulties concentrate in five areas.

    First, the combinatorial complexity of parallelism strategies. Large-model training typically uses data parallelism, tensor parallelism, pipeline parallelism, and sequence/context parallelism at the same time, layered with gradient accumulation, optimizer state sharding (ZeRO-style ideas), and activation recomputation. Each parallelism introduces different collective-communication requirements (AllReduce, AllGather, ReduceScatter, All-to-All), placing extremely high demands on the quality of the collective-communication library implementation and its topology awareness.

    Second, the coverage of operators and graph compilation. Training graphs are full of custom operators, dynamic shapes, and sparse and fused operators. Any small operator that is missing or must fall back to a generic implementation will create a break in the compute graph and cause a cliff-like drop in performance.

    Third, numerical precision and convergence consistency. Under mixed-precision training, loss scaling, accumulation precision in normalization layers, and numerical differences in the softmax and attention implementations can all be amplified into noticeable loss divergence after thousands of steps.

    Fourth, long-running stability and fault tolerance. Running thousands of cards continuously for days, single-card failures, slow cards, and communication jitter are almost inevitable. Checkpointing, elastic fault tolerance, and fault localization are essential; otherwise effective training time is heavily wasted.

    Fifth, observability. Without a mature profiling toolchain, it is impossible to tell whether the bottleneck is in compute, communication, or data supply.

    These five factors determine that migration is not “changing interfaces” but a complete systems-engineering effort.

    3. How to Read a SOTA Claim

    The “text rendering” capability of image generation models is often used as a SOTA benchmark because it simultaneously tests character-level localization, glyph structure, language consistency, and layout composition, a subtask widely recognized as hard to do well in generative models. But when reading a SOTA claim, three follow-up questions are worth keeping in mind:

    One, comparability. Open-source and closed-source models are usually not equivalent in parameter scale, training data volume, or inference steps. The claim should be qualified as “leading among comparable open-source models.”

    Two, evaluation protocol. Does the leaderboard score disclose the prompt set, decoding parameters (temperature, guidance scale, sampling steps), evaluation resolution, and the version of the scoring model? Change the prompt distribution and the ranking can change.

    Three, reproducibility. Are the weights, inference code, and evaluation scripts open? Can a third party reproduce a close score?

    What is truly valuable is not the score itself, but the fact that “this score can be reproduced on a domestic compute foundation.” That turns the claim from marketing into a verifiable engineering fact.

    4. A Standard Procedure for End-to-End Migration

    Below is a fairly reliable order of progression in practice. The core idea is “small to large, align numerics before performance.”

    1. # Illustrative code: a four-layer progression for migration validation, not the API of any specific framework
    2. # 1) Single-op / single-layer numerical alignment
    3. op: logits_new = custom_kernel(x)
    4. logits_ref = reference_impl(x.cpu())
    5. assert torch_close(logits_new, logits_ref, rtol=1e-3, atol=1e-3)
    6. # 2) Single-card, single-step forward/backward alignment
    7. loss_new, grads_new = train_step(model, batch, mixed_precision="bf16")
    8. compare(loss_new, loss_ref); compare(grads_new, grads_ref)
    9. # 3) Small-cluster convergence alignment (same seed, same data order, running hundreds to thousands of steps)
    10. for step in range(N):
    11. loss = train_step(model, batch)
    12. assert abs(loss - loss_curve_ref[step]) < threshold(step)
    13. # 4) Scaling efficiency: fix the global batch, vary the card count, measure throughput and linearity
    14. throughput = benchmark(scale=[8, 32, 128])
    15. scaling_efficiency = ideal_speedup / actual_speedup

    Each layer should form an automated regression: operator-level checks validate numerical tolerances, single-step checks validate gradient norms, multi-step checks validate loss-curve trends, and finally scaling efficiency measures the quality of the communication and parallelism implementation. In addition, the random seed, data order, and mixed-precision policy should all be fixed. If any one of them is not fixed, the conclusion of numerical alignment does not hold.

    5. Infrastructure Cannot Be Overlooked

    Getting a model to train is only half the story. On the data side, a high-throughput parallel file system or object storage combined with local high-speed caching is typically needed so that thousands of cards are not left waiting on data. On the scheduling side, a containerized training platform is needed to support queue management, priorities, preemption, and fault-triggered restarts. On the observability side, logging services, metrics monitoring, and alerting are needed to bring accelerator utilization, memory usage, communication bandwidth, and data-loading wait time into a single view.

    Major domestic cloud providers already offer general capabilities in this area. For example, Baidu AI Cloud provides corresponding foundational services across categories such as heterogeneous compute cluster management, object storage, container platforms, and monitoring and alerting, which can directly support building the training pipelines described above. One rule of engineering experience: at the thousand-card scale, the time spent on data supply and checkpoint writes often exceeds the gains from compute optimization, and deserves priority investment.

    6. What It Means for Different Roles

    For model R&D teams, it means a second compute supply path is available, no longer passively waiting on the scheduling and quotas of a single supply chain, with significantly improved control over training plans.

    For enterprise IT and platform teams, it means private training and fine-tuning become feasible; scenarios involving data compliance and model-asset ownership no longer require sending weights and data outside the internal network.

    For application developers, the most direct change is fine-tuning capability at controllable cost. Model customization for image generation and document understanding, once affordable only to top-tier institutions, is progressively moving down to a scale mid-sized teams can bear.

    The deeper significance lies in a positive ecosystem feedback loop: only when real frontier models are fully trained in production environments will operator gaps, performance shortcomings, and stability defects be fully exposed and fixed.

    7. Three Misconceptions to Avoid

    Misconception one: equating “it runs” with “the same efficiency.” Completing training only proves the pipeline works; whether it offers cost-effectiveness still depends on per-unit-compute throughput, scaling efficiency, and failure rate, and these numbers must be measured with your own model and your own data, not carried over from marketing claims.

    Misconception two: overlooking the hidden labor cost of migration. Filling operator gaps, debugging precision, and locating slow cards often require dedicated engineering effort; scheduling should reserve time for this rather than estimating it as a “zero-cost replacement.”

    Misconception three: underestimating the role of the framework layer. The success rate of hardware replacement depends largely on the depth of upper-layer framework adaptation and graph-compilation optimization, not on hardware parameters themselves.

    Conclusion

    The conclusion is clear: completing full-pipeline training of a frontier model on a domestic compute foundation is a key validation of the transition from “usable” to “genuinely usable.” What it proves is not a particular number, but the completeness of the entire engineering pipeline. For developers and enterprises, the pragmatic approach is to start with small- and mid-scale tasks, build an automated regression system for numerical alignment and scaling efficiency, and then progressively migrate larger-scale workloads onto the foundation.