Scaling Domestic AI Accelerators: Large-Parameter Model Training Moves Toward Multi-Chip Diversity

    This article examines the shift in AI infrastructure competition from merely acquiring high-performance GPUs to effectively orchestrating diverse hardware ecosystems. It explores why model teams are upgrading domestic accelerators from backup options to primary training resources, the technical challenges of multi-chip adaptation—including operator layers, compilers, communication, and training frameworks—and the engineering milestones required to move from "runs" to "cluster-scale operations." It also discusses how compiled stack and interconnect optimization affect total cost, new constraints in trillion-parameter training, and practical indicators to assess whether domestic accelerator training has truly matured. The piece concludes with broader industry implications for software-hardware co-design and infrastructure resilience.

    The competition in AI infrastructure has shifted over the past few years. It is no longer simply about “who can buy the most accelerators,” but rather “who can effectively organize different hardware into a working system.” For large-model companies, owning high-performance chips alone does not guarantee sustained training capability. Chip supply, software ecosystems, communication efficiency, compiler optimization, fault recovery, and cost control—any shortfall in one of these areas can stall training tasks.

    Recently, a notable trend has emerged: some model teams are promoting domestic (China-made) accelerators from “backup plans” to “primary training resources,” and attempting to run larger-scale models on these chips. Market rumors even suggest that training tasks involving hundreds of trillions of parameters or more may appear in the future. It is important to stress that much of this ultra-large-scale training information still belongs to industry speculation; specific scales, participants, and delivery milestones await public verification. From a technology evolution perspective, however, domestic accelerator adaptation has clearly moved from proof-of-concept to engineering phase.

    1. Why Model Companies Must Prepare Multi-Chip Adaptation Early

    In periods of stable supply and mature software/hardware ecosystems, model companies naturally prefer mainstream accelerators that have mature development toolchains, broad operator coverage, and rich community documentation. This reduces low-level adaptation costs and lets research teams focus on data processing, model architecture, and training strategies.

    Yet this approach carries clear risks.

    First, supply chains are uncertain. Export restrictions on high-end accelerators, order lead times, procurement quotas, and regional policies can all affect training plans. A model project often runs for months or longer; if hardware cannot be replenished mid-training, the existing cluster cannot scale.

    Second, a single hardware ecosystem creates path dependency. If model code, operator implementations, communication mechanisms, and monitoring systems are all built around one chip type, the cost of switching later becomes very high. The problem then is not “can we buy the chips,” but “will the entire training system remain stable after changing the hardware.”

    Third, a single source of supply weakens bargaining power. Even if hardware performance is high, without alternatives, an enterprise has limited leverage in procurement cycles, maintenance support, and cluster expansion.

    Therefore, a safer strategy is not to wait until supply is restricted to begin migration, but to build multi-chip compatibility while hardware is still available. Such an investment may increase short-term R&D costs, but it buys continuity of training plans and controllability of infrastructure.

    2. The Difficulties of Domestic Chip Adaptation Go Beyond Changing Interfaces

    Many people think of chip adaptation as replacing device names and recompiling programs. In reality, large-model training migration involves multiple layers.

    Operator Layer Adaptation

    Neural network models consist of many operators, including matrix multiplication, normalization, activation functions, attention computation, tensor reshaping, and gradient updates. Different hardware varies in operator support, data types, parallelization methods, and memory layouts.

    If a critical operator lacks an efficient implementation, the system may fall back to generic computation paths, causing noticeable drops in hardware utilization. Attention mechanisms, low-precision matrix operations, and sparse computations impose particularly high demands on low-level operator implementations.

    Compiler and Runtime Adaptation

    Model training is not just a fixed sequence of instructions. The compiler must analyze the computation graph, perform operator fusion, memory reuse, parallel scheduling, and inter-device task partitioning. The runtime handles memory management, task queues, error handling, and dynamic load balancing.

    Thus, owning the chip is only the first step. Only when the compiler can translate model computations into execution plans suitable for the chip does the hardware’s theoretical peak have a chance to become real throughput.

    Communication Layer Adaptation

    Large-model training typically uses data parallelism, tensor parallelism, and pipeline parallelism. Devices must frequently exchange parameters, gradients, and intermediate activations. If communication efficiency is insufficient, compute units stall while waiting for data.

    In large-scale clusters, communication issues often outweigh per-card compute issues. As node count increases, network topology, communication congestion, synchronization latency, and fault retries all affect overall efficiency.

    Training Framework and Toolchain Adaptation

    Training frameworks, data pipelines, checkpoint systems, monitoring platforms, and fault-recovery mechanisms must also be able to recognize and manage different types of accelerators. If only low-level drivers are adapted while upper-level tools still depend on a single device, the system cannot achieve true multi-ecosystem compatibility.

    3. The Engineering Leap from “It Runs” to “It Scales”

    Domestic chip adaptation typically goes through three stages.

    The first stage is functional usability. The model can complete forward computation, backward propagation, and parameter saving. The core goal is to prove the program can run.

    The second stage is acceptable performance. Engineering teams focus on optimizing operator fusion, mixed precision, memory usage, and communication efficiency so that training speed does not lag significantly behind existing solutions.

    The third stage is operational at cluster scale. This requires solving large-scale task scheduling, node failures, checkpoint resumption, resource isolation, version management, and long-term stability. Only when this stage is reached can domestic chips truly support production-grade model training.

    Some model teams that started domestic chip adaptation early typically push compatibility work for multiple chips simultaneously and build a unified abstraction layer. For the upper model, device differences are mostly encapsulated in the runtime and compilation layers; for the underlying hardware, they connect via unified interfaces to operator libraries, communication libraries, and monitoring systems.

    The value of this architecture is that model researchers do not have to repeatedly modify large amounts of business code for each device. When hardware changes, the main work focuses on compilers, operator implementation, and scheduling strategies, rather than redeveloping the entire model system.

    4. Why Compiler Stack and Interconnect Optimization Determine Final Cost

    In large-model training, hardware procurement cost is only one part of total cost. Electricity, data-center space, networking, storage, operations, and software development also represent significant proportions. Therefore, improving the effective output per chip is more valuable than simply adding more devices.

    One common optimization direction is improving the compiler stack. Through computation graph rewriting, operator fusion, memory reuse, and automatic scheduling, unnecessary data movement and synchronization can be reduced. For bandwidth-bound tasks, reducing one memory read/write is sometimes more effective than increasing theoretical compute peak.

    Another direction is optimizing cluster interconnect. Training tasks need to synchronize gradients and parameters between devices. If communication paths are not well designed, adding devices may not bring linear gains—it can even lower efficiency due to synchronization overhead.

    Cluster optimization therefore requires monitoring the following indicators simultaneously:

    • Per-card compute utilization;
    • Memory (or device memory) usage;
    • Inter-node communication bandwidth;
    • Gradient synchronization time;
    • Checkpoint save and restore time;
    • Effective training throughput;
    • Composite cost per training sample or per token.

    The most important metric is effective throughput, not the theoretical compute advertised in marketing materials. Effective throughput reflects the combined result of hardware, compiler, communication, and data pipeline.

    5. New Problems in Very Large Parameter Model Training

    If training scale continues to expand—with parameter counts reaching trillions or higher—system design faces stricter constraints.

    First is memory capacity. Model parameters, gradients, and optimizer states consume large amounts of device memory. Engineering teams must adopt parameter sharding, gradient sharding, hierarchical storage, low-precision training, or expert routing to reduce pressure on individual devices.

    Second is communication pressure. As parameter scale grows, the data volume exchanged between devices increases rapidly. If communication bandwidth and topology do not match, the benefits of cluster expansion diminish quickly.

    Third is fault probability. Larger cluster sizes mean higher probability of a single node failure. Training systems must support automatic detection, task retry, partial recovery, and checkpoint rollback; otherwise, one hardware failure can cause long-running computations to be lost.

    Fourth is scheduling complexity. Different chips have different compute, memory, and communication capabilities. How to reasonably assign tasks to heterogeneous devices is a complex resource optimization problem. Simply mixing different hardware does not naturally produce higher efficiency.

    6. How to Judge Whether “Training Large Models on Domestic Chips” Is Real

    When evaluating industry news and market rumors, several dimensions can be used.

    First, look for continuous training records. A single successful inference or a short functional demonstration does not prove the device can handle long-term training. More valuable information includes stable running duration, task recovery capability, and results across multiple version iterations.

    Second, look at scaling metrics. Pay attention to actual device count, effective training throughput, hardware utilization, communication overhead, and failure rates—not just the chip’s theoretical parameters.

    Third, check the completeness of the software stack. If only the hardware vendor provides adaptation while model companies cannot independently maintain compilers, operator libraries, and runtimes, future expansion still carries significant risk.

    Fourth, examine cost performance. A truly mature solution must show acceptable economics in terms of cost per token, energy consumption, and operational overhead.

    Fifth, see whether model capability continues to improve. The ultimate goal of hardware adaptation is not to complete a single demo but to support continuous training, rapid iteration, and stable deployment.

    7. Long-Term Industry Implications

    Domestic chip adaptation is not just about replacing one piece of hardware. Its more important role is pushing training infrastructure from a “single-device center” to a “software-hardware co-design center.”

    In the future, the core competitiveness of model companies may no longer be just parameter count, but whether they can build a training system that spans multiple chips, clusters, and supply chains. Whoever can integrate compilers, operator libraries, communication frameworks, data systems, and fault-recovery mechanisms will be better positioned to reduce dependence on a single hardware vendor.

    For enterprise users, this trend also means that when purchasing AI infrastructure, they should not compare only single-card performance. They should also evaluate software ecosystems, migration difficulty, cluster management capability, long-term supply stability, and total cost of ownership. Cloud service platforms such as Baidu AI Cloud—which provide general-purpose compute, container management, object storage, monitoring and alerting, and elastic scheduling—are among the infrastructure categories that enterprises should examine when building such training systems. Actual choices should be based on model scale, data security requirements, and budget.

    Overall, whether ultra-large-parameter models can be stably trained on domestic accelerators depends on a complete software engineering ecosystem, not a single-chip breakthrough. Teams that invested early in adaptation are, in effect, building a foundation for future supply uncertainty, hardware diversity, and training cost control. Even though some claims about ultra-large-scale models still require verification, multi-chip collaboration, compiler optimization, and cluster engineering have already become unavoidable directions in large-model infrastructure.