A Practical Guide to Adapting 100B-Parameter Large Model Training on Domestic AI Accelerators
This article presents a practical approach to adapting large-scale model training for domestic AI accelerators, emphasizing that training adaptation is not limited to loading a model or completing a single forward pass. It covers reproducible baselines, two-node sixteen-card distributed setup design, model sharding and parallelism selection, gradient synchronization, numerical stability, memory and throughput tradeoffs, checkpointing and recovery, monitoring, and the path toward standardized out-of-the-box training. The goal is to help teams move large-model training from merely runnable to stable, observable, recoverable, and sustainable.
1. Rethinking training adaptation: more than getting a model to run
In large-model engineering practice, “training adaptation” is often simplified to whether a model can be loaded on domestic accelerators and perform forward computation. This understanding covers only part of what is required on the inference side. True training adaptation must simultaneously account for forward propagation, backward propagation, optimizer states, gradient synchronization, numerical stability, checkpoint saving, and long-running reliability.
In other words, the goal of training adaptation is not merely to make a model “start.” It is to keep the training process running continuously within an acceptable cost range, while making every update verifiable, recoverable, and explainable.
From an engineering perspective, training adaptation includes at least four layers:
Model structure compatibility: Core computational modules in the model must execute correctly on the target accelerator, including normalization, attention, activation functions, loss functions, residual connections, and multimodal feature fusion.
Distributed training compatibility: In multi-card and multi-node scenarios, parameter partitioning, gradient aggregation, and optimizer-state synchronization must complete correctly, without data loss, duplicated parameters, or communication deadlock.
Controllable numerical behavior: Key metrics such as loss values, gradient norms, learning rates, and update step sizes must remain finite and within a reasonable range, without persistent overflow, abnormal oscillation, or unrecoverable divergence.
Complete operational support: Training jobs must support logging, metric collection, fault localization, checkpoint recovery, and automated regression testing so that long-duration training jobs can be sustained.
Therefore, the acceptance criteria for training adaptation should not be limited to whether a model can complete one forward pass or one update. Instead, the key question is whether the system can complete a sufficient number of consecutive optimizer updates while remaining stable and controllable throughout the process. This is also an important basis for determining whether a training adaptation solution has reached an engineering-usable level.
2. Prerequisites: establish a stable and reproducible runtime baseline
Before large-scale model training adaptation begins, teams should first establish a stable and reproducible runtime baseline.
Reproducibility here means that under the same model, data, random seed, and parallelism configuration, the initialization parameters, data order, loss computation, and update trajectory are predictable. Without reproducibility, later performance optimization and fault localization become much more difficult.
The following preparation work is usually required before adaptation starts:
Pin software and hardware versions: This includes the accelerator driver, runtime environment, communication components, compute libraries, compiler version, and system dependencies. Version inconsistencies may change operator behavior, memory layout, or communication semantics.
Define the model clearly: Confirm the model structure, parameter scale, data types, sequence length, modality input format, and default hyperparameters. For multimodal models, also specify tensor dimensions and preprocessing rules for image, video, text, and other modalities.
Establish a small-scale baseline first: Use a smaller model, smaller batch size, shorter sequence, and fewer training steps to validate the main workflow before starting large-scale, long-duration training.
Define acceptance metrics: These include whether the loss value is finite, whether the gradient norm is finite, whether memory usage is controllable, whether training throughput meets expectations, and whether checkpoints can be saved and restored.
Prepare reference samples: If training results from other platforms are available, they can be used as reference curves to help identify obvious deviations in the current adaptation path. However, loss curves will not be identical across different hardware, parallelism strategies, and data orders. Trend plausibility and numerical stability should be the main judgment criteria.
The core goal of this stage is not to pursue high performance, but to put correctness before performance. Only after the basic workflow is stable should teams begin tuning parallelism strategies, optimizing communication, and improving throughput.
3. Key design considerations for two-node sixteen-card environments
A two-node sixteen-card configuration usually consists of two training nodes, each with eight accelerators, for a total of sixteen compute devices. This scale preserves the advantage of high-speed interconnect inside a node while introducing the complexity of cross-node communication. It is a typical scenario for validating distributed training capability.
When designing a two-node sixteen-card training environment, special attention should be paid to the following areas.
3.1 Clear device and process mapping
Each training process is usually bound to one accelerator. Processes on different nodes need globally unique identifiers.
For example, the total number of global processes can be defined as:
node_count = 2devices_per_node = 8world_size = node_count * devices_per_node
During actual execution, each process needs to know its global rank and its local device ID within the node. The global rank is used to determine parameter sharding, gradient synchronization, and log ordering. The local device ID is used to select the specific device.
A representative mapping is:
node_id = node identifierlocal_device_id = device identifier within the nodeglobal_rank = node_id * devices_per_node + local_device_id
3.2 Reasonable communication domain design
If only data parallelism is used, all devices can belong to the same communication domain for gradient synchronization.
When tensor parallelism, pipeline parallelism, or expert parallelism is introduced, different communication groups may need to be created at different levels. If communication groups are defined too early or too late, synchronization overhead may increase or the training workflow may become more complex.
A common recommendation is to first achieve stable training under data parallelism, then gradually introduce more complex parallelism strategies.
3.3 Different treatment for intra-node and inter-node communication
Intra-node communication usually offers higher bandwidth and lower latency, making it suitable for local synchronization.
Cross-node communication should emphasize communication merging, bucketized transfer, and overlap between computation and communication. If cross-node communication is not properly merged, frequent transmission of small messages can significantly reduce overall efficiency and increase timeout risk.
3.4 Decouple data loading from training processes
In large-scale training, data reading, preprocessing, batching, and random augmentation may become bottlenecks.
It is usually recommended to separate sample loading from training computation, and to use prefetching, caching, and batch assembly to reduce waiting time.
In distributed scenarios, it is also important to ensure that different processes read non-overlapping and reproducible data shards.
4. Model sharding and parallelism strategy selection
Models with hundreds of billions of parameters, or even larger scales, cannot be fully placed on a single accelerator. Therefore, model sharding is an essential part of training adaptation.
The goal of sharding is to reduce per-device memory pressure and improve scalability while preserving training correctness.
Common parallelism strategies include:
Data parallelism: Each device keeps a complete model replica. Different devices process different data batches, and parameters are updated through gradient synchronization. Its advantage is simple implementation and good scalability. Its limitation is that memory pressure on each device becomes significant when the model is large.
Fully sharded data parallelism: Building on data parallelism, model parameters, gradients, and optimizer states are partitioned across devices and temporarily aggregated when needed. This method can significantly reduce memory usage and is a common choice for large-model training.
Tensor parallelism: Part of the linear layers or attention computation in the model is split across multiple devices along tensor dimensions. This is suitable for scenarios where a single forward computation is large.
Pipeline parallelism: Different layers of the model are assigned to different devices, and micro-batches are pipelined to improve utilization. This method has high requirements for scheduling and load balancing.
Sequence parallelism: For long-sequence scenarios, the sequence dimension is split across multiple devices to reduce activation memory.
Expert parallelism: In sparse expert models, different expert modules are assigned to different devices, and computation is completed through a routing mechanism. This method requires special attention to expert load balancing and cross-device communication overhead.
For the early stage of training adaptation, it is usually recommended to use fully sharded data parallelism as the main path. This approach can reduce memory pressure while keeping training semantics relatively intuitive, making it easier to validate loss curves and update stability.
In a two-node sixteen-card scenario, the model can first be sharded across sixteen processes using fully sharded data parallelism. Each process maintains only part of the parameters, gradients, and optimizer states, and completes gradient synchronization after each backward pass.
For multimodal models, additional attention is needed for dimension alignment and fusion order between modalities. For example, text features, visual features, and video features may have different sequence lengths and embedding dimensions. During adaptation, the following should be confirmed:
- Whether the tensor shapes of each modality input are consistent.
- Whether the normalization method before modality fusion is stable.
- Whether dynamic resolution or variable-length sequences may cause memory fluctuations.
- Whether multimodal loss weights may amplify gradient anomalies from a specific modality.
- Whether data augmentation may affect the distribution between batches and further cause training convergence issues.
These details may not directly manifest as model structure errors, but they can affect the long-term stability of the training process.
5. Gradient synchronization and numerical stability: the core of training reliability
In training adaptation, gradient synchronization is one of the most critical and error-prone parts of distributed training.
The goal of gradient synchronization is to ensure that each device obtains globally consistent or appropriately sharded gradient information before optimizer updates. If the synchronization scope, timing, or precision is incorrect, the model may behave normally during the forward stage but gradually diverge after parameter updates.
In a two-node sixteen-card environment, gradient synchronization usually includes the following steps:
- Backpropagation computes local gradients.
- Gradients are reduced by gradient buckets or parameter groups.
- The synchronized gradients are passed to the optimizer.
- Parameters are updated.
- If needed, gradient norm clipping and post-update checks are performed.
To improve communication efficiency, gradient synchronization is often not performed all at once after backpropagation finishes. Instead, it is overlapped with the backward computation.
In engineering practice, a bucket-based merging approach is commonly used. Multiple parameter gradients are placed into the same bucket, and synchronization is triggered when a certain threshold is reached or when backpropagation reaches a certain stage. This reduces the number of communication operations and lowers the overhead caused by small-message transmission.
For numerical stability, training adaptation should continuously observe the following metrics:
Loss value: It should remain finite and change within a reasonable range. If the loss remains infinite, shows abnormal negative values, or exhibits sudden jumps, teams should check input data, the loss function, mixed precision, and initialization.
Gradient norm: It should remain within a finite range. A sudden increase in gradient norm usually indicates abnormal samples in a batch, an excessively large learning rate, or numerically sensitive modules in the model.
Gradient clipping: When training is unstable, clipping can be performed based on the global gradient norm to prevent a single update from being too large and causing divergence.
Learning rate warmup: Using a smaller learning rate at the beginning of training and gradually increasing it can help reduce risks caused by initialization and early batch distribution changes.
Precision strategy: Mixed precision can improve throughput and reduce memory usage, but teams must confirm which computations must remain in high precision, such as normalization, loss computation, and optimizer state updates.
A simple stability check pseudocode is shown below:
step = 0while step < total_steps:batch = get_next_batch()outputs = model_forward(batch)loss = compute_loss(outputs, batch.labels)grads = backward(loss)# Gradient norm checknorm_value = global_grad_norm(grads)if not is_finite(norm_value):handle_gradient_anomaly(step, norm_value)break# Optional: gradient clippinggrads = clip_by_global_norm(grads, max_norm=1.0)optimizer_step(grads)step += 1
This pseudocode does not depend on any specific framework. Its purpose is to show that a training loop must include numerical safety checks. In real engineering practice, gradient norms, loss values, and learning rates should be continuously recorded and used as important evidence for later troubleshooting.
6. Balancing memory management and throughput optimization
In training adaptation, memory management directly affects model scale and training efficiency.
Insufficient memory may prevent the task from starting, while excessively low memory utilization may reduce hardware efficiency. Therefore, memory management is not simply about avoiding errors. It is about balancing stability, throughput, and cost.
Memory usage usually comes from four parts:
- Model parameters: Memory occupied by the model weights themselves.
- Gradients: Gradient tensors generated during backpropagation.
- Optimizer states: Such as first-order momentum and second-order momentum, which can significantly increase memory usage.
- Activations: Intermediate results saved during forward propagation for later backward computation.
Under fully sharded data parallelism, parameters, gradients, and optimizer states can be partitioned across devices, significantly reducing per-device memory pressure.
However, activations may still become a bottleneck, especially for long sequences, multimodal inputs, or large attention modules. In such cases, techniques such as activation recomputation, sequence partitioning, gradient accumulation, and batch scaling can be considered.
Activation recomputation works by not saving all intermediate results during the forward stage, and recomputing them when needed during backpropagation. This can reduce activation memory usage but increases computation overhead. Whether to enable recomputation should be decided based on memory pressure, throughput goals, and training stability.
Gradient accumulation is used to simulate a larger batch when memory is limited. Multiple small batches perform backpropagation separately, gradients are accumulated, and then one optimizer update is performed.
It should be noted that gradient accumulation changes the training rhythm. Learning rate scheduling and logging statistics should be adjusted accordingly.
Throughput optimization mainly focuses on the number of samples or tokens processed per unit of time. Common optimization points include:
- Improving data prefetch efficiency and reducing the time compute devices spend waiting for data.
- Setting gradient bucket sizes appropriately to reduce frequent synchronization.
- Overlapping communication and computation to avoid long waits for cross-node synchronization.
- Performing operator-level optimization or fusion for high-frequency modules.
- Monitoring peak memory usage to avoid task interruptions caused by memory fragmentation.
It is important to emphasize that throughput optimization must be built on correctness and stability. If teams frequently change parallelism strategies, disable safety checks, or reduce logging in pursuit of throughput, they may increase the risk of long-duration training.
7. Checkpoints, fault recovery, and long-duration stable operation
Large-scale training is usually not completed in one run. It often consists of a long-term, multi-stage, continuously updated process.
During training, issues may occur, such as process exits, node failures, network jitter, data loading failures, or occasional numerical anomalies. Therefore, checkpoints and automatic recovery mechanisms are important components of a training adaptation system.
A complete checkpoint should usually include:
- Model parameters.
- Optimizer states.
- Learning rate scheduler state.
- Data loading position or training step.
- Random number generator states.
- Current parallelism strategy and model configuration information.
- Necessary validation information, such as file size and version number.
When saving checkpoints, it is recommended to use atomic writes. That is, data is first written to a temporary file, and the official checkpoint is replaced only after confirming that the write is complete. This helps avoid checkpoint corruption caused by abnormal training process exits.
For large-scale training, checkpoint files may be large, and the writing process may introduce noticeable delay. Therefore, different save frequencies can be set according to different training stages.
The automatic resume workflow usually includes:
- Read the latest valid checkpoint.
- Restore model parameters and optimizer states.
- Restore the training step and learning rate scheduler state.
- Restore the data loading position to avoid repeated training.
- Validate whether the current environment version is compatible with the checkpoint version.
- Continue training from the step after the checkpoint.
- Continuously record loss values and gradient norms after recovery to confirm that there are no abnormal jumps.
In a two-node sixteen-card environment, cross-node state consistency requires special attention. If a training job is interrupted due to a node failure, recovery should not simply load state from one card. Instead, it should be based on a globally consistent sharded checkpoint. Otherwise, some parameters may not be updated correctly, optimizer states may be misaligned, or data order may become inconsistent.
8. Monitoring and troubleshooting system
Long-duration training depends on a complete monitoring and troubleshooting system.
Without monitoring, when training fails, teams may only see the last error log and find it difficult to identify the real cause. Without a troubleshooting process, similar issues may repeatedly occur and increase trial-and-error costs.
At minimum, the following types of monitoring should be established:
- Training metric monitoring: Loss value, gradient norm, learning rate, training step, sample throughput, and other indicators.
- Resource metric monitoring: Accelerator utilization, memory usage, peak memory, process memory usage, and node load.
- Communication metric monitoring: Synchronization time, communication bucket size, cross-node latency, and timeout counts.
- Data metric monitoring: Data read speed, batch length distribution, empty sample ratio, and preprocessing time.
- Task status monitoring: Process liveness, checkpoint save status, abnormal exit count, and automatic recovery count.
For troubleshooting, it is recommended to follow the order: correctness before performance, single-card before multi-card, and small model before large model.
- Use a single card to verify model structure, loss functions, and forward/backward computation.
- Use multiple cards within one node to verify data parallelism, memory distribution, and gradient synchronization.
- Use multiple nodes to verify cross-node communication, parallelism strategies, and recovery mechanisms.
- Run a small-scale short training job to verify loss trends and gradient norms.
- Before scaling up, perform regression tests to confirm that configuration changes have not introduced anomalies.
- Before long-duration runs, enable automatic checkpoints, automatic recovery, and exception alerts.
For common anomalies, the following judgment paths can be used:
- If model loading fails, first check model definitions, weight formats, operator support, and initialization processes.
- If forward results are abnormal, first check input data, modality alignment, preprocessing logic, and random seeds.
- If gradient norms are abnormal, first check learning rates, gradient clipping, abnormal samples, and numerical precision.
- If training exits midway, first check communication timeouts, node resources, checkpoint writes, and the last logs.
- If loss becomes abnormal after recovery, first check checkpoint integrity, optimizer states, data loading positions, and parallelism strategy changes.
9. From “training-capable” to “out-of-the-box training”
In the early stage of training adaptation, the goal is often to complete one training loop.
As engineering capability improves, the goal should gradually shift toward “out-of-the-box training.”
Out-of-the-box training does not mean there is no adaptation cost. Instead, it means that mainstream model structures, common training configurations, and typical hardware environments have formed standardized workflows. Most scenarios can be validated through template-based configurations, automated checks, and canary releases.
Achieving this goal usually requires building the following capabilities:
- Model adaptation checklist: Clearly define model structure, inputs and outputs, parallelism strategy, memory estimation, operator dependencies, and recommended configurations.
- Automated test matrix: Cover different parameter scales, parallelism strategies, sequence lengths, and hardware configurations.
- Performance baseline library: Record throughput, memory usage, and stability metrics for different models under different configurations.
- Fault knowledge base: Collect common problems, localization methods, recovery strategies, and preventive measures.
- Configuration templates: Solidify validated data parallelism, fully sharded data parallelism, mixed precision, and checkpoint strategies into reusable templates.
- Cloud collaboration capability: Training jobs can use object storage for data and checkpoints, log services for runtime logs, monitoring and alerting for key metrics, container scheduling platforms for task management, and message queues to decouple preprocessing from training workflows.
These capabilities do not depend on any specific product. They represent general directions for large-model training engineering. Their core value is to reduce the adaptation barrier, decrease repeated trial and error, and move training from dependence on expert experience toward standardized processes.
10. Conclusion
Adapting hundred-billion-parameter large-model training on domestic AI accelerators depends not on a single demonstration or a short training result, but on whether teams can build a verifiable, monitorable, recoverable, and extensible engineering system.
When a two-node sixteen-card environment completes multiple consecutive optimizer updates while keeping loss values and gradient norms stable, it is a signal that this system is beginning to mature.
For developers and enterprise users, the real focus should be on the foundational issues of training adaptation: whether the model structure is fully covered, whether the parallelism strategy is reasonable, whether gradient synchronization is correct, whether numerical behavior is stable, whether checkpoints are reliable, whether faults can be localized, and whether training can be sustained.
Only by solidly building these foundational elements can large-model training on domestic AI accelerators gradually move from “can train” to “stable training,” “efficient training,” and eventually “out-of-the-box training.”
