Day-0 Adaptation of Ultra-Large Sparse Models on Domestically Built Integrated Training-Inference Accelerators

1. Background: Inference Challenges Posed by Trillion-Parameter Sparse Models

As large model parameter counts move from hundreds of billions toward trillions and beyond, model architectures continue to evolve. Sparse attention, mixture-of-experts, dynamic computation paths, and low-precision weight representations have become important mechanisms for balancing model capability against inference cost. At the same time, model release cycles are accelerating. Many checkpoints enter production validation almost immediately after being published, which means infrastructure teams must complete the entire process from model loading and service startup to a working baseline inference pipeline within a very short window. This Day-0 adaptation capability has become a key indicator of the maturity of an AI compute platform and its software stack.

For domestically built integrated training-inference accelerators, Day-0 adaptation is usually more challenging than on general-purpose platforms. New models are often first implemented against operators, communication libraries, and inference frameworks from a specific ecosystem. The domestic software stack must additionally handle computation graph migration, operator mapping, memory layout adjustment, and numerical alignment. In the case of trillion-parameter sparse models, weights cannot simply be loaded onto a single accelerator or a single node, and inference involves complex dynamic routing and state management. Therefore, completing the full loading, startup, and inference path within a short period requires a highly coordinated software stack design.

This article approaches the problem from a practical engineering perspective and outlines the critical path for completing Day-0 adaptation of a trillion-parameter sparse model on a domestically built integrated training-inference accelerator. The discussion is not tied to any specific product or version. Instead, it focuses on reusable methodology and implementation considerations for AI infrastructure teams.

2. Adaptation Goals and Overall Approach

The model targeted in this adaptation has several notable characteristics. First, its parameter scale reaches approximately three trillion, so the full model cannot reside in the memory of a single accelerator. Second, it supports context lengths on the order of one million tokens, which imposes stricter requirements on KV cache management. Third, it uses sparse attention mechanisms, so its computation path differs significantly from standard dense attention. Fourth, expert weights are stored in a low-precision quantized representation, which means precision conversion must be performed online during inference.

The adaptation goal is not to achieve extreme performance tuning in a single pass. Instead, the priority is to establish a minimal closed loop covering model loading, service startup, and basic text and image inference. This goal can be broken down into three levels:

1. Functional correctness: Model outputs align with the upstream reference implementation, making inference results usable.
2. Execution connectivity: All critical computation paths execute on the target device without crashes or hangs caused by missing operators or inconsistent memory layouts.
3. Service availability: The model can be registered as an inference service and respond to requests, establishing a baseline for subsequent performance optimization.

Based on these goals, the adaptation team used a localized version of a general-purpose inference framework as the hosting layer. The framework already supports common large-model inference capabilities, including model parallelism, pipeline parallelism, expert routing, and dynamic batching. The core adaptation work is to connect the computation paths and data flows required to run the upstream model on the target software stack, rather than rebuilding a complete inference system from scratch.

The following sections cover model loading, sparse attention adaptation, low-precision expert weight processing, and vision and multimodal paths.

3. Model Loading: Cross-Layer Sharding and State Management Initialization

A three-trillion-parameter model cannot be fully loaded onto a single accelerator. Its weights must be sharded across multiple accelerators or nodes. Inference frameworks typically distribute parameters using layer-wise sharding, expert-wise sharding, or hybrid sharding. During the Day-0 adaptation phase, the weight sharding scheme can follow the structure used by the upstream model release. However, certain layout decisions may need to be adjusted based on the memory capacity and interconnect bandwidth of the target device.

Specifically, the mixture-of-experts component accounts for a very large share of the total parameters, and a single layer may contain many expert modules. During loading, each expert needs a unique storage location and routing index. If a global expert-parallel strategy is used, all expert weights can be pooled across a group of devices and dynamically assigned by a scheduler according to incoming requests. If an expert-sharding strategy is used, each device permanently holds a subset of expert weights and cross-device reads occur during routing. For the Day-0 stage, it is advisable to prioritize a sharding scheme consistent with the upstream structure to reduce state management complexity.

At the same time, the software stack on the target device must provide a unified description of cross-layer residual connections and KDA state management. KDA can be understood as a dynamic state representation that is updated as execution progresses through layers, supporting KV cache compression, context extension, or inter-layer information passing. During initialization, the KDA state must be allocated in a contiguous and indexable buffer, and its lifecycle must be maintained throughout forward propagation. For million-token contexts, state buffers can occupy a substantial amount of memory and must be planned together with the KV cache to avoid memory fragmentation.

A successful model-loading stage does not simply mean that all parameters have been written to devices. It means the inference framework can correctly locate all relevant weights and state buffers for a given request and complete the first forward pass. Therefore, the loading path must be validated together with the subsequent service startup path.

4. Service Startup: Connectivity Design for the Minimal Inference Path

After model loading, the next step is to build an inference service that can respond to standard requests and return results. For Day-0 adaptation, service startup does not need to support full production-grade features such as online scheduling, request queuing, and automatic scaling. However, it must cover the following aspects:

1. Request entry and output exit: The service must accept text input and return text output. For multimodal models, it must also support an image input path.
2. Computation graph binding: The symbolic computation graph from the upstream model must be bound to operator implementations on the target device so the executor can find a low-level implementation for every path.
3. Dynamic memory management: KV caches, activations, and expert workspaces must be dynamically allocated and released during inference. A reasonable memory pool and reclamation policy must be configured.
4. Runtime error visibility: Basic logs and error messages must be clear enough to help locate the first failing operator or an out-of-bounds memory access.

To isolate problems quickly, adaptation teams usually start with a single request, single concurrency, and short output. They first validate the prefill phase, then the decode phase, and finally continuous multi-round generation. For multimodal input, it is better to bring up the text path first and then add the image encoding path, avoiding the complexity of debugging two complicated pipelines simultaneously.

During service startup, the compiler and operator library in the software stack play a central role. Upstream model code is usually written for a general-purpose GPU ecosystem, and the target device must convert it into locally executable code. This conversion involves graph optimization, operator matching, memory allocation, and precision control. At the Day-0 stage, the conversion path should be kept simple and reliable, without introducing aggressive optimizations that increase uncertainty.

5. Sparse Attention Adaptation: From Index Selection to Computation Path Reconstruction

Sparse attention is one of the defining characteristics of this model. Unlike standard dense attention, sparse attention performs index selection before computation based on the relevance between queries and keys. Only a subset of KV entries participates in the attention calculation, reducing computational complexity for long contexts. Adapting sparse attention requires addressing three areas.

5.1 Index Selection Path

Index selection determines which KV entries participate in a given computation. The selection process may be based on attention scores, positional distance, clustering results, or learned sparsity patterns. On the target device, index selection needs to work with the collective communication and sorting capabilities provided by the software stack. When KV caches are stored across multiple accelerators, index selection may also involve cross-device querying and aggregation. Therefore, during Day-0 adaptation, correctness of index selection should be ensured before optimizing index construction performance.

5.2 KV Cache Layout

Sparse attention imposes specific requirements on KV cache layout. Dense attention typically organizes caches by layer, batch, sequence length, and head dimension. Sparse attention may organize caches by block or by index so that selected KV segments can be retrieved quickly. On the target device, the cache layout must be designed to fit the memory model of the software stack. For example, KV entries can be organized into fixed-size blocks so that index selection results map directly to block offsets, avoiding element-by-element reads and writes.

5.3 Prefill and Decode Computation Paths

During prefill, attention must be computed over the entire input sequence at once, while decode generates tokens step by step. Under sparse attention, both phases must first determine which KV ranges participate in the computation. Prefill usually retains a relatively large number of KV entries, so sparsity benefits are limited. Decode, by contrast, only needs to access a small number of KV entries per step and is therefore the phase that benefits most from sparsity. Day-0 adaptation must ensure both paths execute correctly and share the same index selection and cache reading logic, avoiding errors caused by prefill and decode using different state formats.

In the target software stack, the sparse attention operator should be written as a unified interface that also supports a dense fallback mode. When the index selection strategy is unavailable or the model enters a special mode, the operator can temporarily fall back to dense computation, which makes problem diagnosis easier.

6. Low-Precision Expert Weight Processing: Online Conversion and Workspace Reuse

In a three-trillion-parameter model, expert weights are stored at scale using a low-precision representation to reduce storage and communication overhead. The weight precision used by this model can be broadly described as a 4-bit floating-point format, but actual computation typically requires higher-precision inputs. Therefore, inference must convert between low-precision storage and higher-precision computation.

A key design decision is not to pre-generate full high-precision weight copies. For a three-trillion-parameter model, offline conversion of all expert weights into a higher-precision format would require enormous storage and conversion time, making it difficult to meet the Day-0 fast startup goal. The adaptation team adopted an online conversion approach: before each expert computation, the required low-precision weights are dynamically converted to the computation precision and used directly. Conversion results are stored in a reusable workspace rather than persisted.

This approach offers two benefits. First, storage pressure drops significantly because no additional high-precision weight files need to be saved. Second, initialization time is much shorter because the full conversion pass is eliminated. The trade-off is that partial conversion must be performed during every computation, increasing runtime overhead. However, at the Day-0 stage, loading and startup speed are usually more important than per-call peak performance, so this trade-off is reasonable.

In terms of implementation, the workspace uses a cross-layer reuse strategy. Memory required for higher-precision intermediate formats such as FP8 is defined as a global workspace, and different layers reuse the same memory region at different times. This keeps peak memory usage manageable while reducing fragmentation caused by frequent allocation and deallocation. The workspace size depends on request batch size, sequence length, and expert routing behavior, so capacity must be estimated before deployment.

7. Multimodal Path: Unified Execution of Text and Vision Branches

In addition to text input, the model supports image input, making it a multimodal inference system. Multimodal scenarios require separate processing of the text encoding branch and the vision encoding branch, with feature fusion at specific positions. For Day-0 adaptation, the key point is that vision attention and text attention share the same underlying execution mechanism, avoiding the maintenance of separate operator paths for the two modalities.

Vision input first passes through a vision encoder to generate visual feature representations. This process typically includes several convolution or attention operations, and the output features then enter the main LLM together with text features. Because the main model uses sparse attention, the KV cache and index selection for visual features must also support image features. During adaptation, it is necessary to ensure that vision attention uses the same sparsification strategy as text attention and that visual features are correctly propagated through cross-layer residuals and state management.

Baseline inference validation can start with a single text input and then extend to a single image plus text input. Image preprocessing, resizing, and feature conversion should use general-purpose pipelines where possible, avoiding acceleration libraries that depend heavily on specific hardware and thus reducing Day-0 debugging complexity. Once both text and vision main paths are confirmed to be correct, performance analysis can begin.

8. Software Stack Collaboration: Compiler, Operator Library, and Runtime Management

This adaptation involves collaboration across multiple software stack components, including an operator template library, programming interfaces, a domain-specific language compiler, and a low-level code generator. Clear separation of responsibilities among these components is essential for supporting a new model quickly.

The operator template library provides device-specific implementations of common operators, including matrix multiplication, normalization, activation, and attention. For the new sparse attention operators required by the model, corresponding implementations must be added to this library while keeping the input and output interfaces consistent with the general framework. Programming interfaces and the domain-specific language are used to quickly develop computation paths that cannot fully reuse standard operators. Through the high-level programming model, developers can describe custom computations in a script-like manner, and the compiler then generates device code. This approach can significantly shorten the development cycle for new operators.

During adaptation, the compiler is used to convert upstream model code into an execution plan for the target device. It must understand model structure, operator dependencies, and memory requirements, and produce execution units that can be scheduled across multiple accelerators. The compiler does not necessarily need to generate an optimal schedule in a single pass, but it must produce a runnable and debuggable schedule. At the Day-0 stage, it is advisable to preserve more intermediate representations and debugging information so that errors can be diagnosed layer by layer.

Runtime management handles memory allocation, stream control, and multi-accelerator synchronization during execution. For mixture-of-experts models with dynamic routing, the runtime must also support request-level scheduling and expert workspace distribution. Day-0 configurations can use simplified scheduling strategies, but the interface design should preserve the possibility of integrating more complex strategies later to avoid large-scale refactoring.

9. Common Issues and Troubleshooting Suggestions

During the adaptation process described above, engineers commonly encounter computation graph binding failures, inconsistent cache layouts, expert routing errors, and excessive numerical deviations. These problems often do not surface as compiler errors. Instead, they appear at runtime as null pointers, out-of-bounds access, or abnormal outputs. The following are several troubleshooting suggestions.

First, create a minimal reproducible script. Reduce the model to a single layer or a single expert module and validate output shapes and value ranges with fixed inputs. This helps quickly determine whether the issue lies in model structure parsing, operator implementation, or runtime scheduling.

Second, align intermediate results. Save intermediate tensors at key positions and compare them layer by layer with the reference implementation. Start by comparing input token embeddings and first-layer outputs, then move forward progressively. Allow reasonable floating-point error during comparison, but shapes and signs must match.

Third, pay close attention to indices and offsets. Both sparse attention and expert routing involve explicit index calculations. Many problems originate from slice boundaries or incorrect index base addresses. It is recommended to add assertions at index generation points to check that all indices satisfy upper-bound constraints.

Fourth, clear reused workspaces or record their lifecycles. When reused workspaces are used for online low-precision weight conversion, stale data from a previous computation may cause hidden contamination if it is not fully overwritten. The workspace reuse policy should explicitly define the write range and lifecycle of each memory region.

Fifth, prioritize functional correctness before memory optimization. At the Day-0 stage, overly aggressive memory planning frequently causes out-of-bounds access. It is acceptable to first increase memory headroom and then tighten it gradually after key metrics become normal.

10. Summary and Future Optimization Directions

This article has examined the Day-0 adaptation of a three-trillion-parameter sparse model on a domestically built integrated training-inference accelerator, covering the complete execution path from model loading, service startup, sparse attention, and low-precision expert weights to multimodal inference. The key to successful adaptation is not the peak performance of a few individual operators, but whether the software stack components can form a collaboration mechanism that is locatable, verifiable, and extensible.

Future optimization can proceed in the following directions:

1. Sparse attention performance optimization: Focus on index graph construction, block reads, and computation fusion to achieve more significant acceleration for long-context scenarios.
2. Eliminating repeated online weight conversion overhead: Cache conversion results for frequently used experts or introduce asynchronous conversion to reduce the impact of repeated conversion on prefill and decode throughput.
3. Enhanced expert parallelism and scheduling: Implement finer-grained expert placement and dynamic load balancing based on request load and device compute capacity.
4. Memory pool optimization: Further compress KV caches and state caches, and explore cache eviction and hierarchical management to support higher concurrency and longer contexts.
5. Multimodal path fusion: Unify the computation graph representation of text and vision to reduce branch differences and lower ongoing maintenance costs.

Day-0 adaptation is not a one-time project. It is a capability accumulated through the continuous evolution of AI infrastructure. By systematically capturing adaptation experience for new model architectures, improving software stack collaboration efficiency, and increasing automation, teams can complete deployment validation faster when future models are released and gain an early advantage in model inference services. For organizations, sustained investment in an in-house software stack and engineering capability remains a necessary path to building long-term competitiveness.