Day-0 Adaptation for Million-Context LLMs on Domestic GPUs: Technical Pathways and Engineering Practices

Background: The “Last Mile” Challenge in LLM Deployment

As the parameter scale of Large Language Models (LLMs) surpasses the trillion level, model deployment faces unprecedented engineering complexity. Take the recently released million-context window model as an example. Its core architecture employs a sparse attention mechanism (DSA) and mixed-precision expert weights (MXFP4). While this significantly boosts the model’s theoretical performance, it introduces a triple challenge for actual inference services:

1. Memory Layout Conflicts: Sparse attention causes conflicts between KV cache layout and traditional continuous memory allocation, requiring deep customization of the index computation path.
2. Low-Precision Execution: 4-bit expert weights cannot directly execute matrix operations on general-purpose computing platforms, necessitating an online conversion strategy.
3. Asynchronous Scheduling: The asynchronous characteristics of the Prefill and Decode phases require the inference engine to possess refined memory and scheduling management capabilities.

For computing infrastructure, Day-0 adaptation means completing the entire pipeline from model files to a serviceable inference link immediately upon the public release of model weights. This is not only a demonstration of technical capability but also a key metric for evaluating the maturity of AI infrastructure.

Overall Architecture: The Technical Foundation of a Unified Training and Inference Platform

Achieving rapid adaptation for such a large-scale model relies on a well-structured, highly customizable software stack. The overall architecture can be divided into three layers from bottom to top:

1. Hardware Layer: Utilizes intelligent computing cards designed specifically for unified training and inference scenarios. Their core advantage lies in supporting both training and inference workloads, avoiding the resource fragmentation caused by separating training and inference cards in traditional solutions. These computing cards are equipped with high-bandwidth memory and high-speed interconnect interfaces, providing the physical foundation for trillion-parameter model residency.
2. Compilation and Operator Layer: This layer is the core area of adaptation work. To support upstream community model implementations, the technical team built a toolchain containing four key components:
   • Basic Operator Library (MATE): Provides deeply optimized general matrix multiplication, normalization, activation functions, and other basic operators.
   • Domain-Specific Language Compilers (Triton-MUSA & TileLang-MUSA): Allow developers to write high-performance custom operators using Python-like syntax.
   • Low-Level Compiler (MTCC): Responsible for efficiently mapping high-level language code to the hardware instruction set.
3. Inference Service Layer: Based on a customized version of the mainstream SGLang inference framework (SGLang-MUSA), responsible for high-level logic such as request scheduling, batching strategies, and continuous batching.

Key Path 1: Comprehensive Adaptation of Sparse Attention Computation

The implementation of the million-context window relies on the sparse attention mechanism. Its core idea is that not every token needs to compute attention scores with all historical tokens; it only needs to focus on tokens at specific positions. This introduces the dynamism of index selection and the non-contiguity of the KV cache layout.

Index Selection and Cache Layout Reconstruction

The technical team first needed to handle the index selection phase before attention computation. Under the DSA mechanism, the model dynamically determines which historical KV entries to access based on the current generation position. Traditional continuous KV caching requires all historical states to be arranged sequentially along the sequence dimension, whereas sparse attention requires support for arbitrary jump-based access.

The specific engineering implementation方案 are as follows:

• MUSA Software Stack Adaptation Layer: Custom kernels were written to implement the dynamic generation and sorting of index vectors, ensuring that each attention head can independently complete key-value localization in sparse mode.
• Block-level KV Cache Management: The KV cache is divided into fixed-size blocks (e.g., 64×128 2D blocks), and a block-level index table is used to record the mapping between the physical address and logical sequence number of each block.

Prefill and Decode Dual-Stage Pipeline

Given that the Prefill phase requires parallel attention computation for all input tokens, while the Decode phase generates tokens sequentially, there is a fundamental difference in their computation modes. The technical team designed two independent computation paths:

• Prefill Phase: Adopts a dense computation mode, expanding sparse indices into a mask matrix, and utilizing a dedicated sparse operator library (MATE sparse attention operators) to achieve high-throughput parallel processing.
• Decode Phase: Only requires computing the attention of a single token with the KV cache. A lightweight sparse query kernel is used here to avoid redundant computation of a full mask.

Furthermore, to support visual modality input, the model introduces cross-layer residual connections and visual attention branches. The software stack must ensure that these two parts of computation can share the KV cache management mechanism with text attention, avoiding memory fragmentation.

Key Path 2: Online Conversion Strategy for Mixed-Precision Expert Weights

Current advanced LLMs generally adopt the Mixture-of-Experts (MoE) architecture, and the latest trend is to store expert weights in the MXFP4 (4-bit floating-point) format to drastically reduce memory usage. However, during actual computation, general matrix multiplication kernels typically require FP8 or BF16 format inputs.

Why Not Offline Conversion?

A seemingly straightforward approach is to pre-convert all MXFP4 weights to FP8 and store them during the model loading phase. However, calculations show that this solution has significant drawbacks:

• Doubled Memory Overhead: Converting all 3 trillion parameters to FP8 format would occupy a massive amount of additional memory.
• Drastically Increased Loading Time: The conversion process involves substantial data movement, extending service startup time from minutes to hours.

Online Conversion + Cross-Layer Workspace Reuse

The technical team adopted a dynamic conversion strategy, executing format conversion only when an expert is actually invoked:

# Pseudo-code illustration: Online conversion process
def expert_forward(expert_id, input_tensor):
    # Load MXFP4 format expert weights from memory
    mxfp4_weights = load_expert_weights(expert_id)  
    # Online convert to FP8 format (reusing unified workspace)
    fp8_weights = convert_mxfp4_to_fp8(mxfp4_weights, workspace)
    # Execute FP8 matrix multiplication
    output = fp8_matmul(input_tensor, fp8_weights)
    return output

The core design of this solution includes:

• Cross-Layer Reuse of Conversion Workspace: A fixed FP8 workspace is allocated and immediately released after each layer’s computation is completed, allowing it to be reused by the next layer. This ensures that peak memory does not grow linearly with model depth.
• LRU Cache Mechanism: For frequently invoked expert weights, their converted FP8 versions are cached to reduce redundant conversion overhead.
• Asynchronous Pre-conversion: Combined with the execution plan of the inference scheduler, expert weights scheduled to be called in the next step are converted in advance, achieving pipeline overlap between computation and conversion.

Performance Trade-offs and Optimization Suggestions in Engineering Practice

Three-Tier Model for Memory Planning

For a 3-trillion-parameter large model, memory planning must follow a three-tier strategy: “hot data resident, warm data loaded on-demand, cold data disk-mapped”:

1. Hot Data Layer (Resident Memory): FP8 weights of currently active experts, KV cache of all Attention layers, and key model tensors.
2. Warm Data Layer (Dynamic Memory-Host Swap): Original MXFP4 weights of other experts.
3. Cold Data Layer (Disk Mapping): Rarely invoked expert weights, loaded via asynchronous I/O only in special scenarios.

Key Metrics for Performance Tuning

During the actual adaptation process, the following performance metrics need continuous monitoring:

• Time to First Token (TTFT): Reflects the computational efficiency of the Prefill phase.
• Inter-Token Latency: Reflects the serial computation efficiency of the Decode phase.
• Service Startup Time: Covers model weight reading, format conversion, and graph compilation time.
• Effective Memory Utilization: The proportion of memory actually used for core computation.

Extracting a General Methodology

From this adaptation practice, a general methodology applicable to the adaptation of other large-scale models can be extracted:

1. Operator Difference List-Driven Development: First, compile a list of differences between the operators used by the model and the standard operator library, designing custom implementations for each differing operator.
2. Hierarchical Optimization of Computation Graphs: Complete operator replacement at the framework layer, and fuse redundant kernels at runtime through graph optimization techniques.
3. Progressive Verification Strategy: Verify single-layer correctness first, then multi-layer concatenation, and finally full-model testing to reduce debugging costs.

Conclusion and Future Outlook

The successful practice of this Day-0 adaptation demonstrates that through systematic software stack customization and refined engineering optimization, domestic intelligent computing platforms are fully capable of undertaking the deployment needs of frontier LLMs at the first opportunity. From index reconstruction for sparse attention to online conversion of mixed-precision weights, every step tests the engineering team’s dual understanding of hardware architecture and model characteristics.

In the future, as model scales continue to expand and new computation paradigms constantly emerge, the adaptation capability of AI infrastructure will become a core competitive advantage. Continued investment in foundational capabilities such as compiler technology, automatic operator generation, and memory management algorithms will be the necessary path for building an efficient large model service system.