Day-0 Adaptation for Trillion-Parameter LLMs: Sparse Attention and Low-Bit Inference in Practice

As large language models scale to trillions of parameters and widely adopt Mixture of Experts architectures with ultra-long context, the underlying computing infrastructure faces unprecedented challenges. Recently, a computing team successfully completed Day-0 adaptation for a 3-trillion-parameter LLM based on a fully self-developed hardware and software stack, covering the full service chain from model loading to multimodal inference. This effort not only validated the compatibility of domestic heterogeneous compute, but also provides practical solutions for developers dealing with sparse attention adaptation and low-bit computation in large-scale MoE deployment.

1. Adapting Ultra-Long Context and Sparse Attention Mechanisms to the Underlying Stack

Modern LLMs increasingly rely on dynamic sparse attention mechanisms such as DSA to support context windows of up to 1 million tokens. By dynamically selecting key tokens for computation, these mechanisms effectively control inference latency and memory usage. However, they also place stringent demands on operator libraries and computation path optimization in the underlying software stack.

1.1 Index Selection and KV Cache Layout Restructuring

In traditional dense attention, the KV cache is typically laid out sequentially by head dimension. Under sparse attention, however, compute units must extract KV blocks non-contiguously based on dynamically generated index masks. The software stack needs to implement a customized KV cache layout tailored to sparse indices, restructuring contiguous tensor storage into block-based sparse storage. This reduces memory fragmentation and improves bandwidth utilization during non-contiguous access.

1.2 Separate Adaptation for Prefill and Decode Computation Paths

LLM inference consists of two core stages with fundamentally different compute requirements:

• Prefill stage: compute-intensive. The stack fuses MATE sparse attention operators to combine Scale, Mask, and Softmax operations within multi-head attention into a single kernel, significantly reducing data movement between GPU and memory.

• Decode stage: memory-access-intensive. To accelerate this stage, the software stack integrates KDA state management to cache intermediate states from sparse attention, avoiding repeated computation of attention weights for historical tokens.

1.3 Cross-Layer Residual and Multimodal Attention Fusion

For multimodal LLMs that support image and text inputs, feature vectors generated by the visual encoder must interact with text features across layers. The adaptation team combined visual attention and cross-layer residual connections into a unified execution path. Through instruction stream reordering at the compiler level, they hid residual copy latency and ensured efficient alignment of multimodal data before it enters the text generation layers.

2. Dynamic Online Conversion of MXFP4 Low-Bit Expert Weights

To reduce memory usage, trillion-parameter MoE models often store expert weights in MXFP4 format. However, most current hardware natively supports only FP8 or higher precision compute units. Offline conversion of all FP4 weights to FP8 would require enormous storage overhead and effectively double memory consumption.

2.1 Online Conversion: Eliminating Storage Redundancy of Offline Conversion

To resolve this contradiction, the team abandoned the traditional path of pre-generating complete offline FP8 weights and instead introduced an online conversion mechanism. At the moment an expert network is activated, the system dynamically dequantizes the required MXFP4 weights into FP8 format in memory and feeds them directly into Tensor Cores for matrix multiplication. This process is implemented through operator-level fusion and introduces almost no additional computation latency.

2.2 Cross-Layer Reuse of the FP8 Workspace: Minimizing Memory Footprint

In MoE models, expert parameters are independent across layers, but only a small number of experts are active at any given moment during inference. Based on this characteristic, the software stack employs a memory management strategy of cross-layer FP8 workspace reuse. The system allocates only a single fixed-size FP8 buffer in memory. After the current layer completes computation, the buffer is released and reused by the next layer’s active experts. This pooling mechanism minimizes temporary memory overhead and makes deployment of trillion-parameter models feasible on single-card or 8-card clusters.

The following pseudocode illustrates the core implementation logic of online conversion and cross-layer reuse:

# Pseudocode: Online conversion and cross-layer reuse of MoE expert weights
class OnlineMoECalculator:
    def __init__(self, num_layers, fp8_workspace_size):
        # Allocate a globally shared FP8 workspace
        self.fp8_workspace = allocate_buffer(fp8_workspace_size)
    def forward(self, layer_idx, hidden_states, routing_indices):
        active_experts = get_active_experts(routing_indices)
        expert_output = []
        for expert_id in active_experts:
            # 1. Get the MXFP4 weight pointer for this expert in this layer
            mxfp4_weight = self.get_weight(layer_idx, expert_id)
            # 2. Online conversion: dynamically dequantize MXFP4 to FP8
            #    and write to the reused workspace
            fp8_weight = self.online_convert_mxfp4_to_fp8(
                mxfp4_weight, 
                output_buffer=self.fp8_workspace # Cross-layer reused workspace
            )
            # 3. Execute FP8 GEMM computation
            output = fp8_gemm(hidden_states, fp8_weight)
            expert_output.append(output)
        return aggregate(expert_output)

3. Full-Stack Compilation and Operator Ecosystem: The Foundation for Accelerated Day-0 Adaptation

The core challenge of Day-0 adaptation lies in seamlessly migrating model code written by the open-source community for mainstream ecosystems to a self-developed underlying architecture. Completing this process efficiently requires a comprehensive software stack and compiler system.

3.1 Synergy Between Operator Libraries and Domain-Specific Languages

The team provided highly optimized basic compute primitives through a low-level tensor operator library. At the same time, by supporting Triton-like domain-specific languages and low-level tensor programming languages, developers can write custom operators with minimal effort without diving into assembly-level instructions. For non-standard computation paths introduced by new model architectures, the collaboration between operator libraries and DSLs greatly shortened the development cycle.

3.2 Automated Optimization with the MTCC Compiler

The compiler plays a critical role in transforming high-level operator code into hardware instructions. Beyond instruction generation, it performs architecture-specific optimizations such as loop unrolling, constant folding, and memory bank conflict elimination. By automatically determining optimal tile sizes and pipeline scheduling strategies, the compiler enables migrated models to fully utilize the hardware’s peak compute capacity.

4. Summary and Outlook

This Day-0 adaptation of a trillion-parameter LLM demonstrates that domestic heterogeneous compute now has end-to-end engineering capabilities for handling cutting-edge LLM architectures. Through deep analysis of sparse attention memory access optimization and dynamic online conversion of MXFP4 low-bit weights, it is clear that future LLM inference optimization will increasingly depend on refined software stack design and automated compiler capabilities.

For cloud service providers building LLM compute infrastructure, the same need exists to address adaptation challenges across diverse model architectures and underlying hardware. By consolidating reusable operator libraries, enhancing container platform scheduling for heterogeneous compute, and integrating efficient logging and monitoring systems, providers can deliver a truly stable and efficient Day-0 model deployment experience for developers. As model architectures continue to evolve, deep software-hardware co-optimization will remain the only path to breaking through compute bottlenecks.