Day-0 Adaptation for 3-Trillion-Parameter LLMs: Enabling Text and Vision Inference on Domestic GPUs

Background: Deployment Challenges of 3-Trillion-Parameter Models

As large language models enter the trillion-parameter era, their deployment and inference face unprecedented engineering challenges. Take a recently released 3-trillion-parameter model as an example. It supports a 1-million-token context window and introduces a hybrid attention mechanism—combining traditional full attention with Deep Sparse Attention (DSA). It also utilizes MXFP4 format for expert weights to reduce storage pressure. While this complex architecture significantly improves performance in code generation, complex reasoning, and long-horizon agentic tasks, it places extreme demands on the underlying compute platform:

• VRAM Capacity Bottleneck: Even with 4-bit quantization, storing 3 trillion parameters requires terabytes of VRAM, far exceeding the capacity of a single card.
• Hybrid Compute Requirements: The model combines dense matrix computations (full attention, feed-forward networks) with sparse computations (sparse attention indexing, expert routing). The compute platform must efficiently switch between these two workloads.
• Dynamic Inference Pipeline: In long-context scenarios, KV cache management, cross-layer residual connections, and vision-text modality fusion are deeply intertwined. The inference engine must flexibly orchestrate the computation graph.

Against this backdrop, a domestic AI accelerator vendor announced the Day-0 adaptation of this model using its self-developed AI training and inference card and accompanying software stack. Day-0 adaptation means that on the day of the model’s official release, the complete pipeline—from weight loading and service startup to text and image inference—was fully operational. This achievement provides a crucial reference for localizing ultra-large model deployments.

I. Overall Technical Path for Day-0 Adaptation

The core goal of Day-0 adaptation is to efficiently migrate open-source community model code to target hardware platforms and provide stable inference services within a highly compressed timeframe. The adaptation process follows the principle of “compatibility first, layered optimization,” with a technical architecture comprising three layers:

1.1 Framework Layer: Reusing Mature Inference Engines

The adaptation did not build the inference service from scratch. Instead, it leveraged mainstream open inference frameworks (such as SGLang) to host the upstream model implementation. The framework inherently provides model definition, scheduling strategies, and batching capabilities. The adaptation team’s primary work focused on mapping the framework’s operator dispatch logic to the target hardware via the self-developed software stack’s interface layer.

The key point is that the software stack offers runtime interfaces highly compatible with industry-standard programming models, allowing framework-side code to migrate without significant modifications. This means that optimization patches, inference configurations, and community experience accumulated around the model can be smoothly transitioned to the domestic hardware platform.

1.2 Operator Layer: Deep Customization of Key Compute Paths

Not all computations in the model can be directly covered by existing operator libraries. Unconventional operations newly introduced by the model, such as sparse attention and MXFP4 weight unpacking, must be handled separately at the operator layer. The adaptation team used a combination of operator compilers and manual optimization to customize the core compute paths:

# Pseudocode: Sparse attention compute path adaptation
def sparse_attention_with_dsa(query, key, value, sparse_indices):
    # 1. Select KV pairs participating in the computation based on sparse indices
    selected_keys = key.gather(sparse_indices)
    selected_values = value.gather(sparse_indices)
    # 2. Call the deeply optimized sparse attention operator
    output = sparse_attention_kernel(
        query, selected_keys, selected_values,
        layout=KV_CACHE_LAYOUT_DSA  # KV cache layout adapted for DSA
    )
    # 3. Residual fusion with the full attention branch results
    return output + full_attention_branch(query, key, value)

This process involved three core decision points:

• Index Selection Strategy: Under the DSA mechanism, each query token must dynamically select a set of sparse indices to attend to based on positional encoding and content features. The adaptation team pushed this logic from the Python layer down to the kernel layer to avoid frequent CPU-GPU data transfers.
• KV Cache Layout Rearrangement: For sparse attention, the KV cache is no longer a simple contiguous tensor but a fragmented structure organized by index blocks. A completely new memory layout and allocation strategy had to be designed.
• Prefill/Decode Separated Computation: The prefill phase and the token-by-token generation phase have entirely different access patterns for sparse attention. Separate computation graphs must be constructed, and optimal kernels selected for each.

1.3 Deployment Layer: Online Conversion Strategy for Ultra-Large Weights

Faced with MXFP4 expert weights (mixed-precision floating-point 4-bit), the conventional approach is to offline-convert all weights to the target precision (e.g., FP8) before loading them for inference. However, for a 3-trillion-parameter model, offline conversion takes hours and requires staggering amounts of temporary storage space.

The adaptation team adopted an innovative solution combining online conversion with cross-layer reuse: during model loading, only the expert weights required for the current layer are unpacked in real-time, converted to FP8 format, and sent directly to the compute unit. Simultaneously, an FP8 workspace pool was designed to reuse allocated buffers during layer transitions, avoiding repeated VRAM allocation and deallocation. This approach compressed deployment time from hours to minutes, making Day-0 delivery possible.

II. Complete Adaptation Scheme for Hybrid Attention Mechanisms

Another core technical feature of the model is its hybrid attention architecture, which simultaneously contains standard full attention and DSA-based sparse attention channels. While this design improves long-text processing efficiency, it also increases the complexity of the inference engine.

2.1 State Management for Cross-Layer Residual Connections

In the hybrid attention architecture, the input of each Transformer layer must aggregate the previous layer’s output, the sparse attention branch’s results, and visual features (when processing mixed image-text inputs). To achieve this, the adaptation team implemented a unified state management module responsible for maintaining intermediate states passed across layers and reducing frequent VRAM allocations through a custom memory pool strategy.

In practice, the most complex scenario this module handles is the hybrid attention of visual and text tokens: image regions are split into several visual tokens, concatenated with text tokens, and fed uniformly into the attention computation. However, visual tokens typically do not need to be generated. Therefore, during the Decode phase, they must be prioritized for eviction from the KV cache to save VRAM. This “visual token-aware cache eviction” strategy is a critical component of the model’s inference performance optimization.

2.2 Engineering Implementation of Sparse Attention Operators

The implementation of sparse attention operators involves multi-level index data structures. First is the coarse-grained block-level index, used to quickly locate candidate KV blocks. Second is the fine-grained token-level selection, which further picks the KV pairs within the block for computation. During implementation, the operator utilizes a two-level lookup table and parallel reduction to achieve efficient data exchange within GPU thread blocks.

It is worth noting that the performance characteristics of sparse attention differ significantly between the Prefill and Decode phases:

• Prefill Phase: The input sequence is long, and all tokens compute attention in parallel. The overhead of constructing sparse indices can be effectively amortized. The focus here is optimizing the memory access locality of the index matrix.
• Decode Phase: Only one token is generated at a time, and the dynamic nature of sparse indices is extremely strong, making pre-caching impossible. The key optimization point for the operator here is the pipeline overlap between index generation and KV cache access.

The adaptation team designed different kernel implementations for both phases and exposed them transparently to the upper layer through a unified scheduling interface.

2.3 Coordination with the Visual Attention Module

When the model processes mixed image-text inputs, in addition to the text’s DSA attention, it must also perform visual attention computation for image regions. The latter is characterized by a relatively small number of participating tokens, but a higher feature dimension per token. To address this, the adaptation team implemented a dedicated visual attention kernel within the unified operator framework, sharing KV cache management logic with the main sparse attention path to avoid VRAM waste caused by two independent memory systems.

III. Engineering Practices for Mixed-Precision Inference

Handling MXFP4 expert weights is another major technical highlight of this adaptation. Unlike conventional weight quantization schemes (like INT8 or FP8), MXFP4 employs a block-wise scaling strategy, where every 32 weight elements share a single scaling factor. This preserves the numerical dynamic range as much as possible while maintaining lower precision.

3.1 Online De-quantization Compute Flow

In the adapted implementation, expert weights are stored globally in VRAM in MXFP4 format. When an expert is activated by the router, the inference engine performs the following steps:

1. Load compressed weights: Read the original byte stream in MXFP4 format (4 bits per weight).
2. Unpack and de-quantize: Unpack the 4-bit values into a higher-precision representation, multiply by the corresponding scaling factor, and obtain FP8 weights.
3. Execute matrix multiplication: Feed the FP8 weights and activations into the Tensor Cores for highly efficient computation.
4. Write back results: If the computation result is needed as activations for the next layer, it remains in FP8 format to save bandwidth; if it is the final output, it is upcasted to a higher precision.

The advantage of this flow is that it avoids large-scale format conversion for the entire model’s weights. Each expert is only unpacked on a small scale when needed. Furthermore, if the same expert is activated multiple times consecutively during inference, the unpacked FP8 weights can be temporarily stored in the workspace for reuse.

3.2 Cross-Layer FP8 Workspace Reuse Strategy

Because the model uses a Mixture-of-Experts (MoE) architecture, the number of experts across different layers is massive, but only a small fraction is activated during each inference pass. Allocating a separate FP8 conversion buffer for each expert would cause significant VRAM waste. The adaptation team adopted the following strategy:

• Maintain a globally fixed-size FP8 workspace pool (sized according to the weight volume of the largest single expert).
• Each time an expert is activated, request a block from the pool to store the unpacked weights; release it back to the pool after inference is complete.
• Since expert computations across different layers are serial in time (due to inter-layer dependencies), the same workspace block can be safely reused across layers.

Experiments showed that this strategy reduces additional VRAM overhead from the theoretical upper limit (unpacking all experts) down to the volume of a single expert, representing only a minute fraction of the model’s overall VRAM consumption.

IV. Performance Tuning and Best Practices

Based on the adaptation experience above, the following reusable best practices are summarized for deploying ultra-large 3-trillion-parameter models:

4.1 Memory Planning Strategy

• Layered Weight Residency: Frequently activated shared experts (e.g., general knowledge experts) reside permanently in VRAM; sparsely activated domain experts are loaded on-demand and stored in NVMe storage via memory-mapped files.
• Activation Compression: Enabling FP8 activation compression at residual connections can reduce activation VRAM usage by approximately 40%, with negligible impact on precision.
• Dynamic KV Cache Adjustment: Dynamically allocate KV cache blocks based on the context length of different requests to avoid idle waste caused by over-provisioning.

4.2 Service Scheduling Optimization

• Request Batching: Batching long and short requests with similar contexts can significantly improve the utilization rate of sparse attention operators.
• Separation of Vision and Text Requests: The inference paths for pure text requests and mixed image-text requests differ greatly. Processing them in separate queues at the scheduler level prevents mutual interference.
• Warm-up Mechanism: After service startup, execute a few standard inference requests first to populate operator caches and memory pools, avoiding excessively high latency for the first request.

4.3 Observability and Troubleshooting

• Operator-Level Performance Monitoring: Enable timeline tracing in the software stack to record execution time, VRAM usage, and occupancy rate for each operator, facilitating rapid bottleneck identification.
• Weight Loading Verification: Calculate hash values for key tensors during model loading to ensure numerical correctness during the online de-quantization process.
• Layered Logging: Framework, operator, and driver layers log events at three distinct levels, balancing troubleshooting efficiency with log file size.

V. Summary and Outlook

This article systematically outlines the complete technical path for achieving Day-0 adaptation of a 3-trillion-parameter large language model on a domestic AI accelerator card. The core experiences can be summarized in three points: First, a highly compatible software stack is the cornerstone of rapid adaptation. Reusing industry-standard interfaces and framework implementations can significantly compress the migration cycle. Second, deeply customized operator implementations are critical for performance assurance. Special optimizations tailored to hardware characteristics are necessary, especially for the model’s sparse attention and new quantization formats. Third, innovative memory management strategies are prerequisites for deploying ultra-large models. Online conversion and workspace reuse effectively bypass the VRAM wall problem.

Looking ahead at future evolution, on one hand, as model sizes continue to grow, communication optimization for multi-card parallel inference will become a research focus in the next phase. On the other hand, designing hierarchical KV cache management strategies for inference scenarios with even longer contexts (millions of tokens) holds significant engineering exploration value. It is foreseeable that with the co-evolution of domestic compute platforms and large model ecosystems, Day-0 adaptation will become an industry standard capability, providing a solid foundation for the rapid deployment of intelligent applications.