Technical Path for Day-0 Adaptation of Trillion-Parameter Models on Domestic GPUs

1. Background: Trillion-Parameter Model Challenges and the Value of Day-0 Adaptation

When model parameters cross the trillion-level threshold, the engineering complexity of inference systems undergoes a qualitative shift. Dense or sparse Mixture-of-Experts (MoE) models with around 3 trillion parameters impose stringent requirements not only on memory capacity and bandwidth but also introduce new computational paradigms in attention mechanisms, expert routing, and cache management. For any new hardware platform, the ability to achieve “Day-0 adaptation”—meaning model loading, service startup, and the basic inference pipeline are fully operational immediately after model weights are released—has become a key indicator of software stack maturity.

The value of Day-0 adaptation lies not in benchmarking scores but in validating a complete engineering pathway: whether upstream community model code can be efficiently migrated to target hardware, if critical computational paths can be correctly mapped, and if the runtime can stably support long-context and multimodal inputs. Using the practice of adapting a 3-trillion-parameter model on a domestic AI training and inference compute card as a sample, this article abstracts a reusable technical path for developers engaged in building large-scale model inference systems.

2. Overall Architecture: From Community Code to Inference Service

2.1 Intermediate Layer Design for Upstream Integration

The greatest risk when adapting frontier models on new hardware is “rewriting.” Writing custom kernels for each model is unsustainable in terms of manpower and struggles to keep pace with community iteration. In practice, a layered strategy of “upstream model implementation + hardware backend adaptation” is typically adopted: the inference framework’s scheduling layer handles model structure definition, request orchestration, and batching logic, while the hardware side only needs to provide operator implementation and compiler support.

Following this path, the team leveraged a serving inference framework to承接 upstream model implementations. They completed the migration and integration of key computational paths for the new architecture using a proprietary operator library, a Triton backend tailored for the target hardware, operator generation capabilities based on TileLang, and an in-house compiler. The core philosophy of this combination is: keep the model code “as-is” and converge hardware differences at the operator and compiler layers, thereby transforming community model code into a practical inference service on the target hardware rather than a forked port.

2.2 Layered Breakdown of Adaptation Work

From an engineering perspective, a complete Day-0 adaptation can be broken down into four layers:

• Model Loading Layer: Weight format parsing, sharded loading, and memory layout planning.
• Computation Graph Layer: Operator mapping for critical paths such as attention, MoE expert computation, normalization, and residual connections.
• Runtime Layer: KV cache management, batch scheduling, and Prefill/Decode phase switching.
• Serving Layer: Request ingestion, streaming output, and multimodal input parsing.

A bottleneck in any of these layers will result in “service failing to start” or “abnormal inference results.” Therefore, adaptation work must aim to unblock the end-to-end pipeline rather than just getting individual operators to run.

3. Adaptation Practices for Critical Computational Paths

3.1 Implementing Sparse Attention

The model employs a sparse attention design. Its core lies in an index selection mechanism that computes attention only at key positions within long sequences, thereby controlling computational volume at a context length of around 1 million tokens. This mechanism imposes three specific requirements on hardware adaptation:

1. Operatorization of Index Selection: Index scoring and Top-K filtering require efficient reduction operators, and the data layout must remain consistent with subsequent sparse computations.
2. Non-Contiguous KV Cache Layout: Under sparse attention, the KV cache is no longer a simple contiguous tensor; it must be organized by block or index, requiring the cache manager to support non-contiguous addressing.
3. Separation of Prefill and Decode Paths: The Prefill phase features long sequences and compute-intensive operations, while the Decode phase features short sequences and memory-intensive operations. Their sparse computation patterns differ and require separate adaptations.

In practice, the software stack completed the adaptation of index selection, KV cache layout, and both the Prefill and Decode computational paths, integrating them with a proprietary sparse attention operator. This operator, alongside the state management module, cross-layer residual connections, and visual attention module, forms a complete execution chain—meaning both text and image inputs share the same attention infrastructure, requiring no additional branches for multimodal inference.

3.2 Online Conversion of Low-Precision Expert Weights

In MoE models, expert weights consume the majority of memory. The model stores expert weights in MXFP4 format, a 4-bit low-precision floating-point format. Its advantages are low memory consumption and fast loading, but it requires conversion to higher precision during computation.

Two common processing methods are offline pre-conversion (generating a complete high-precision weight copy in advance) and online conversion (converting on-the-fly during computation). The former introduces additional memory overhead and storage costs, which is often unfeasible for a 3-trillion-parameter model.

The team adopted an online conversion and cross-layer workspace reuse strategy: during expert computation, low-precision weights are instantly converted to FP8 precision for calculation, reusing a cross-layer shared FP8 workspace to avoid allocating conversion buffers for each layer. The engineering benefits of this design are twofold: eliminating the need to generate full offline high-precision weights significantly reduces peak memory usage; meanwhile, cross-layer workspace reuse reduces the frequency of memory allocation and deallocation, enhancing stability during long-sequence inference.

The pseudo-code below illustrates the basic concept of online conversion and workspace reuse:

# Pseudo-code: Online low-precision conversion for expert computation
workspace = alloc_fp8_workspace(max_expert_chunk)  # Cross-layer shared workspace
def expert_forward(x, expert_weight_mxfp4):
    # Chunked conversion to avoid expanding all weights at once
    for chunk in split_chunks(expert_weight_mxfp4, workspace.capacity):
        w_fp8 = convert_to_fp8(chunk, out=workspace)  # In-place write to shared workspace
        y = matmul(x_chunk, w_fp8)
    return y

It is worth emphasizing that online conversion places higher demands on operator fusion: the conversion and matrix multiplication must be pipelined as much as possible; otherwise, the conversion overhead will offset the benefits of low-precision storage. Optimization in this area typically relies on the compiler’s scheduling capabilities for the computation graph.

4. Engineering Essentials and Common Experiences

4.1 Synergy Between Compiler and Operator Generation

Operator development for new hardware follows two paths: hand-written kernels and DSL-based generation. The former offers controllable performance but high manpower costs, while the latter iterates quickly but depends on compiler maturity. In practice, the team utilized both the Triton backend for the target hardware and TileLang-based operator generation, completing scheduling optimizations via their in-house compiler. Their experience suggests: first use DSL to quickly cover long-tail operators, then replace performance hotspots with hand-written operators, forming a gradient strategy of “breadth via generation, depth via hand-writing.”

4.2 Memory Planning for Long Contexts

A 1-million-token context means the KV cache alone can reach tens of gigabytes. Adaptation requires advance planning: cache chunk size, whether to use paged management, and the additional metadata space occupied by sparse index structures. It is recommended to perform a memory budget modeling before service startup, estimating weights, KV cache, workspace, and temporary buffers separately to avoid Out-Of-Memory (OOM) errors during execution.

4.3 Day-0 Adaptation Verification Checklist

A deliverable Day-0 adaptation should pass at least the following checks:

• Model weights are fully loaded without precision overflow or format errors.
• The service starts normally, and health checks pass.
• Text inputs yield reasonable outputs in both single-turn and multi-turn conversations.
• The basic inference pipeline for image inputs is functional.
• Long-context inputs do not crash, and outputs show no significant degradation.
• Prefill and Decode phase switching operates normally, with no interference under batching.

4.4 Takeaways for Developers

Three general principles can be extracted from this practice:

1. Adapt, Don’t Rewrite: Prioritize reusing upstream model implementations and converging differences at the operator and compiler layers to reduce maintenance costs.
2. End-to-End Priority: Unblock the load-startup-inference pipeline first, then optimize performance point by point to avoid getting trapped in local optima of single-operator tuning.
3. Memory is a First-Class Citizen: The bottleneck for trillion-parameter models is often memory capacity, not compute power. Low-precision storage, online conversion, and workspace reuse are essential engineering techniques.

5. Conclusion

Day-0 adaptation for trillion-parameter models is essentially a full-pipeline stress test for a hardware software stack. It tests not only single-point operator performance but also the overall capability of model integration, compiler scheduling, and memory management. The paths outlined in this article—sparse attention adaptation, online conversion of low-precision expert weights, layered adaptation, and the verification checklist—are broadly applicable and can serve as a reference framework for developers adapting frontier models on new hardware platforms. As model structures and precision formats continue to evolve, this methodology of “integrating upstream, converging differences, and verifying end-to-end” will remain the core philosophy of inference system engineering.