Day-0 Adaptation Practices for Ultra-Large-Scale MoE Models on Domestic Full-Function GPUs

1. Background: When Model Scale Crosses the Trillion Parameter Threshold

As large model parameter scales transition from hundreds of billions to the trillion level, the fundamental bottleneck structure of inference systems undergoes a paradigm shift. In the era of dense models, compute power and memory bandwidth were the primary constraints. However, for models with trillions of parameters utilizing the Mixture-of-Experts (MoE) architecture, the system faces a tightly coupled set of challenges:

• Weight Volume Far Exceeds Single-GPU Memory: Even with low-precision storage, complete weights require aggregation across multiple GPUs and machines. The model loading process itself becomes a distributed engineering challenge.
• Irregular Memory Access from Sparse Activation: Each token is routed to only a minimal subset of experts, making the access pattern for expert weights highly discrete and imposing stringent demands on the random read performance of the memory subsystem.
• Long Context Amplifies KV Cache Pressure: Million-level context lengths mean that KV cache capacity and lifecycle management become independent design topics.
• Implementation Costs of Low-Precision Weight Formats: The low-precision expert weight formats used by model publishers often do not match the native compute precision supported by the hardware, requiring additional conversion and scheduling strategies.

“Day-0 adaptation” refers to completing the inference pipeline setup on the exact day the model weights are publicly released. This is not merely a demonstration of engineering speed; it indicates that the software stack possesses sufficient foresight and abstraction capabilities to quickly integrate upstream community implementations without requiring deep, model-specific customizations.

2. Overall Adaptation Strategy: Layered Integration and Progressive Replacement

When facing an entirely new model architecture, a rational adaptation path is not to “rewrite from scratch.” Instead, the approach is to integrate upstream community implementations layer by layer, progressively replacing critical compute paths with hardware-optimized implementations. The process can be divided into four layers:

1. Model Logic Layer: Reuse the upstream inference framework’s structural description of the model (forward computation graph, routing logic, attention organization) to ensure semantic correctness.
2. Operator Implementation Layer: Map critical operators in the computation graph (sparse attention, expert computation, normalization) to the target hardware’s high-performance operator libraries.
3. Compilation and Code Generation Layer: For fusion patterns not covered by the operator library, use a domain-specific compiler and a Triton-like frontend to compile computation descriptions into target hardware instructions.
4. Service Runtime Layer: Establish the pathways for weight loading, service startup, request scheduling, batching, and multimodal input/output.

The core value of this layered structure is that when the model architecture changes, only the mapping relationships in the operator implementation and compilation layers need modification. The code in the model logic and service layers can be maximally reused.

3. Migration of Sparse Attention: Index Selection and KV Cache Layout

The sparse attention mechanism employed by this model operates on a core principle: for each query, only a highly relevant subset of historical tokens is selected to participate in the attention computation. This reduces the computational complexity of attention from linear growth with context length to an approximately constant level.

3.1 Computation Path Breakdown

From the inference engine’s perspective, sparse attention can be broken down into four stages:

• Index Selection: Compute the relevance scores between queries and historical keys, and select the Top-K indices. This stage is purely memory-bound computation, requiring highly efficient reduction and sorting implementations.
• KV Cache Layout: Sparse attention requires gathering historical key-values by index. Therefore, the KV cache cannot use a simple contiguous layout; it requires a blocked layout designed for efficient random access, while accommodating paged management and defragmentation.
• Prefill Path: Processes the complete prompt, requiring full-scale index selection and attention computation in a single pass. This path has high computational density and multiple parallel dimensions.
• Decode Path: Generates tokens sequentially. Each step processes only a single query, meaning index selection degrades into maintaining incremental relevance. This path is extremely sensitive to latency.

3.2 Adaptation Key Points

For the four stages mentioned above, the software stack engineering efforts include:

• Integrating the index selection operator into the hardware-optimized sparse attention operator library, utilizing its built-in vectorized scoring and approximate Top-K implementation to avoid inefficient element-wise comparisons at the general framework level.
• Redesigning the blocking granularity of the KV cache so that the memory locality of the gather operation aligns with the hardware’s cache line width. Simultaneously, reuse the same layout description between Prefill and Decode to reduce format conversion overhead.
• Orchestrating sparse attention, state management mechanisms (for maintaining intermediate states across layers and steps), cross-layer residual connections, and attention modules in the vision encoder into a complete execution chain. This ensures consistent intermediate tensor lifecycles across all branches under multimodal inputs.

An illustrative execution chain is described as follows (pseudo-code demonstrating orchestration):

for layer in model.layers:
    x = layer.residual_in(x)
    idx = sparse_index_select(q, kv_cache.index_view)      # Index selection
    attn_out = sparse_attention(q, kv_cache.gather(idx))   # Sparse attention
    state = state_manager.update(state, layer.state_slot)  # State management
    x = fuse_residual(x, attn_out, state)                  # Cross-layer residual
if inputs.has_image:
    x = vision_attention_branch(x)                         # Vision branch

From an engineering perspective, it is crucial to emphasize that any tensor layout mismatch within this chain will result in implicit format conversions, which will completely erode the performance gains brought by sparsification. Therefore, layout conventions must be unified during the chain design phase.

4. Online Conversion of Low-Precision Expert Weights and Workspace Reuse

The expert layers of this model utilize a 4-bit floating-point weight format. This format has a limited representation range. Directly involving it in matrix multiplication requires the hardware to have corresponding low-precision multiply-accumulate units. If the hardware does not natively support this, the weights must be converted to a hardware-friendly 8-bit floating-point format before computation.

4.1 Trade-offs Between Two Conversion Strategies

• Offline Conversion: Before service startup, all expert weights are pre-converted to 8-bit format and saved to disk. The advantage is zero conversion overhead during inference; the disadvantages include doubled storage footprint, conversion time, and the cost of reconverting when weight versions are updated.
• Online Conversion: Before expert computation occurs, the 4-bit weights are converted to 8-bit on-demand and immediately used for computation. The advantages are no need to pre-generate complete offline weights, lower storage footprint, and flexible weight updates. The disadvantage is that it places higher demands on the throughput and pipeline overlapping of the conversion operator.

This practice chose the online conversion route. Its feasibility is built on two optimizations:

1. Pipeline Overlapping of Conversion and Matrix Multiplication: The weights are chunked. While the i-th chunk is being converted, matrix multiplication is executed on the (i-1)-th chunk, effectively hiding the conversion latency behind computation.
2. Cross-Layer Reuse of FP8 Workspace: Expert weights from different layers are not computed simultaneously. Therefore, a fixed-size 8-bit workspace can be allocated and cyclically reused across layers. This design compresses the additional memory required for conversion from “proportional to total weight volume” to “proportional to a single layer’s expert weights,” significantly reducing the memory footprint.

4.2 Implementation Considerations

• Numerical Consistency: The 4-bit to 8-bit mapping must align with the expected semantics of the model publisher to avoid systematic bias.
• Lifecycle Management: Workspace reuse requires strict lifecycle management. In pipeline parallel or multi-request concurrent scenarios, read-after-write dependencies in the workspace must be correctly synchronized, typically implemented using double buffering combined with event mechanisms.
• Discrete Access Patterns: The discrete nature of expert routing means workspace access is non-contiguous. The blocking granularity should be optimized jointly with expert granularity and cache line width.

5. Software Stack Collaboration: Multi-Frontend Integration of Community Model Code

Upstream community model implementations are typically provided as “model definitions + several custom operators.” To run this code on the target hardware, a multi-layered software stack collaboration is required:

• Graph Execution and Scheduling Framework: Integrates the model definition and handles computation graph partitioning, batching organization, and memory planning. In this practice, a graph execution backend oriented towards the target hardware fulfills this role. Its programming interface exposed to the upper layer remains consistent with mainstream community frameworks, thereby reducing the migration cost of model code.
• High-Performance Operator Library: Covers high-frequency operators such as attention, normalization, activation, and matrix multiplication, providing dedicated implementations for sparse attention.
• Triton-like Compilation Frontend: For fusion patterns not covered by the operator library, this allows developers to describe computations using Python-like syntax, which the compiler then generates into target hardware code. This layer is the critical flexibility needed to “quickly integrate new models.”
• Domain-Specific Compiler: Responsible for sinking high-level computation descriptions into hardware instructions, handling vectorization, pipeline scheduling, and register allocation.
• Operator Auto-Tuning Tool: For newly emerging computation patterns, this tool automatically searches for optimal parameters like tiling sizes and thread configurations, reducing manual tuning efforts.

The engineering significance of this combination is that when facing a completely new model architecture, the team does not need to wait for the operator library to be incrementally completed. Instead, they can use the compilation frontend to quickly validate the semantics, and then progressively replace the hot paths with manually optimized operators, forming a “correct first, fast later” iterative rhythm.

6. From Weights to Service: Day-0 Acceptance Criteria

Whether Day-0 adaptation is successful should be measured by reproducible acceptance criteria, rather than just looking at a single-point benchmark:

1. Weight Loading: Complete weights can be sharded, loaded, and verified within a reasonable timeframe, and memory consumption aligns with planning.
2. Service Startup: The inference service starts normally, health checks pass, and batching and request queue behaviors meet expectations.
3. Text Inference Pipeline: Single-turn and multi-turn conversations, as well as long-context inputs, return correctly. The output is semantically consistent with the reference implementation.
4. Basic Image Inference Pipeline: The intermediate states of the vision encoder branch and the language branch connect correctly. Multimodal inputs do not trigger additional format conversion overhead.
5. Performance Baseline: Under fixed concurrency and input length, record the Time To First Token (TTFT) and throughput as a control baseline for subsequent optimizations.

It must be emphasized that the goal of Day-0 is “pipeline connectivity and semantic correctness.” Performance optimization should be pursued as an independent subsequent phase. Mixing these two together often leads to失控 (uncontrolled) adaptation cycles.

7. Summary and Reusable Engineering Experience

Reviewing the entire adaptation process, four universally applicable experiences can be distilled for similar endeavors:

• Layered Integration is Better than Rewriting: Reusing upstream model logic and concentrating changes in the operator and compilation layers is the core of controlling adaptation costs.
• Layout Conventions First: Sparse attention, KV cache, cross-layer residuals, and vision branches share numerous intermediate tensors. Inconsistent layouts will continuously erode performance through implicit conversions.
• Joint Design of Low-Precision Conversion Strategy and Memory Planning: Online conversion + cross-layer workspace reuse is a viable path to balance storage footprint and flexibility.
• Day-0 Acceptance Based on Pipeline Integrity: Ensure loading, startup, text, and multimodal inference are fully connected before discussing performance. This ensures a controllable iterative rhythm.

For developers building heterogeneous inference infrastructures, the transferable value of the methodology described above is not limited to a specific piece of hardware or a single model. It outlines the abstraction layers and engineering disciplines that a software stack must possess when facing rapidly evolving model architectures.