Day-0 Adaptation for Trillion-Parameter Models: From Weight Release to Dual-Modal Inference Availability

1. Background: When Model Scale Crosses the Three-Trillion-Parameter Threshold

Large model parameters are rapidly moving from the hundreds-of-billions scale to the trillion scale. When a dense or sparse Mixture-of-Experts (MoE) model reaches roughly three trillion parameters and its context window extends to a million tokens, the challenge is no longer just about whether there is enough compute. The real question is whether the entire software stack can still run end to end.

Typical problems include:

• Weight loading: How to efficiently shard hundreds of gigabytes or even terabytes of weights, map them to device memory, and complete initialization.
• Compute path migration: New models often introduce new attention structures, new expert routing methods, and new low-bit numerical formats. General-purpose operator libraries may not work out of the box.
• Serving: There is a large gap between running on a single card and supporting multi-card parallelism, request scheduling, batching, and prefix caching.
• Multimodal pipelines: When text and images are mixed as inputs, the vision encoder, cross-modal alignment layers, and language backbone must work together within the same inference graph.

“Day-0 adaptation” means that on the same day model weights are publicly released, the target hardware platform can run the model end to end: the model can be loaded, the serving process can start, and basic inference requests can return correct results. This tests not single-point optimization, but the overall compatibility and migration efficiency of the software stack.

This article uses a Day-0 adaptation effort for a three-trillion-parameter MoE model as a case study. It breaks down the key technical steps and extracts reusable engineering practices.

2. Adaptation Goals and Validation Criteria

Before starting, it is important to define what “runnable” means. Otherwise, it is easy to fall into the trap of a model that loads but is not actually usable. In practice, the following four-layer validation approach is recommended:

1. Loading layer: Weights are fully read, sharded correctly, and free of precision overflow or tensor shape mismatches.
2. Startup layer: The inference serving process starts normally, communication between the scheduler and worker processes is correct, and memory usage is within the expected range.
3. Single-request layer: Text prompts produce coherent output, and image inputs can go through the vision encoding and fusion paths.
4. Stability layer: Multiple consecutive requests, inputs of different lengths, and mixed-modal inputs do not cause crashes or obvious numerical anomalies.

These four layers are progressive. Many adaptation efforts get stuck between layer 2 and layer 3: the service starts, but the output is gibberish. That usually points to inconsistent operator semantics or incorrect numerical precision handling.

3. Software Stack: High Compatibility Is the Prerequisite for Day-0

The Day-0 adaptation window is extremely short, usually only a few hours to one day. To complete migration within that window, the key is not rewriting kernels from scratch. The software stack itself must be able to absorb upstream implementations.

A mature AI software stack typically includes the following layers. Compatibility at each layer directly affects migration speed.

Layer	Role	Migration Focus
Programming model and runtime interfaces	Provide device abstraction, memory management, streams, and events	Whether interface semantics align with mainstream programming models
AI framework adaptation layer	Accept model definitions and execution graphs from upstream frameworks	Whether dynamic graphs and static graphs are supported, and whether community model code can be reused directly
Operator library	Provide dense and sparse matrix multiplication, normalization, activation, and other basic operators	Coverage and the cost of extending with new operators
Compiler and DSL	Compile high-level operator descriptions into device code	Support for emerging numerical formats and cutting-edge structures
Communication library	Multi-card collective communication	Match with parallelism strategies

Within this system, the serving-layer adaptation of the inference framework is especially critical. Taking a common inference serving framework as an example, its responsibilities include continuous batching, KV cache management, prefix cache reuse, and speculative decoding scheduling. The core of adaptation is to let these mechanisms reuse mature interfaces on new hardware rather than reinvent them.

In practice, the approach was as follows: based on the hardware backend abstraction layer of the serving framework, accept the upstream model implementation, then use the operator library and DSL compiler to replace and connect the key compute paths. This ultimately turns community model code into an actual inference service on the target device. The value of this path is that when the model structure changes, modifications are concentrated at the operator and compute graph levels, while the serving framework’s scheduling logic remains largely unchanged.

4. Key Path 1: Implementing Sparse Attention

Three-trillion-parameter MoE models commonly use sparse attention to control computation under long contexts. Taking structures such as DeepSeek Sparse Attention (DSA) as an example, the core idea is that each query no longer attends to all historical tokens. Instead, a lightweight indexer first selects the top-k most relevant keys for each query, and full attention is computed only over those k keys.

This creates three areas that must be adapted one by one.

4.1 Index Selection

The indexer is usually a small scoring network that outputs a relevance score between each query and each key, then selects the top-k. Adaptation points:

• Top-k selection is a performance hotspot for long sequences. It requires efficient sorting and selection operators.
• When k changes dynamically with sequence length, kernel support for dynamic shapes must be ensured.
• Score computation involves low-precision matrix multiplication. Accumulation precision and overflow behavior must be confirmed.

4.2 KV Cache Layout

Under sparse attention, the KV cache can no longer simply append data continuously. It must support gather operations by index. Adaptation points:

• The cache block granularity must align with the indexing granularity to avoid amplified random access.
• The cache access patterns during prefill and decode are very different and need separate designs.
• Under paged cache management, gather operations must correctly handle cross-block indexing.

4.3 Prefill and Decode Compute Paths

The adaptation strategies for the two phases are different:

• Prefill: Sequences are long and parallelism is high. The benefit of sparse attention comes mainly from reducing useless computation, so it is suitable for block-based sparse kernels.
• Decode: Each step processes only one token. The bottleneck is memory access rather than compute, so memory access must be optimized for the gathered KV data.

These three parts together form the complete sparse attention execution path. Together with state management modules such as state passing in linear attention or gated attention, cross-layer residual connections, and vision attention branches, they make up the overall execution graph for hybrid attention. If any part has inconsistent semantics, it will show up as output quality degradation under long contexts. This type of problem often does not appear in short-prompt tests, so long-sequence test cases must be specially constructed during validation.

5. Key Path 2: Online Conversion of Low-Bit Expert Weights

MoE expert weights are often stored in low-bit floating-point formats such as MXFP4 to reduce model size. At inference time, however, the compute units may not natively support matrix multiplication in that format. There are two common approaches:

• Offline conversion: Convert weights in advance to a device-friendly format such as FP8, persist them to disk, and load them directly at inference time.
• Online conversion: Load the original low-bit weights and convert them to the compute format at runtime.

For a three-trillion-parameter model, offline conversion means generating an additional complete copy of the weights. Storage and transfer costs are high, and it is not conducive to the Day-0 goal of deploying as soon as weights are released. Therefore, this effort used an online conversion plus cross-layer workspace reuse approach:

# Pseudocode: Online weight conversion and workspace reuse for expert computation
def expert_forward(x, expert_weight_mxfp4, workspace_fp8):
    # 1. Convert MXFP4 weights to FP8 online and write them into the preallocated workspace
    w_fp8 = convert_mxfp4_to_fp8(expert_weight_mxfp4, out=workspace_fp8)
    # 2. Use the workspace to perform matrix multiplication
    y = matmul_fp8(x, w_fp8)
    # 3. The workspace is reused across layers, so no separate allocation is needed for each expert
    return y

Key design points of this approach:

• Workspace reuse: The FP8 workspace is allocated per layer rather than per expert. Conversion and computation are performed expert by expert in sequence, reducing memory usage from all expert weights to a single expert plus one workspace.
• Controllable conversion overhead: Conversion is element-wise bit manipulation and can be pipelined with expert computation to avoid becoming a bottleneck.
• Numerical correctness: MXFP4 block-scale factors must be correctly applied during conversion. Otherwise, systematic deviation will occur.

The direct benefit is that there is no need to generate a complete offline FP8 weight set in advance. Very large models can be deployed immediately after weight release, significantly shortening the time from obtaining weights to service availability.

6. Key Path 3: Vision and Multimodal Pipelines

Supporting image inference means the model has a vision encoder and injects vision features into the language backbone. During adaptation, pay attention to the following:

1. Operator coverage for the vision encoder: Convolution, LayerNorm, GELU, and other basic operators are usually already covered. However, if the encoder uses window attention or positional encoding interpolation, implementation consistency must be confirmed.
2. Differences between vision attention and language attention: They may use different masking and caching strategies and must be correctly separated in the compute graph.
3. Feature alignment layers: The projection layer from vision features to the language embedding space is not small in parameter count. Pay attention to sharding during loading.
4. Scheduling for multimodal inputs: The serving layer must correctly handle preprocessing, batching, and memory estimation for mixed text-plus-image requests.

It is recommended to separately construct three types of validation cases: text-only, image-only such as image captioning, and mixed image-text question answering. Each path should be verified independently.

7. Engineering Checklist

Based on the above practices, here is a reusable Day-0 adaptation checklist:

• Weight file integrity verification: shard count, hashes, tensor shapes
• Correct parsing of scale factors for low-bit formats
• Boundary conditions for sparse attention top-k indexing: k greater than sequence length, fully masked cases, and so on
• Cross-block correctness of KV cache paging and gather
• Separate stress testing for prefill and decode paths
• Numerical error of online weight conversion within acceptable range
• Collective communication and load balancing under multi-card parallelism
• All three multimodal test case types pass
• Stability validation for long sequences near the maximum context length

8. Summary

Day-0 adaptation for trillion-parameter models is essentially a test of software stack compatibility. The more cutting-edge the model architecture, the higher the requirements for operator library coverage, compiler expressiveness, and serving framework abstraction. In this effort, a highly compatible software stack absorbed the upstream model implementation, and targeted migration and optimization were then applied to three new compute paths: sparse attention, low-bit expert weights, and the vision pipeline. As a result, the model went from weight loading to end-to-end dual-modal inference on the day the weights were released.

For R&D teams, the reusable lesson is to decouple adaptation work by layer: reuse mature interfaces at the serving layer, extend operators as needed, and strictly validate at the numerical layer. With clear layering, migration cost becomes controllable. When cost is controllable, Day-0 is not a lucky outcome. It is the inevitable result of engineering capability.