Day-0 Adaptation of Ultra-Large Sparse Expert Models on Domestic Compute

1. The Problem: When Model Scale Crosses the Trillion-Parameter Threshold

As large models move from hundreds of billions to trillions of parameters, the engineering challenge is shifting from “can we train it” to “can we actually run it.” A typical 3-trillion-parameter sparse expert (MoE) model supporting million-token context lengths has weight files measured in terabytes. During a single Prefill pass, the activations, KV cache, and expert weights compete simultaneously for memory bandwidth and capacity. For a general-purpose accelerator targeting both training and inference, this means the full pipeline—from weight loading and service startup to basic text and image inference—must work on the model’s release day (Day-0). Otherwise, “adaptation” is just a slogan.

This article is based on a Day-0 adaptation practice for an ultra-large sparse expert model on a domestic accelerator. Specific product models and brand names are stripped out. Instead, we reconstruct from a technical-path perspective what this kind of adaptation work actually involves, where the difficulties lie, and how they are broken down layer by layer. Readers can treat it as an engineering methodology for landing trillion-scale MoE models on general-purpose compute platforms.

First, let’s clarify the technical characteristics of this type of model: it uses a sparse expert architecture where each token activates only a subset of experts, keeping per-forward computation manageable even as parameter count explodes. It introduces sparse attention mechanisms that compute attention only over selected historical tokens via index selection. It stores expert weights in low precision (e.g., 4-bit floating point) to compress memory footprint. It also includes auxiliary structures such as cross-layer residuals and state management. Each of these characteristics corresponds to a compute path that requires dedicated adaptation.

2. The Overall Layering of Adaptation: From Community Code to a Usable Service

Transforming an open-source model implementation into an actual inference service on a given accelerator requires an entire software stack. This pipeline can be divided into four layers:

1. Model layer: Model definitions, weight formats, tokenizers, and inference scripts provided by the upstream community.
2. Framework layer: A high-performance inference framework that hosts the model implementation, handling scheduling, batching, KV cache management, and sampling.
3. Operator and compiler layer: Operator libraries, tensor-oriented domain-specific languages and their compilers, and general compiler backends that map high-level operators to target hardware instructions.
4. Hardware layer: The accelerator and its memory system and interconnect bandwidth.

The core principle of adaptation work is: reuse upstream framework implementations as much as possible and concentrate modifications in the operator and compiler layers. The reason is practical—upstream model code iterates extremely fast. If every update required invasive changes at the model layer, maintenance costs would quickly spiral out of control. The operator layer’s abstractions are relatively stable; once key compute paths are mapped, subsequent minor model version upgrades can often be absorbed with zero or minimal changes.

In this practice, the team used an inference framework branch supporting multiple hardware backends to host the upstream model implementation, then completed migration and integration of key compute paths for the new architecture through a self-developed operator library, a tensor DSL and its compiler, and a general compiler toolchain. The key word here is “hosting”: not rewriting the model, but ensuring every category of computation in the model finds an efficient landing point on the target hardware.

3. Adapting Sparse Attention (DSA): Breaking Down a Three-Stage Compute Path

Sparse attention is the part of this type of model that requires the most careful handling. Traditional full attention computes similarity between every Query and all historical Keys. Sparse attention first uses a lightweight index selection stage to pick the Top-K most relevant historical positions, then performs exact attention computation only over those positions. Adaptation must cover three stages:

(1) Index Selection. This stage typically involves dimensionality-reduced projection of historical Keys, similarity scoring, and Top-K filtering. The difficulty is that Top-K sorting is not a naturally efficient operator on accelerators—it requires combining block-wise reduction with bitmask techniques. In engineering practice, it is generally implemented as a three-step process: “block-wise scoring + intra-block reduction + cross-block merge,” avoiding the synchronization overhead of global sorting.

(2) KV Cache Layout. Sparse attention requires the ability to randomly gather historical KV by index, so the cache cannot simply be laid out as a contiguous sequence. Adaptation requires designing a layout that supports fast random access—for example, organizing by page, contiguous within a page, and jumping between pages via an index table. This ensures gather memory access efficiency while allowing on-demand cache allocation, preventing memory explosion at million-token context lengths.

(3) Two Compute Paths: Prefill and Decode. The Prefill stage processes an entire prompt at once, with a large number of Queries and compute-intensive workloads—suitable for fusing index selection and exact attention into large-granularity operators. The Decode stage processes one token per step, a classic memory-bandwidth-bound workload, requiring index selection to be incremental—reusing the previous step’s index structure and only computing supplementary updates for new positions. The operator fusion strategies for the two paths are completely different and must be tuned separately.

Beyond this, sparse attention operators must work together with the state management module, cross-layer residual connections, and the visual attention branch to form a complete execution pipeline. Cross-layer residuals mean some intermediate results must be preserved across multiple layers, imposing additional requirements on memory lifecycle management. Visual attention indicates the model is multimodal; the image branch’s attention pattern differs from the text branch and must be integrated separately.

4. Online Conversion of 4-Bit Expert Weights and FP8 Workspace Reuse

Expert weights in trillion-scale MoE models are stored in 4-bit floating point—this is the prerequisite for compressing terabyte-scale weights into limited memory. However, compute units typically have limited support for 4-bit precision, and using it directly for matrix multiplication would sacrifice throughput. A viable approach is online conversion: keep the 4-bit format during loading, dequantize to 8-bit floating point immediately before computation, and release immediately after—without retaining a full 8-bit weight copy.

There is an easily overlooked optimization here: MoE experts are activated in batches, with only a few experts participating in computation at any given moment. Therefore, the 8-bit dequantization workspace can be reused across layers—preparing an 8-bit weight copy for every layer wastes memory and is unnecessary. Only one workspace large enough to hold the “maximum activated expert set for a single layer” is needed, combined with asynchronous copy and compute overlap. In engineering implementation, synchronization points must be handled properly: dequantization must complete before the compute instructions that use that expert’s weights, and the workspace’s next reuse must wait until the previous computation has finished reading. A common approach is to use events rather than global synchronization for fine-grained dependency management, hiding waits within the compute pipeline.

Pseudocode: Online quantization pipeline for expert computation
for layer in layers:
    experts = route(tokens, layer.gate)        # Select activated experts
    for e in experts:
        wait(workspace_free[e])                # Wait for workspace to be writable
        dequant_async(W4[layer][e] -> ws_fp8)  # 4-bit -> 8-bit, async
    wait(dequant_done)
    gemm(tokens, ws_fp8)                       # 8-bit matrix multiply
    signal(workspace_free)                     # Release workspace

This strategy of “not pre-generating a complete offline 8-bit weight copy” reduces memory usage from “total weight size × 1.5” to “total weight size + single-layer workspace”—one of the keys to running trillion-scale models on a single multi-GPU machine.

5. Engineering Cadence and Validation Criteria for Day-0 Adaptation

Day-0 adaptation is not about “getting a demo to run”—it’s about completing a serviceable pipeline on the day model weights are published. A reasonable cadence is roughly:

• Before T-0: Based on the model architecture preview and historical implementations of similar architectures, pre-build operator templates, pre-compile key kernels, and prepare weight format conversion scripts.
• On T-0: Complete weight loading verification, single-GPU forward numerical alignment (comparing logits error against a reference implementation), multi-GPU parallelism strategy configuration, service startup, and health checks.
• From T+1 onward: Complete multimodal input paths, long-context stress testing, and batch scheduling tuning.

Validation criteria should cover three levels: numerical correctness (relative error of key layer outputs against the reference implementation within threshold), pipeline completeness (both text and image inputs can traverse Prefill and Decode), and service availability (throughput and first-token latency under concurrent requests). Reporting only “adaptation complete” without these criteria offers no reference value to engineering teams.

6. Lessons for Similar Work

1. Prioritize reusing upstream implementations and converge modifications to the operator layer. The fewer changes at the model layer, the easier it is to keep up with subsequent updates.
2. Sparse structure adaptation must be designed separately for Prefill and Decode. The two have opposite workload characteristics, and fusion strategies cannot be shared.
3. The value of low-precision weights lies in “online conversion + workspace reuse,” not offline pre-conversion—the latter significantly diminishes the memory advantage.
4. KV layout for random access must be designed upfront. Changing it later will ripple through the entire attention operator implementation.
5. Day-0 capability must be measured with reproducible criteria. Numerical alignment, pipeline completeness, and service availability—none can be omitted.

Landing trillion-scale sparse expert models on general-purpose accelerators is fundamentally a collaborative software stack engineering effort: the compiler maps operators to hardware, the operator library provides efficient primitives, and the inference framework handles scheduling and caching—all three are indispensable. Codifying this methodology means that the next time we face a larger-scale, more multimodal model, Day-0 won’t become Day-N.