Achieving Day-0 Adaptation of a Trillion-Scale MoE Model on Domestic AI Accelerators: Sparse Attention and Low-Precision Expert Inference in Practice

Introduction: When Model Architecture Evolution Outpaces Hardware Adaptation

The gap between open-source model weight releases and hardware readiness has long been a fixed cost in the inference deployment chain. For a model with hundreds of billions of parameters, migrating from weight publication to stable token generation on a specific accelerator traditionally takes weeks or more, including time for porting, debugging, and operator implementation. When parameter counts cross the trillion scale, context lengths approach one million tokens, and attention mechanisms shift to sparse structures, every step in this chain becomes significantly more demanding.

This article uses a real-world case—establishing a basic inference pipeline on the very day weights were released—to dissect the engineering methodology behind deploying a trillion-scale Mixture-of-Experts (MoE) model on a domestic all-in-one training-inference AI accelerator. Four technical focus areas are covered: software stack compatibility, sparse attention integration, low-precision expert weight handling, and memory layout optimization.

Background: The Challenge of Rapidly Changing Model Architectures

The target model in this adaptation effort is a MoE architecture with roughly three trillion parameters. It supports million-token contexts and has been fine-tuned with post-training and reinforcement learning for code generation, 3D content creation, financial research, knowledge-based Q&A, and long-horizon agent tasks. Public benchmarks indicate that its performance on terminal operation and long-form code tasks is approaching or matching top-tier closed-source models, suggesting it will quickly enter production inference queues.

This model presents three major challenges to the hardware and software stack:

1. Parameter scale and memory pressure. Even at low-precision storage, three trillion parameters impose extreme demands on per-device memory and inter-device interconnect bandwidth. Expert-parallel sharding strategies must align with the communication topology of the hardware.
2. Complex attention structure. The model uses a hybrid architecture combining sparse attention (DSA) with linear-attention-style state management. Traditional dense attention KV cache layouts and computation paths cannot be reused directly.
3. Heterogeneous weight precision. Expert weights use MXFP4, a low-precision block floating-point format, while operator libraries and compute units typically operate efficiently in FP8 or BF16. A low-overhead conversion path between these formats is required.

“Day-0 adaptation” in this context means completing the model loading, service startup, and basic text and image inference paths on the same day the weights are released. This is not full performance tuning; it is a milestone defined as “runs, produces correct outputs, and serves as a baseline for further optimization.”

Overall Strategy: Reusing Upstream Implementations via a Compatible Software Stack

The key to rapid adaptation is not rewriting the model from scratch but maximizing reuse of model implementations already validated by the upstream community. The overall strategy can be summarized in three layers:

• Framework layer: A mature high-throughput inference framework is used to host the upstream model code, preserving its parallelization strategy, scheduling logic, and sampling flow. No invasive changes are made at the model semantic level.
• Operator and compilation layer: A self-developed operator library, a Triton-like operator programming frontend, a domain-specific tensor programming language, and a custom compiler map the key computational paths of the new architecture onto the target hardware.
• Runtime and communication layer: This layer covers device execution, framework scheduling, and multi-device communication, ensuring the migrated computation graph runs stably in a multi-card environment.

The value of this layered design is that new computational paths only need to be added at the operator level; framework scheduling and parallel logic can be reused as-is. The operator layer also exposes mature interfaces, reducing coupling across the stack. The following pseudocode illustrates this relationship:

# Conceptual illustration: framework layer hosts upstream model definition, operator layer dispatches by backend
from sglang_backend import Runtime, ModelLoader
model = ModelLoader.load(
    arch="moe_sparse_attn",      # upstream-registered model architecture
    weights="model_weights/",     # weight directory containing low-precision expert weights
    dtype="bf16",                # activation and backbone compute precision
    ep_size=8,                   # expert parallelism, aligned with device count
)
runtime = Runtime(
    backend="musa",               # target hardware backend
    kv_cache_layout="paged_dsa", # paged cache layout for sparse attention
    comm="allreduce_ep",         # expert-parallel communication primitives
)
runtime.serve(model, tasks=["text", "image"])

Technical Focus 1: Enabling Sparse Attention and the Hybrid Attention Path

The model’s attention mechanism is not a single mechanism. It is a hybrid execution path consisting of sparse attention (DSA), a state management module, cross-layer residual connections, and visual attention. Adaptation work must be implemented at three levels simultaneously:

Index selection. Sparse attention first determines which historical tokens participate in the current computation. The index selection logic must be implemented as an operator that executes efficiently on the target hardware, avoiding additional global synchronization that could slow down the Prefill phase.

KV cache layout. Under a sparse structure, the KV cache is no longer a simple contiguous tensor. It must be organized in blocks and support indexed, sparse access. In practice, a paged cache layout is used, with distinct access patterns designed for Prefill and Decode: Prefill favors batched dense writes, while Decode favors indexed sparse reads.

Separate Prefill and Decode path adaptation. The two phases have fundamentally different computational characteristics—Prefill involves parallel computation over long sequences, while Decode is incremental token-by-token computation. Each path is integrated with the sparse attention operator separately, then chained with state management, cross-layer residuals, and visual attention to form a complete execution pipeline. Missing any component may manifest as degraded output quality or inconsistent results under long contexts, rather than an immediate error. Therefore, pipeline completeness must be validated with long-context consistency tests.

Technical Focus 2: Online Conversion of MXFP4 Expert Weights with Workspace Reuse

Most of the model’s parameters reside in the expert layers. In this case, expert weights are stored in MXFP4 format. While this offers low storage overhead, compute units typically operate at higher efficiency in FP8. A common approach is to pre-generate a full FP8 copy of the weights offline. However, this has two drawbacks: it consumes additional memory or storage comparable to the expert weight size, and it extends the time between weight release and service readiness.

The adaptation approach uses online conversion combined with cross-layer reuse of a shared FP8 workspace:

• Expert weights remain resident in their original MXFP4 format; no full dequantization is performed at load time.
• Before computation, the needed expert blocks are converted to FP8 on the fly and written into a pre-allocated FP8 workspace.
• This workspace is reused across layers, avoiding separate buffer allocations for each layer.

The engineering benefit is that both memory footprint and deployment preparation time are reduced, at the cost of introducing a conversion step before each expert computation. In production, the conversion and computation must be pipelined to hide latency: converting the next batch of experts on a separate stream while the current batch computes can effectively mask the conversion delay. A conceptual illustration follows:

# Conceptual illustration: online conversion + workspace reuse
workspace = alloc_fp8_workspace(max_expert_tokens)  # reused globally, cycled across layers
for layer in decoder_layers:
    for expert_batch in schedule(layer.experts):
        w_fp8 = convert_mxfp4_to_fp8(expert_batch, out=workspace)  # online conversion
        y = moe_gemm(x, w_fp8)                                     # expert computation

It is important to note that online conversion has a direct impact on numerical consistency. An engineering process should include a consistency check before and after conversion, comparing output distributions under fixed inputs to confirm that conversion precision loss is within acceptable bounds before moving to performance tuning.

Technical Focus 3: Coordinating Memory Layout and Multi-Device Communication

A three-trillion-parameter model cannot fit on a single device. It requires a combination of expert parallelism and tensor parallelism. Three coordination issues must be addressed:

Parallel sharding aligned with interconnect topology. All-to-all communication volume and latency in expert parallelism significantly affect overall throughput. Sharding should align expert placement with the grouping structure of inter-device interconnect bandwidth to minimize cross-group communication.

Memory contention between KV cache and expert weights. Million-token contexts mean the KV cache itself consumes substantial memory, while expert weights must remain resident. In practice, the KV cache is allocated dynamically using a paging scheme, and expert weights along with the conversion workspace are managed within a unified memory pool to avoid fragmentation from static partitioning.

Isolation between conversion workspace and communication buffers. If the online FP8 workspace and communication buffers share the same memory region, read/write conflicts may occur under high load. The two should be separated, with a fixed allocation quota reserved for communication buffers.

Validation Checklist: What Defines Day-0 Readiness

“Establishing a basic inference pipeline” requires verifiable acceptance criteria. At minimum, the following five items should be covered:

1. Model loading completeness: All layers and expert weights load successfully with no missing replacements.
2. Service startup stability: Multiple consecutive startups complete without out-of-memory errors or initialization failures.
3. Text inference correctness: Outputs for fixed prompts match a reference implementation within an acceptable threshold.
4. Image inference path availability: The visual encoder-to-text decoder path is functional, and mixed image-text inputs return results.
5. Long-context consistency: With inputs near the context limit, sparse attention index selection and cache reuse show no significant degradation.

Only after passing these five checks should the team move to performance tuning for throughput, latency, and concurrency. The significance of Day-0 adaptation is precisely this: resolving the “can it run” question on the day weights are released, while leaving the “how fast can it run” question to ongoing optimization.

Conclusion

Adapting a trillion-scale MoE model to new hardware is essentially a matter of establishing three parallel pipelines: the framework layer reuses the upstream model implementation to retain scheduling logic; the operator and compilation layer fills in new computational paths such as sparse attention and low-precision expert computation; and the runtime layer ensures stable memory layout and multi-device communication. Among the engineering decisions made in this effort, the most representative are the separate Prefill/Decode adaptation for sparse attention and the online conversion of MXFP4 expert weights with workspace reuse. For production-oriented inference deployment teams, it is recommended to harden the validation checklist above into the entry criteria for any adaptation workflow, and to build a continuous performance regression system on top of the Day-0 baseline.