Hardware-Software Co-Design for Domestic GPUs and 10K-Card Model Deployment: From Sparse Attention to Quantized Inference

1. From “Runs” to “Runs Well”: Infrastructure Challenges in the Trillion-Parameter Era

When large language model parameter counts jump from hundreds of billions into the trillions—reaching scales such as 3 trillion—deployment logic fundamentally shifts. Traditional inference pipelines built on the NVIDIA CUDA ecosystem expose three critical bottlenecks under this parameter explosion: the memory wall, the bandwidth wall, and operator fragmentation. For any compute provider or cloud vendor, achieving first-day (Day-0) usability of ultra-large models on proprietary silicon has become the dividing line for software-stack maturity.

The case examined here involves the adaptation of a domestically architected training-and-inference accelerator to an open-source model with a one-million-token context window, a mixture-of-experts (MoE) design, and sparse attention. This was not a simple port or parameter tweak; it was a full-stack rewrite from the computational graph down to the instruction level. The technical path is highly relevant to R&D teams building AI infrastructure.

2. Challenge Breakdown: Dual Pressure from Sparse Attention and Quantized Weights

2.1 The Non-Linear Nature of Sparse Attention

Unlike traditional dense attention, next-generation frontier models increasingly rely on Dynamic Sparse Attention (DSA). The core idea: when generating each token, the model does not perform a full weighted sum over all historical key-value (KV) pairs. Instead, a lightweight prediction head dynamically selects a small set of the most relevant index positions and computes attention only for those.

The benefit is clear: in ultra-long-context scenarios, computational complexity drops from $O(n^2)$ to approximately $O(n \cdot k)$, where $k$ is the sparsity hyperparameter. However, DSA imposes three hard challenges on hardware and software stacks:

1. Computation-dependent index selection. Which KV positions to use depends on the previous step’s prediction, introducing dynamic data dependencies that prevent the operation from being pre-scheduled into a fixed matrix-multiply shape like dense attention.
2. Randomized KV-cache layout. Selected KV pairs are scattered across cache partitions in different layers and heads rather than occupying contiguous memory blocks, requiring operators with on-demand aggregation capability.
3. Prefill-decode decoupling. During prefill, long input sequences demand parallel attention across all positions. During autoregressive decode, only one token is generated at a time, with an entirely different compute pattern. The two stages have divergent needs for operator shapes, parallelism granularity, and cache-reuse strategies.

2.2 MXFP4 Expert Weights in Non-Standard Precision

Beyond sparse attention, the model uses MXFP4 (Microscaling Floating Point 4-bit) format to store expert weights in its MoE modules. MXFP4 is a block-scaled low-bit floating-point format designed to compress each parameter to 0.5 bytes while retaining near-FP16 precision.

MXFP4 is not a natively supported compute precision on most current GPUs or domestic accelerators, whose matrix-multiply units typically handle only FP16, BF16, FP32, and INT8. Using MXFP4 weights at inference time forces a choice between two approaches:

• Full offline conversion: Convert all MXFP4 weights to FP16 at model-load time and keep them in memory. This is simple but eliminates the storage savings. For a 3-trillion-parameter model, converting everything to FP16 quadruples memory requirements, making single-card deployment impossible.
• Online per-layer conversion: At inference time, convert only the MXFP4 weights of activated experts to the compute precision on the fly. This demands extremely efficient scheduling in the software stack so that conversion itself does not become a new bottleneck.

3. The Software Stack’s Scalpel: Adaptation Strategies from Graph to Instruction Level

Faced with these challenges, the software stack in this case was not helpless. Its core strategy: keep upper-layer model code semantics unchanged while completely replacing and reordering the underlying compute path.

3.1 Sparse Attention via Custom Operator Libraries

To support DSA, the software team implemented core sparse-attention operators through a proprietary operator library (referred to here by the neutral designation MATE). Two key design decisions stand out:

• Two-level index buffer caching. The first level holds the raw index sequence output by the prediction head. The second level holds a reordered pointer array pointing to actual KV physical addresses. This avoids global memory address lookups on every decode iteration, using L2 cache for fast index-to-address mapping instead.
• Blocked KV-cache layout. The KV cache is physically partitioned into fixed-size blocks (e.g., 128 tokens per block) with a block-level index table maintained. When sparse attention picks a few discrete positions, the operator first reads the block index, then loads only the corresponding one or two memory pages, minimizing I/O overhead.

3.2 Dual-Track Prefill and Decode Execution

During prefill, input lengths are large (thousands of tokens may be processed at once in the context-filling phase), so sparse-attention index selections can be computed once in advance. The software stack compiles this phase into a highly parallel “batch index gather + matrix multiply” kernel that leverages TMA (Tensor Memory Accelerator) or equivalent hardware channels, sidestepping the synchronization cost of frequent reductions.

During decode, only one token is generated per step—compute is minimal but latency sensitivity is extreme. The stack abandons pipeline parallelism in favor of a lightweight “single-instruction, multiple-thread” kernel that fuses index prediction and attention computation within the same loop body, cutting one kernel-launch’s fixed latency.

3.3 Online Quantized-Weight Conversion with Cross-Layer Reuse

Rather than converting all weights offline, the team designed a runtime-scheduled workspace reuse mechanism:

1. Pre-allocate an FP8 workspace. A fixed-size contiguous memory region (scaled to one expert’s activation footprint) is reserved in device memory for temporary converted weights.
2. On-demand load and convert. When the MoE router determines which experts a given token is dispatched to, the stack reads the corresponding MXFP4 data from the weight area, performs dequantization and upcasting via vector units, and writes directly into the FP8 workspace.
3. Cross-layer reuse. Across multiple MoE iterations, different layers’ experts are called repeatedly; the workspace can be overwritten without new allocations each time. This reduces temporary weight storage by roughly an order of magnitude.

Choosing FP8 rather than FP16 as the conversion target was deliberate—FP8 matrix-multiply throughput is typically double that of FP16, and conversion bandwidth is half. In inference scenarios where memory bandwidth is the primary bottleneck, this yields substantial performance gains.

4. End-to-End Pipeline Orchestration

Optimizing individual operators is only step one; what truly makes a model run is pipeline-level coordination. This adaptation focused on synchronizing four modules:

Module	Responsibility	Coordination Key Point
Index Selection Unit	Generates Top-K position indices	Must share the same scheduling table with the KV-cache manager
KV Cache Manager (KDA)	Manages key-value lifecycle per head and layer	Must prefetch based on index prediction results
Cross-Layer Residual Connections	Maintains gradient flow through deep layers	Must insert lightweight synchronization during sparse-compute gaps
Visual Attention Module	Handles multimodal input alignment	Uses a different thread-bundle allocation strategy from text attention

In testing, decoupling index selection from KV prefetch caused approximately 12% of decode time to be wasted waiting for memory readiness. To address this, the stack introduced a predictive prefetch mechanism: using index-change trends from the previous five decode iterations to forecast the likely KV access range in the next round and issue prefetch commands early. This optimization nearly eliminated memory-wait time.

5. General Takeaways: A Methodology for Future Model Adaptation

This case represents not a one-off effort but a reproducible methodology.

5.1 The Software Stack Needs Four Layers of Programmability

A mature domestic AI software stack must expose user-space programmable interfaces at four levels:

• Graph-compilation layer: Supports user-defined operators and dynamic control flow, not just static graphs.
• Operator layer: Provides an embedded language similar to Triton or TileLang so algorithm engineers can write high-performance kernels without learning low-level assembly.
• Runtime layer: The memory allocator must support elastic workspaces, memory reuse, and cross-kernel sharing.
• Scheduling layer: Allows user intervention in thread-bundle scheduling and pipeline-stage partitioning to accommodate custom parallelism patterns across models.

5.2 Models Should Also Proactively Adapt to Hardware

It is worth noting that non-standard precisions like MXFP4 are becoming a mainstream way to reduce inference cost. For chipmakers and cloud providers, rather than waiting for international standards to fully mature, it is better to embed a highly flexible “precision-conversion layer” in the runtime that supports common block-scaled formats. This gives hardware greater resilience when facing diverse quantization specifications.

6. Conclusion

As AI compute demand continues to grow exponentially, the value of hardware-software co-optimization becomes ever more apparent. This adaptation demonstrates that for a 3-trillion-parameter pioneer model, as long as the software stack provides thorough computational-graph rewriting, an efficient operator library, flexible memory scheduling, and sufficient low-level programmability, even a non-mainstream domestic chip can achieve Day-0 usability. This is not merely a proof of a single product’s capability—it is a signal that the underlying software ecosystem is maturing. Going forward, as sparse computation, low-bit quantization, and multimodal hybrid architectures become standard, this kind of deep hardware-software adaptability will be the core competitive advantage of every AI infrastructure provider.