Day-0 Guide: Adapting Domestic Heterogeneous AI Compute Cards for 3-Trillion-Parameter Models

Adapting Domestic Heterogeneous AI Compute Cards for 3-Trillion-Parameter Models: Day-0 Practices and Key Technologies

As large model parameter scales surpass the trillion-level, rapidly deploying ultra-large model inference on domestic heterogeneous AI compute cards has become a core challenge in AI infrastructure. This article focuses on a 3-trillion-parameter sparse large model supporting a million-token context and multimodal (text and image) reasoning capabilities. It systematically outlines the Day-0 adaptation workflow—from software stack setup and sparse attention adaptation to mixed-precision weight conversion—providing developers with reusable technical pathways and risk alerts.

1. Core Hardware Challenges of Ultra-Large Models

The target model, undergoing massive post-training and reinforcement learning, excels in code generation, 3D creation, financial research, and long-horizon Agent tasks. However, its architectural features impose three strict requirements on underlying compute power:

Memory Pressure from Ultra-Long Contexts: A 1-million-token context window means both Prefill and Decode phases must manage massive KV Cache, posing severe challenges to memory capacity and bandwidth.

Non-Standard Operator Requirements for Sparse Attention: The model introduces the DSA (Dynamic Sparse Attention) mechanism, requiring underlying support for dynamic index selection and non-contiguous KV cache access. Traditional dense attention operators cannot be directly reused.

Computational and Conversion Complexity of Mixed Precision: Some expert weights are stored in MXFP4 (4-bit floating-point) format. The hardware must simultaneously support MXFP4 de-quantization and runtime synergy with FP8/FP16; otherwise, precision overflow or computation interruption will occur.

Directly relying on traditional framework mappings often leads to frequent memory copies and missing operators. Therefore, building a highly compatible, scalable proprietary software stack (covering compilers, operator libraries, and communication libraries) is the key to solving these issues.

2. Highly Compatible Software Stack: The Engineering Foundation for Rapid Migration

In Day-0 adaptation, the software stack acts as both a “translator” and an “accelerator.” It deeply covers four layers: framework execution, operator compilation, device runtime, and multi-card communication, forming a complete link from upper-level inference frameworks to underlying hardware instructions. The team leveraged upstream community implementations to avoid the resource consumption of building from scratch.

Core Toolchain Synergy: The development team utilized three key components:

1. A highly optimized matrix operator library supporting multi-precision linear algebra operations.
2. A Domain Specific Language (DSL) based on Triton (the code in this article is illustrative pseudo-code, depending on Triton 2.x and above, targeting a specific domestic GPU architecture). It allows writing underlying kernels using Python-like syntax, significantly lowering the development barrier.
3. A dedicated compiler that accurately maps the intermediate representation generated by the DSL to the hardware’s streaming multiprocessors, achieving instruction-level parallel optimization.

# Illustrative: MoE fused operator pseudo-code based on Triton DSL (not runnable code)
# Dependencies: triton>=2.1, target architecture is a specific domestic GPU
@triton.jit
def fused_moe_kernel(
    A_ptr, B_ptr, C_ptr,
    N, K, stride, BLOCK_M: tl.constexpr, ...
):
    # Fuse expert routing and computation to reduce memory read/write
    pid = tl.program_id(axis=0)
    # ... omit specific pointer offsets and data loading logic ...
    # Core matrix multiplication fused with non-linear activation (FP8 precision context)
    acc = tl.dot(a, b)
    tl.store(C_ptr + offsets, acc)

Parallel Reuse and Extension: The team adopted a strategy of “reusing mature interfaces + targeted incremental development.” They added model-specific computation paths on top of existing communication libraries and execution engines, ensuring stability while shortening the adaptation cycle.

3. DSA Sparse Attention and Multimodal Pipeline Adaptation

The DSA mechanism requires changing the KV Cache from a fixed contiguous layout to dynamic block management. The core challenges are twofold: first, the runtime must safely access non-contiguous memory by parsing the sparse index tensors output by the model; second, memory fragmentation caused by sparse access must be avoided.

Index Parsing and KV Cache Reconstruction: The software stack implements a dynamic index parsing module to complete safe non-contiguous memory access within the kernel. Simultaneously, the KV Cache is divided into fine-grained blocks, establishing a block-based metadata index pool to control fragmentation effectively.

Dual-Path Optimization for Prefill and Decode: The Prefill phase uses coarse-grained parallelism to maximize compute utilization. The Decode phase reduces bandwidth waste through vectorized memory access and data reordering.

Multimodal Fusion: Image features extracted by the visual encoder are injected into the main model trunk through an adapter projection layer. This works in tandem with a cross-layer residual mechanism, ensuring multimodal information is transmitted stably through deep networks, forming a complete multimodal execution pipeline.

4. MXFP4/FP8 Mixed Precision Online Conversion Solution

Since some expert weights are stored in MXFP4 format, traditional offline conversion requires generating massive FP8 weight files, severely slowing down Day-0 deployment. The team designed a mechanism combining “on-demand online conversion + cross-layer FP8 workspace reuse”:

• On-Demand Loading and Conversion: After the inference scheduler determines the activated experts, it reads MXFP4 weights from object storage or local high-speed cache and de-quantizes them to FP8 in memory in real-time.
• Cross-Layer Workspace Reuse: Different Transformer layers share a fixed-size FP8 workspace, effectively controlling memory peaks.

Comparison Dimension	Offline Conversion Solution	Online Conversion Solution
Storage Footprint	Requires full FP8 weights, massive size	Stores only MXFP4, size reduced by ~4x
Deployment Time	Hours to days	Minute-level startup
Memory Peak	High (full FP8 resident)	Controllable (shared workspace reuse)

Online Conversion Execution Flow: Router outputs gating signal → Determine Top-K experts → Extract MXFP4 weights → De-quantize and write to shared FP8 workspace → Trigger FP8 matrix multiplication → Clear workspace and proceed to the next layer.

Risk Alert: If online conversion times out or memory is insufficient, the system should fall back to offline FP8 weights or trigger degraded inference (e.g., reducing the number of activated experts) to avoid service interruption.

5. Common Issues and Engineering Insights

Common Pitfalls:

1. Memory Fragmentation — Requires regular defragmentation of the Block pool and monitoring fragmentation rates.
2. Sparse Index Out-of-Bounds — Runtime requires boundary checks and exception handling.
3. Precision Overflow — Strict alignment of rounding strategies is needed between MXFP4 de-quantization and FP8 computation.

Core Insights:

1. Software stack decoupling and standardization are prerequisites for rapid adaptation to new architectures.
2. Underlying infrastructure must possess the capability to handle non-standard data layouts and low-bit formats online.
3. Multimodal and long-context adaptation is a systematic engineering effort involving state management, cross-layer communication, and memory scheduling.

What should be noted when adapting domestic AI compute cards to trillion-parameter models? The key lies in: selecting a highly compatible software stack to reduce migration costs, prioritizing the validation of sparse operators and mixed-precision pipelines, and establishing robust fallback and fault-tolerance mechanisms. Looking forward, empowered by container platforms and cloud-native monitoring, this underlying adaptation capability will provide enterprise users with more stable and elastic intelligent compute support, driving complex Agent tasks and vertical domain intelligence to new heights.