Domestic AI Accelerator Completes Day-0 Adaptation for a Next-Generation Trillion-Parameter Multimodal Model

1. Background: Trillion-Parameter Models Meet Domestic Compute

A leading AI research organization recently released the full weights of a next-generation trillion-parameter multimodal model. It is the first version in its series and the most capable three-trillion-parameter model to date, with support for a context window of up to one million tokens. After large-scale post-training and reinforcement learning, the model shows substantial improvements in code generation, 3D creation, financial research, knowledge-based Q&A, and long-horizon agent tasks.

On third-party agent leaderboards, the model ranks first among open-weight models and leads in work, coding, and chat categories. Code and agent benchmarks show that its long-horizon task scores are significantly higher than the previous generation, with overall performance reaching or approaching leading closed-source models.

At the same time, the domestic AI compute ecosystem is maturing rapidly. A cloud vendor’s training-inference AI accelerator and its accompanying software stack completed adaptation immediately after the model release, achieving Day-0 support for model loading, serving startup, and basic text and image inference. The significance of this progress is not just that the model “runs.” It also validates the domestic software stack’s ability to quickly absorb changes in the programming model, runtime interfaces, and AI framework layers.

2. Adaptation Challenges: Engineering a Trillion-Parameter Multimodal Model

2.1 Memory and Communication Pressure from Model Scale

A three-trillion-parameter model produces extremely large weight files that cannot fit in the memory of a single accelerator. Inference typically requires tensor parallelism, pipeline parallelism, or other distributed strategies to shard the model across multiple cards. This places high demands on inter-card communication bandwidth and latency. Any communication bottleneck directly reduces end-to-end throughput.

In addition, a context window of one million tokens significantly increases KV cache usage. In long-context scenarios, the KV cache can even exceed the memory footprint of the model weights themselves. Efficient memory management and KV cache compression are therefore unavoidable parts of the adaptation process.

2.2 Migrating Sparse Attention

The model uses a sparse attention mechanism that selects key tokens through indexing to participate in attention computation, reducing computational complexity while preserving long-context capability. This differs significantly from traditional dense attention in the computation graph and requires reimplementing critical paths such as index generation and sparse aggregation.

When migrating community model code to the domestic software stack, sparse attention operator implementation is one of the core difficulties. It requires correctness in index generation, numerical stability in sparse computation, and compatibility with hybrid attention mechanisms.

2.3 The Multimodal Vision Pipeline

Supporting text-and-image inference means the model must process both text tokens and vision tokens simultaneously. Vision encoders typically include convolution, normalization, and attention modules whose computational characteristics differ from those of language models. On the domestic software stack, the vision computation path must be migrated and connected so that image features are correctly aligned to the language model’s input space.

3. Adaptation Approach: A Software-Stack-Driven Fast Migration Path

3.1 Overall Architecture

The adaptation work focused on the model architecture and computational characteristics, using the domestic software stack to migrate critical inference paths. The overall solution can be summarized as follows:

┌─────────────────────────────────────────────┐
│          Model serving layer (SGLang)        │
├─────────────────────────────────────────────┤
│      AI framework layer (PyTorch-compatible) │
├─────────────────────────────────────────────┤
│   Operator library layer (MATE + Triton + TileLang) │
├─────────────────────────────────────────────┤
│      Compiler layer (MTCC) and runtime       │
├─────────────────────────────────────────────┤
│  Device layer (MTT S5000 training-inference card) │
└─────────────────────────────────────────────┘

The software stack covers framework execution, operator compilation, device execution, and multi-card communication. This allows model adaptation to reuse mature interfaces while optimization efforts focus on newly introduced computation paths.

3.2 Inference Serving Through SGLang-MUSA

The team used SGLang-MUSA to host the upstream model implementation. SGLang is an efficient serving framework for large model inference that supports optimizations such as RadixAttention and continuous batching. By mapping SGLang’s runtime interfaces to the domestic software stack, the team quickly gained mature serving capabilities, including:

• Request scheduling and dynamic batching
• KV cache management and prefix sharing
• Streaming output and concurrency control

Reusing this layer greatly shortened the adaptation cycle and allowed the team to concentrate on optimizing newly added computation paths.

3.3 Operator Migration: MATE, Triton, and TileLang

The operator layer is the core of the adaptation. The team relied on the MATE operator library, Triton-MUSA, and TileLang-MUSA to migrate and integrate the critical computation paths of the new architecture.

MATE provides optimized basic operators covering matrix multiplication, normalization, activation functions, and other common operations. Standard operators can be reused directly with good performance.

Triton-MUSA allows developers to write custom operators in a Python-like DSL, with the compiler automatically generating device code. For non-standard operations such as index computation and sparse aggregation in sparse attention, Triton provides an efficient implementation path.

TileLang-MUSA targets lower-level operator development, giving developers fine-grained control over thread block partitioning, shared memory usage, and pipeline scheduling. For performance-sensitive vision computation paths, TileLang offers optimization headroom close to hand-written low-level code.

3.4 Integrating Sparse Attention and Hybrid Attention

For the model’s sparse attention mechanism, the software stack implemented index computation, sparse aggregation, and integration with hybrid attention. The work included:

1. Index generation: Compute the key positions that each query needs to attend to based on the input sequence, then generate sparse indices.
2. Sparse aggregation: Perform attention computation only on the selected key-value pairs and skip irrelevant positions.
3. Hybrid attention compatibility: The model may use different attention patterns in different layers, such as a mix of global attention and sliding-window attention. The sparse mechanism must work correctly with these mixed patterns.

The following is a simplified illustration of sparse attention forward computation:

# Pseudocode: sparse attention forward pass
def sparse_attention(query, key, value, index):
    # index: [batch, heads, seq_len, top_k]
    # Gather the corresponding keys and values according to the index
    selected_key = gather(key, index)      # [batch, heads, seq_len, top_k, dim]
    selected_value = gather(value, index)
    # Compute attention scores
    score = einsum("b h q d, b h q k d -> b h q k", query, selected_key)
    score = score / sqrt(dim)
    # Softmax and weighted sum
    attn = softmax(score, dim=-1)
    output = einsum("b h q k, b h q k d -> b h q d", attn, selected_value)
    return output

In a real implementation, causal masking, numerical stability, and memory access efficiency must also be considered.

3.5 Migrating the Vision Computation Path

Text-and-image inference requires the model to process image inputs. A vision encoder typically contains:

• Convolution layers for local feature extraction
• Normalization layers for training and inference stability
• Attention layers for modeling global dependencies
• Projection layers for mapping visual features to the language model dimension

The team migrated and integrated these modules on the domestic software stack, ensuring that encoded images are correctly concatenated with the text token sequence and participate in subsequent language model inference.

4. Validation and Results

After adaptation, the model runs stably on the domestic training-inference AI accelerator. Validation covered the following areas:

Validation Item	Description
Model loading	Weight sharding and loading, memory allocation, distributed initialization
Serving startup	Inference service launch, health checks, request intake
Text inference	Single-turn and multi-turn dialogue, long-text generation, code completion
Image inference	Image understanding, vision-language Q&A, visual description
Multi-card scaling	Tensor parallelism, pipeline parallelism, communication overhead

Test results show that the model can perform basic inference tasks on the domestic compute platform, with both sparse attention and vision computation paths executing correctly. This adaptation validates the software stack’s ability to support rapid compatibility migration and lays the foundation for further performance optimization.

5. Lessons Learned and Best Practices

5.1 Reuse Mature Interfaces and Focus on New Paths

Large model adaptation does not need to start from scratch. By reusing inference serving frameworks, base operator libraries, and communication interfaces, teams can focus limited engineering resources on model-specific new computation paths such as sparse attention and vision encoders. This is the key strategy behind Day-0 support.

5.2 Optimize in Layers

Adaptation can proceed layer by layer:

1. Functional correctness: First ensure the model runs end to end and produces outputs consistent with the reference implementation.
2. Performance baseline: Establish end-to-end performance metrics and identify bottlenecks.
3. Operator optimization: Apply hand-written optimization or auto-tuning to hot-spot operators.
4. System tuning: Optimize batching strategy, KV cache management, and multi-card communication.

5.3 Pay Attention to Numerical Stability

Trillion-parameter models are sensitive to numerical precision. During operator migration, it is important to pay attention to accumulation order, normalization methods, softmax numerical stability, and related details to avoid output anomalies caused by precision issues.

5.4 Multimodal Adaptation Requires Attention to Alignment

Alignment between vision and language modalities is central to multimodal models. During adaptation, ensure that visual features and text embeddings interact correctly in the same space, avoiding semantic drift caused by differences in operator implementation.

6. Outlook

As the domestic AI compute ecosystem continues to mature, large model adaptation cycles are expected to shorten further. Future work can focus on the following directions:

• Automated adaptation tools: Use graph compilation techniques to automatically identify and replace incompatible operators.
• Performance tuning frameworks: Provide auto-tuning capabilities to reduce manual optimization effort.
• Expanded multimodal support: Support inference for additional modalities such as video and audio.
• Ecosystem compatibility: Keep pace with mainstream AI frameworks and inference engines.

This adaptation effort demonstrates that the domestic software stack is now capable of supporting a trillion-parameter multimodal model. Achieving Day-0 support and a complete text-and-image inference path required coordinated work across the programming model, runtime interfaces, operator compilation, and multi-card communication. For developers, understanding this adaptation path can help deploy large models efficiently on domestic compute platforms and accelerate real-world AI applications.