Trillion-Parameter MoE Model Inference: A Practical System Design and Deployment Guide

    A practical walkthrough of building inference systems for next-generation trillion-parameter Mixture-of-Experts (MoE) models with hybrid attention. It explains why the real engineering challenge is not parameter scale but the heterogeneous state space of the attention runtime, then breaks the problem into five core modules—addressing, compute, parallelism, scheduling, and decoding. Coverage includes multi-pool KV management with shadow-index prefix caching, hybrid attention kernels (sliding-window, compressed, and block-sparse Top-K), expert/context/data-parallel strategies, Prefill/Decode (PD) disaggregation, and multi-token prediction (MTP) speculative decoding. It closes with GPU memory budgeting, capacity planning, a phased tuning roadmap, and a pre-launch checklist for engineers deploying large-scale MoE models.

    Introduction: How a New Generation of LLMs Reshapes Inference Systems

    Large-model inference is undergoing a structural shift. Over the past two years, long context and low-cost high concurrency have been the two main threads, and inference system optimization has centered on memory access efficiency, batch scheduling, and GPU memory capacity. Now, a new batch of ultra-large-scale Mixture-of-Experts (MoE) models is going open source. With trillions of total parameters and tens to hundreds of billions of activated parameters, these models take “compressing information along the sequence dimension” as a core architectural mechanism.

    Models in this class typically share several characteristics:

    • Fine-grained expert routing: each expert is small, but the total number of experts is large and the sparse activation ratio is high.
    • Hybrid attention: global attention, sliding-window attention, and compressed attention appear alternately layer by layer.
    • Multi-Token Prediction (MTP): the main model carries built-in drafting capability and can predict multiple subsequent tokens in a single forward pass.
    • Extremely long context: usable windows reach hundreds of thousands or even millions of tokens.

    Parameter count and expert count do not by themselves create new difficulty for inference systems—mature frameworks already support expert parallelism. What truly changes the engineering boundary is the heterogeneity of the state structure in the attention layers.

    This article takes “landing a class of trillion-scale hybrid-attention MoE models on a general-purpose inference engine” as its main thread and, following the order of addressing → compute → parallelism → scheduling → decoding, breaks the inference system into five core modules:

    1. Multi-level KV pools and prefix caching;
    2. Hybrid attention operators;
    3. Expert parallelism, context parallelism, and attention data parallelism;
    4. Prefill/Decode (PD) disaggregated deployment;
    5. MTP speculative decoding.

    The article closes with GPU memory budgeting, capacity planning, and a phased tuning methodology.

    1. The Core Difficulty: The attention runtime State Space Goes from “One Slot per Layer” to “Multiple Pools per Layer”

    1.1 The Hidden Assumption Behind Traditional KV Cache

    In common inference engines, each request’s context is carried by a contiguous (or paged) KV cache: layer L‘s Key/Value occupies its own segment of slots for that layer, with total length equal to the sequence length. This model implies three assumptions:

    • All layers’ states are “homogeneous” and can be allocated and released in the same way;
    • Each layer’s state length equals the sequence length and grows linearly during generation;
    • Prefix reuse only requires checking whether a segment of the token sequence is already cached.

    Hybrid-attention models break all three assumptions at once.

    1.2 Three Changes That Alter the Cost Model

    (1) State heterogeneity. Sliding-window layers retain only the KV of the most recent W tokens; compressed attention layers additionally store a long-range summary state; only global attention layers need the full sequence KV. Each state type has a different capacity and lifetime, so you can no longer allocate them uniformly with “one segment per layer, billed by sequence length.”

    (2) Cost redistribution. Under common configurations, most layers only maintain KV within the window. This substantially reduces KV reads during decoding, moving attention compute from “memory-bound” toward “compute-bound.” The system’s main overhead shifts accordingly:

    • Full (global) layers processing long context see both memory access and compute rise sharply;
    • The all-to-all communication of the MoE FFN layers gradually becomes the bottleneck.

    (3) State-request decoupling. The semantics of prefix caching expand from “cache a segment of tokens” to “cache a snapshot of a segment of tokens’ state across the corresponding pools at every layer,” and different pools may have different valid prefix lengths. CUDA Graph requires fixed shapes, PD disaggregation requires moving state across nodes, and PagedAttention-style physical blocks require unified reclamation—all of these mechanisms must be redesigned around this state mapping.

    1.3 Mainstream Adaptation Approach

    The industry typically abstracts KV management into a mapping layer from “logical coordinates → physical addresses,” of which the shadow index mentioned above is a representative example. A request holds logical token coordinates, while the actual storage locations are allocated by each layer’s pool according to its own strategy, optionally combined with HiSparse-style sparse offloading. The implementations discussed below all build on the principle of “separating the logical layer from the physical layer.”

    2. Layer One: Unified Addressing—Multi-Pool KV Management and Shadow-Index Prefix Caching

    2.1 Pool Design Grouped by Layer Type

    In engineering practice, you can build independent pools by layer type:

    Layer type Stored content Typical lifetime Primary purpose
    Global layers K/V of all historical tokens within the window Entire session Guarantee that long-range information is not lost
    Sliding-window layers (SWA) K/V of the most recent W tokens Ring eviction Local modeling, constant memory cost
    Compressed layers (CSA/HCA-like) Compressed summaries of the full layers, and intermediate state needed during compression Entire session Carry long-range information with little state
    Indexer pool Low-rank index K for sparse block selection Corresponds to the compression window Support query-aware Top-K selection

    Sliding-window allocation can be viewed as a “ring queue with expiry time”: the number of request slots is determined by the window size, and as decoding advances, the head slot can be handed off immediately for reuse by other requests.

    2.2 The Shadow Index: Making Ring Slots Reusable by Prefix Caching

    Because ring slots get reused, the common prefix-caching strategy of “locating physical blocks by content hash” fails. The typical approach builds a two-level mapping:

    • Upper level (shadow index): keyed by the token sequence, it records the list of physical slots for that prefix in each layer’s pool;
    • Lower level (physical pool): each layer’s pool reclaims slots according to its own policy.
    1. # Illustrative code: multi-pool KV management and shadow indexing (locks, concurrency control, and other details omitted)
    2. class LayerPool:
    3. """KV pool for a single layer, e.g., a sliding-window or compressed layer"""
    4. def __init__(self, slot_bytes, num_slots):
    5. self.free = list(range(num_slots))
    6. self.slot_bytes = slot_bytes
    7. def alloc(self, n):
    8. slots = []
    9. for _ in range(n):
    10. if not self.free:
    11. raise OutOfMemoryError("need eviction / offload")
    12. slots.append(self.free.pop())
    13. return slots
    14. class ShadowIndex:
    15. """Shadow index: token prefix -> slots per layer"""
    16. def __init__(self):
    17. self.entries = {} # key: (prefix_hash, length), value: list of slots
    18. def lookup(self, prefix_hash, length):
    19. return self.entries.get((prefix_hash, length))
    20. def register(self, prefix_hash, length, slots_by_layer):
    21. self.entries[(prefix_hash, length)] = slots_by_layer
    22. class UnifiedKVMgr:
    23. def __init__(self, pools: dict[str, LayerPool], index: ShadowIndex):
    24. self.pools, self.index = pools, index
    25. def prefill(self, req, token_ids):
    26. hit = self.index.lookup(hash_prefix(token_ids), len(token_ids))
    27. if hit: # prefix hit: reuse existing slots directly
    28. return hit, 0
    29. slots = {name: pool.alloc(req.n_tokens[name])
    30. for name, pool in self.pools.items()}
    31. self.index.register(hash_prefix(token_ids), len(token_ids), slots)
    32. return slots, len(token_ids)
    33. def evict(self, victim):
    34. for slots in self.index.pop(victim.key):
    35. self.pools[victim.layer].free.extend(slots)

    The engineering weight of the shadow index lies in consistency:

    • Each layer’s pool needs to record separately “where valid data starts,” and the boundary of the reusable prefix is taken as the earliest-expiring layer;
    • On rollback (for example, when speculative decoding is rejected), only the newly allocated slots are returned, and the already-hit prefix is left untouched;
    • CUDA Graph requires that each layer’s tensor addresses stay fixed during replay, so slot allocation must complete before graph capture;
    • During PD disaggregation transfers, data must be moved grouped by “layer → pool”—the global layers, sliding-window layers, and index pool are transferred separately—each with valid-length metadata attached.

    3. Layer Two: Hybrid Attention Compute—SWA, Compressed Attention, and Block-Level Sparse Top-K

    Addressing solves “where the data lives”; the operator solves “what to compute.” Attention in this architecture is usually divided by layer:

    • Sliding-window layers: read only the most recent W tokens. The kernel is similar to FlashAttention’s online softmax, but must handle ring boundaries and positional-encoding wrap-around; you can think of it as “fixed-length local causal attention”;
    • Global layers: read the full history;
    • Compressed attention layers: for history outside the window, first aggregate it by compression ratio into “compressed tokens,” then attend over these together with in-window tokens;
    • Sparse block selection: in global layers or long-sequence scenarios, first filter out a small set of KV blocks most relevant to the current query, and perform fine-grained computation only within those blocks.

    3.1 Compressed Attention: Replacing Raw History with a Learnable “Summary”

    Compressed attention can be abstracted into two steps:

    1. Compression: divide a raw KV sequence (compression ratio r, i.e., every r tokens combine into 1 compressed token) into segments by position, aggregate each segment into compressed tokens via a small network, and rewrite the positional encoding to that segment’s representative position (such as the segment center);
    2. Reading: the query accesses both in-window raw tokens and compressed tokens, using a causal mask and positional bias to avoid ambiguity.

    It is complementary to sliding-window attention: the sliding window handles “near-field detail” while compression handles “far-field silhouette.” Stacking the two across different layers forms a multi-resolution context representation that goes from local to global and from fine to coarse.

    3.2 Two Sources of Compression Error

    • Compression is lossy: the degree of approximation depends on the compression ratio, the capacity of the compression network, and the training data distribution;
    • Prefix caching can amplify the error: once a prefix is cached, it is fixed to the compression result at a certain moment. If the compressed-layer state keeps being updated during generation (incremental compression), reusing the cache requires distinguishing “snapshot semantics” from “incremental semantics.”

    Engineering recommendation: before deployment, validate the combination of compression ratio and window size using long-context benchmarks such as “needle in a haystack” and multi-hop QA. Also check whether the engine implementation exactly matches the model’s training-time masking semantics, including: at which layer the causality boundary takes effect, whether compressed tokens can be seen by subsequent queries, and what positional-encoding interpolation method is used.

    3.3 Block-Level Sparse Top-K: Turning “See Everything” into “See Part of It”

    On extremely long sequences, the cost of full attention grows with sequence length. Query-aware hierarchical selection (Lightning Top-K-style approaches) splits attention into two stages:

    1. Block selection: use a lightweight index (low-rank index K, a shared single head, or a few heads) to compute the similarity between the query and each KV block, then take the Top-K blocks;
    2. Fine computation: run full attention only over the selected blocks and the local region covered by the sliding window.

    Its performance and payoff depend on several factors:

    • Representation of the index head: index K is usually lower-dimensional and stored at lower precision, to reduce the cost of reading;
    • Block granularity: larger blocks make the index cheaper but increase wasted reads;
    • Number of selection heads: a shared single head is enough to provide global routing; a few heads are more stable but costlier.

    In implementation, avoid letting “block selection” become a separate serial step; using a Flash Decoding-style “split-and-reduce” structure, you can fuse scoring, partial-softmax reduction, and output writes into the same kernel.

    1. # Illustrative code: block-level Top-K + two-stage attention
    2. def sparse_attention(q, k, v, block_size=128, topk=16, local_window=1024):
    3. # Stage 1: read in blocks and compute, in parallel, the similarity between the query and each block
    4. parts = []
    5. for blk_start in range(0, k.shape[0], block_size):
    6. blk = slice(blk_start, blk_start + block_size)
    7. score = (q @ k[blk].T) * q.scale # [B, H, blk_len]
    8. parts.append(partial_softmax(score, v[blk])) # local softmax statistics
    9. # Stage 2: aggregate block scores and select the Top-K blocks
    10. block_scores = merge_blocks(parts) # [B, H, n_blocks]
    11. chosen = top_k_blocks(block_scores, topk)
    12. # Stage 3: run fine-grained attention only over the chosen blocks and the local window
    13. return refine_with_local(q, k, v, chosen, local_window)

    Also, evaluate the index head’s precision separately. The block-selection bias introduced by a low-precision index shows up directly as a drop in long-context retrieval accuracy—a regression that is hard to detect from throughput metrics.

    4. Layer Three: Parallelism for Large-Scale MoE—EP, TP, and Context Parallelism

    4.1 Expert Parallelism: Communication Becomes the Main Battlefield

    Trillion-scale MoE models usually have hundreds of experts or more. Tensor Parallelism (TP)’s All-Reduce communication grows with the activation dimension, while Expert Parallelism (EP) introduces two All-to-Alls: one to dispatch tokens and one to gather results. Engineering must focus on:

    • Load imbalance: when a few “hot experts” are hit by many tokens, the corresponding devices finish compute first while the rest idle;
    • Dynamic capacity factor: set a capacity limit when bucketing tokens; overflow is either dropped or falls back to the shared path;
    • Expert redundancy: deploy multiple replicas of hot experts and let the routing layer do soft assignment (similar to the EPLB approach widely used in the DeepSeek series).

    Illustrative implementation:

    1. # Illustrative code: token dispatch and load balancing in expert parallelism
    2. def moe_layer(x, router, experts, capacity_factor=1.25, n_redundant=1):
    3. # 1) Routing: top-k gating
    4. weights, topk_idx = router(x) # [T, k]
    5. # 2) Load balancing: replicate hot experts and balance load across devices
    6. placement = rebalance(topk_idx, experts, n_redundant=n_redundant)
    7. # 3) Dispatch: All-to-All sends tokens to the devices hosting the target experts
    8. x_ep, meta = all_to_all_dispatch(x, topk_idx, placement)
    9. # 4) Bucketed compute: Grouped GEMM over expert-grouped batches
    10. capacity = int(capacity_factor * meta.max_load)
    11. buckets = pad_and_bucket(x_ep, capacity)
    12. y_ep = grouped_gemm(buckets, experts)
    13. # 5) Combine: All-to-All merges the results
    14. y = all_to_all_combine(y_ep, weights, meta)
    15. return y

    4.2 Context Parallelism: The “Second Dimension” for Long Context

    When the sequence length reaches hundreds of thousands of tokens, having a single device carry all attention state is unrealistic. Context Parallelism (CP, also called sequence parallelism) partitions K/V along the sequence dimension and uses Ring Attention-style communication so that every shard can see the full context:

    1. # Illustrative code: Ring-style context-parallel attention
    2. def cp_attention(q_local, kv_shards, my_rank, world_size):
    3. acc, lsum = None, 0.0
    4. for step in range(world_size):
    5. src = (my_rank + step) % world_size
    6. dst = (my_rank - step) % world_size
    7. kv = kv_shards[src] # shard currently held
    8. out_i, l_i = flash_attn(q_local, kv)
    9. acc, lsum = online_merge(acc, lsum, out_i, l_i)
    10. send_recv_inplace(kv, dst) # ring transfer: overlapped with compute
    11. return acc / lsum

    CP and TP focus on different things:

    • TP partitions weights and attention heads, with communication concentrated inside each layer and dominated by All-Reduce;
    • CP partitions the sequence, with communication concentrated inside the attention layer; after overlapping with compute the overhead is manageable, but it changes the numerical reduction order.

    CP also imposes direct requirements on KV pool management: pool slots must be organized by “sequence shard × layer type,” every physical slot in the shadow index should carry a rank dimension, and PD transfers must align by shard.

    4.3 DP Attention: Data Parallelism Along the Attention-Head Dimension

    Under a “many experts + many heads” configuration, pure TP replicates every attention head across all TP ranks, causing redundant compute and repeated KV reads. DP Attention works by data-parallelizing the attention part across requests while keeping TP/EP for the FFN and MoE parts. Key implementation points include:

    • Layout conversion between the two stages (similar to All-Gather/Reduce-Scatter), overlapped with compute as much as possible;
    • Independent KV pool management per DP group, with batch scheduling done at the DP-group level;
    • Cross-group load balancing, otherwise a “bucket effect” appears across groups with different loads.

    4.4 Deployment Form Comparison

    Dimension TP EP CP DP Attention
    Partition target Weights + attention heads Experts Sequence Requests
    Main communication All-Reduce All-to-All Ring P2P Gather/Scatter
    Main benefit Reduces per-device weight and KV pressure Scales expert count Supports ultra-long context Eliminates redundant attention compute
    Main cost Communication grows with activation dimension Load imbalance, routing jitter Communication latency on long sequences Higher scheduling complexity

    Real deployments are usually a combination of several parallelism strategies rather than an either/or choice. The principle: first use TP/EP to solve what a single device cannot hold, then use CP to solve sequences that are too long, and finally use DP Attention to remove redundancy on the attention side.

    5. Layer Four: PD Disaggregation—Letting Two Workloads Run at Their Best “Shapes”

    5.1 Why PD Disaggregation Is Needed

    The Prefill stage processes an entire input, is compute-intensive, and suits large batches and high-compute devices. The Decode stage generates only one token per step, is memory-intensive and latency-sensitive, and is more sensitive to the number of concurrent requests within a batch. Co-locating the two causes mutual interference: a long-prompt prefill can “stall” requests that are decoding, causing jitter in inter-token latency (ITL).

    PD disaggregation works as follows:

    1. Use a group of instances dedicated to prefill, and once generation completes, hand off the KV state to the decode instances;
    2. The decode instances handle only incremental generation.

    The key is that KV transfer must overlap with compute: the transfer layer is organized by “layer → pool,” using chunked transfer with concurrent multi-streams, and metadata carries each layer’s valid length; the completion check for transfer must align with the launch timing of the first decode computation, to avoid a “waiting for KV” bubble.

    1. # Illustrative code: KV transfer and scheduling under PD disaggregation
    2. def pd_pipeline(prefill_done_event, decode_worker, kv_transfer):
    3. # Prefetch: prepare metadata for requests that are about to arrive
    4. meta = kv_transfer.wait_metadata(prefill_done_event)
    5. # Layer-by-layer, pool-by-pool transfer, overlapped with first-token compute
    6. kv_transfer.stream_layers(meta, pool_order=["global", "swa", "compress", "indexer"])
    7. # Decode side receives in chunks, computing as data arrives
    8. decode_worker.run(streaming=True, meta=meta)

    5.2 Working with CUDA Graph

    CUDA Graph requires fixed shapes. In practice, you typically pre-capture a set of graphs for “max batch × max sequence,” then pad runtime requests up to the nearest bucket. For hybrid-attention models, before graph capture you must complete:

    • Static address binding for each pool;
    • Initialization of the sliding-window ring pointers;
    • Pre-allocation of index K and compression state.

    Otherwise, the first real request triggers implicit memory allocation, causing graph-replay failure or latency jitter.

    6. Layer Five: MTP Speculative Decoding—Building the Draft Model into the Main Model

    6.1 Principle

    Traditional speculative decoding needs a separate small model as the drafter, incurring three overheads: weight loading, KV maintenance, and scheduling. Multi-Token Prediction (MTP) builds the drafting capability directly into the main model: the main model’s forward pass additionally outputs multiple candidate tokens, which the main model itself then verifies in a single pass.

    Its advantage: draft and verification share the same set of weights and KV state, so they align naturally; the extra overhead is mainly one lightweight forward pass, not a full-model invocation.

    6.2 Engineering Points

    • Adaptive draft depth: longer drafts accept more tokens per step, but verification cost also grows. A common approach is to adjust draft depth dynamically based on historical acceptance rate rather than using a fixed value;
    • KV rollback: KV writes corresponding to rejected candidate tokens must be rolled back precisely—the same mechanism as the slot-reclamation logic of the shadow index;
    • Coordination with PD disaggregation: MTP verification happens in the decode stage, so prefill output must fully preserve the hidden states needed for the “next step”;
    • Coordination with sparse block selection: block-selection results depend on the query, and candidate tokens change the query, so the verification stage must re-run block selection and cannot reuse the previous selection set.
    1. # Illustrative code: main loop of MTP speculative decoding
    2. def mtp_decode(model, req, max_new_tokens):
    3. out_tokens, hidden = [], req.last_hidden
    4. while len(out_tokens) < max_new_tokens:
    5. drafts = model.draft(hidden, depth=model.adaptive_depth()) # multiple candidates
    6. accepted, hidden = model.verify_and_commit(drafts) # single verification
    7. out_tokens.extend(accepted)
    8. if model.finished(accepted):
    9. break
    10. return out_tokens

    It should be emphasized that the benefit of speculative decoding depends on the acceptance rate, and the acceptance rate is strongly correlated with the task distribution. Acceptance rates for “low-entropy” tasks like code and structured output are usually higher, while open-ended conversation varies more. Before launch, run A/B validation on real traffic.

    7. Capacity Planning and Tuning: From “It Runs” to “It Runs Stably”

    7.1 GPU Memory Budget

    A single instance’s GPU memory requirement can be approximately split into four parts:

    1. GPU memory need ≈ weight memory + total KV pool capacity + activations and temporary buffers + communication buffers

    Here, KV pool capacity is the key variable:

    1. KV bytes ≈ num_layers × bytes per KV element per token × sum of state elements per layer × concurrent sequence length

    The advantage of hybrid-attention models lies exactly here: the state elements of sliding-window layers are truncated by the window size, and those of compressed layers are shrunk by the compression ratio. Under the same concurrency, KV usage may be only a fraction of a traditional full-KV scheme. This is also why “long context” for these models often does not mean “memory explosion.”

    7.2 List of Tunable Parameters

    Parameter Role Tuning direction
    Sliding-window size W Local receptive field Align strictly with the training config; do not change arbitrarily
    Compression ratio r Long-range information retention vs. state cost Prioritize recall for long-context tasks
    Top-K blocks / block granularity Precision-speed tradeoff of sparse attention Start with a large K as a baseline, then converge gradually
    Index head precision Block-selection accuracy Low-precision indexes need recall verification
    Expert capacity factor Drop rate vs. memory Tune against the real routing distribution
    Number of redundant experts Load-balancing effectiveness Enable first in scenarios with many hot experts
    Draft depth Acceptance rate vs. verification cost Adaptive beats fixed values
    Max batch / chunked prefill granularity ITL jitter vs. throughput Prioritize small chunks for latency-sensitive scenarios

    7.3 Phased Tuning Roadmap

    1. Align semantics first: confirm that the engine implementation matches the model’s training-time masking, positional encoding, and compression semantics. Deviations here do not show up as crashes, but as quality degradation;
    2. Then plan capacity: work backward from the concurrency target to size the KV pools and sliding-window configuration, ensuring eviction is not triggered at peak concurrency;
    3. Then tune the parallelism strategy: settle the TP/EP combination first, then decide whether to introduce CP and DP Attention;
    4. Next, stress-test latency: measure Time-to-First-Token (TTFT) and inter-token latency (ITL) separately, and locate whether the bottleneck is prefill or decode;
    5. Finally, enable speculative decoding: introduce MTP only after throughput and latency targets are met, to avoid premature optimization masking the real bottleneck.

    7.4 Pre-Launch Checklist

    • Do long-context benchmarks (needle in a haystack, multi-hop QA) pass?
    • Are the numerics at sliding-window boundaries consistent with the reference implementation?
    • After a prefix-cache hit, are each layer’s valid lengths correct?
    • When eviction happens, is there any case of “an already-hit prefix being reclaimed by mistake”?
    • Is expert load balanced? Are there devices idle long-term?
    • Under PD disaggregation, does KV transfer overlap with first-token computation?
    • Does CUDA Graph capture cover all common shapes?
    • Is speculative decoding’s acceptance rate stable on real traffic?

    8. Cloud Deployment: Breaking a Complex System into Operable Modules

    In the cloud, the modules above typically map to a few foundational capabilities: a container platform to host inference-instance orchestration and elastic scaling, object storage to hold model weights and index data, monitoring and alerting services to track metrics such as TTFT, ITL, GPU memory usage, and expert load, and logging services to accumulate routing distributions and eviction events for later tuning.

    On platforms such as Baidu AI Cloud, these capabilities are typically offered as general-purpose products that developers can combine as needed. It should be noted, however, that the specific parallelism strategy, and the sliding-window and compression configuration for a hybrid-attention model, still have to be determined by the business side based on the model structure. The cloud platform provides the hosting and operations foundation, not a substitute for model semantics.

    Conclusion: The Center of Gravity in Inference Systems Is Shifting from “Stacking Compute” to “Managing State”

    Looking back across the whole pipeline, the real challenge trillion-scale MoE models pose to inference systems lies not in parameter count itself, but in the fact that the attention runtime’s state space is split into multiple layers, multiple pools, and multiple lifetimes. A system that handles this gracefully needs four capabilities at once:

    • Unified addressing: use a shadow index to map heterogeneous physical pools onto a consistent logical coordinate space;
    • Efficient compute: use SWA, compressed attention, and block-level sparse Top-K to bring down the cost of long sequences;
    • Flexible parallelism: use EP, CP, and DP Attention to partition weights, sequence, and requests along three dimensions respectively;
    • Decoupled scheduling: use PD disaggregation and MTP speculative decoding to optimize the two workloads—compute and memory access—separately.

    For developers, the difficulty of landing this architecture lies not in any single technology but in the interface contracts between modules. How pools are indexed, how the index is transferred, how the transfer is graph-captured, and how graph capture is scheduled—any inconsistency in semantics at any link gets amplified into a quality problem in long-context scenarios.

    If you are selecting an inference solution for an ultra-large-scale MoE model, it is worth first evaluating on the dimension of “is state management clear?” before discussing throughput numbers. The former determines the system’s ceiling; the latter only determines how far it is from that ceiling.