Deploying Ultra-Large MoE Models on AI Accelerators: Memory, Kernels, and Load Testing

    This article examines the practical constraints of deploying ultra-large Mixture-of-Experts (MoE) models on AI accelerators. It covers single-node memory and bandwidth accounting, fused decode kernels, sparse attention for long-context inference, replay-based load testing, and a reusable path from deployment to production. It explains why sparse activation reduces compute but not resident memory, why activated parameters set the decode-speed floor, why KV cache dominates long-context bandwidth, and how teams should read load-test metrics without overestimating capacity.

    Introduction: Inference Optimization Is Shifting Toward Systems-Level Design

    The evolution of large-model inference is becoming clearer. In the early stage, the focus was on general-purpose compute, with optimization centered on batching and memory management. In the next stage, the focus moved to precision compression and operator fusion. Today, competition is extending to long-context workloads, agentic workloads, and hardware-software co-design.

    One important signal is that self-developed AI accelerator cards from chip vendors are now being used for inference of ultra-large MoE models. According to public information, one open-source inference engine has deployed, on a single node, a MoE model with 320B total parameters, about 18B activated parameters, and FP8 precision, targeting a self-developed AI training accelerator from a cloud provider. The inference engine is released under the permissive Apache 2.0 license.

    This matters for more than simply proving that the model can run. It shows that inference performance now comes less from raw compute stacking alone and more from systematic coordination across memory and bandwidth management, kernel fusion, attention algorithms, and batch scheduling. This article unpacks the technical logic behind this deployment pattern and provides a reusable analysis framework for teams deploying large models on heterogeneous compute.


    1. Scenario Breakdown: Three Constraints for Single-Node Ultra-Large MoE Deployment

    Real traffic generated by agents differs from evaluation workloads that focus on a single long-output turn. It has three characteristics:

    • Extremely long context: full conversation history, tool-call traces, and retrieval results accumulate, so a single request may reach hundreds of thousands or even millions of tokens.
    • Concentrated concurrency: dozens of agents may be active during the same period.
    • Stretched throughput metrics: beyond total throughput, the decode speed of each request is also critical.

    To meet these goals on a single node, three constraints must be solved:

    1. Fit: weights and KV cache must not exceed memory.
    2. Survive: attention must not crush decoding under million-token context.
    3. Run fast: heterogeneous accelerators often lack mature operator libraries, so high-performance kernels may need to be developed in-house.

    The following sections break these down.


    2. Memory and Bandwidth: The Real Cost Behind MoE Inference

    2.1 Weights: MoE Does Not Inherently “Save Memory”

    A common misunderstanding is that sparse activation in MoE automatically saves memory. Each token activates only a small subset of experts, so it is easy to assume that this means less memory usage.

    Sparse activation reduces the compute and weight reads required for each token, but it does not reduce the total amount of weights that must remain resident in memory. Experts that are not selected for a given token still need to stay resident so they can be routed to later.

    For a 320B-total-parameter MoE model:

    • Under BF16, the weights require about 640GB.
    • Under FP8, they require about 320GB, roughly half.

    Lower precision helps both capacity and bandwidth, but expert-routing overhead also becomes relevant. In addition, under expert parallelism or tensor parallelism, the actual per-card read volume depends on the parallelism strategy. The numbers above should not be applied blindly across different deployment modes.

    2.2 Activated Parameters Set the Floor for Decode Speed

    Autoregressive decoding reads the activated expert weights once for every generated token. This makes it a typical memory-bandwidth-bound workload.

    In a cross-chip tensor-parallel deployment, the weights read per decoding step are approximately:

    1. activated parameter count × bytes per parameter

    For example, with 18B activated parameters and FP8 computation, a single sequence must move at least 18GB of weights for each token. Across 16 accelerator cards in a node, this is about 1.1GB per card.

    If each card has an effective memory bandwidth of 1TB/s, the 16 cards provide about 16TB/s in aggregate. In that case, one weight-read step takes about 1.1ms.

    From this, we can estimate an approximate upper limit for single-sequence decoding when counting only weight reads and excluding KV-cache reads and collective communication: about 1000 token/s.

    This limit does not include KV-cache reads, communication, or scheduling overhead, so the real floor can only be higher. It is also a single-sequence estimate. Aggregate throughput under multiple concurrent requests must be estimated separately according to concurrency and context length, rather than copied directly. This is one of the fundamental reasons to keep batch size small and to make attention sparse.

    2.3 KV Cache Is the Real Bottleneck for Long Contexts

    Weight memory usage is constant, but KV cache grows linearly with context length:

    1. KV bytes ≈ 2 × number of layers × number of KV heads × head dimension × sequence length × bytes per element

    For example, with 60 layers, 8 KV heads, head dimension 128, and FP8 storage, each token takes about 120KB:

    • 32,768 tokens: about 3.9GB
    • 262,144 tokens: about 31GB
    • 1,048,576 tokens: about 123GB

    In other words, the KV cache of a single million-token request is already close in magnitude to all the FP8 model weights.

    The bandwidth problem is even harder. During decoding, every step must traverse all historical KV entries. When context doubles, the KV read volume per token also doubles.

    For example, a single-sequence 1M context requires about 123GB of KV cache. If 32 sequences are all at full context, the cluster must hold about 3.9TB in total. Assuming 16 cards with 1TB/s each in aggregate, that is about 246GB per card. Reading the KV cache once would take about 250ms.

    At this point, aggregate decode speed is pushed down to about 128 token/s:

    1. 32 tokens / 0.25s ≈ 128 token/s

    This leads to an important conclusion: under dense attention, cluster aggregate throughput does not grow linearly with concurrency. The longer the context, the lower the single-sequence decode speed becomes. Concurrency is a double-edged sword: it can increase total throughput when attention overhead is small, but under million-token context it makes each request slower and slower.

    These numbers are order-of-magnitude illustrations used to identify optimization directions. Actual results should be based on measurement.


    3. Decode Kernels: Replacing Dozens of Small Kernels with Two Fused Kernels per Layer

    3.1 Breaking Down the Decode Path

    Take a typical “attention + MoE” layer as an example. A single decode forward pass usually contains more than a dozen fine-grained operators:

    1. qkv_proj → rope → attention → o_proj → all_reduce
    2. → router → permute → expert_gemm → unpermute
    3. → residual+norm → mlp_proj → moe_proj → all_reduce …

    When each operator is launched separately, each one becomes an independent kernel launch. During decoding, many matrix operations are matrix-vector products or thin matrix-matrix products. A single operator may run for only a short time, so launch and synchronization overhead become amplified.

    Intermediate results are written back to memory and then read again, creating avoidable memory traffic. Every communication step also introduces synchronization waiting, which can make accelerators slow each other down.

    If the ideal lower bound of weight reading per step is only about 1ms, but one layer alone costs several hundred microseconds, then across dozens of layers the wasted scheduling and memory traffic can exceed the time spent on “real work.”

    3.2 Three Levels of Fusion: From Operator Fusion to Megakernel

    Fusion Level Approach Benefits Limitations
    Operator-level fusion Fuse attention kernels, fuse MoE kernels Small modification, easy to validate Still more than ten launches and memory round trips within a layer
    Intra-layer fusion Compress one layer into one or a few kernels Reduces launches and intermediate memory traffic Puts heavy pressure on registers and on-chip memory in a single kernel
    Megakernel Put the entire forward pass into a few very large kernels Intermediate results stay on chip, lowest latency Complex development, difficult debugging, poor portability

    A megakernel integrates computations that were originally performed by many small kernel functions into a single very large kernel. Intermediate results remain resident on chip and move only as needed. This approach is clearly more expensive from an engineering perspective, but it can provide obvious benefits in latency-sensitive decoding scenarios. Research and system practice have explored this direction on platforms such as GPUs.

    3.3 Why Two Fused Kernels per Layer?

    The deployment discussed here chooses a compromise: each decoding layer is split into two fused kernels, namely a front-half kernel and a back-half kernel.

    • The front-half kernel handles QKV projection, positional encoding, attention, output projection, and any necessary tensor-parallel communication.
    • The back-half kernel handles MoE routing, expert computation, weighted summation, and the corresponding communication.

    These two half-kernels have a natural data-flow boundary. They only need to pass one hidden-state tensor between them, so their coupling surface is small.

    This split has three advantages:

    1. Manageable scale: a half-kernel is much smaller than a full megakernel, which greatly reduces on-chip resource pressure and development difficulty.
    2. Natural cut point: the boundary between attention and expert computation is natural and does not require extra glue logic.
    3. Preserved synchronization point: the boundary between the two half-kernels can serve as a synchronization point between layers or devices, making pipelining and communication overlap easier.
    1. # Pseudocode: split each decode layer into two fused kernels (front half + back half)
    2. for layer in model.layers:
    3. h = attn_half_kernel(hidden, layer) # Front half: projection + RoPE + sparse attention + output projection + communication
    4. h = ffn_half_kernel(h, layer) # Back half: MoE routing + expert computation + weighted summation + communication
    5. hidden = h

    Compared with launching many individual operators several times per layer, this “two-kernel-per-layer” approach compresses both the number of launches and memory round trips within a layer by roughly an order of magnitude.

    3.4 Costs and Validation: Correctness Regression Cannot Be Skipped

    Fused kernels also come with costs:

    • Development and maintenance costs rise.
    • The team must understand the chip-specific kernel programming model.
    • Debugging a faulty fused operator is much harder than debugging an operator-by-operator implementation.
    • FP8 quantization combined with changed fused computation order may introduce numerical differences. Key layer outputs should be compared against operator-by-operator reference implementations, with tolerance thresholds defined.
    • A non-fused implementation should be retained as a fallback path for debugging and gradual rollout.

    4. Million-Token Context: Sparse Attention Is a Prerequisite

    The bandwidth calculations in Section 2.3 show that even with fully fused kernels, KV-cache reads under dense attention grow linearly with context length. To support both long context and dozens of concurrent requests, attention computation must read less KV.

    4.1 Main Technical Routes

    Route Core Idea Characteristics Fit with Decode Batching
    Fixed pattern Local window plus a small number of global tokens Simple to implement, predictable Fixed memory usage, easy for paged management
    Block selection Split KV into blocks, compute only selected blocks based on scores Friendly to on-demand retrieval workloads Requires extra selection and gather overhead
    Dynamic sparse Select Top-k tokens based on attention scores Flexible expression Selection logic per step becomes complex
    Compression / reuse Compress historical KV, share prefixes across requests Reduces total volume Naturally combines with paged caching

    Industry exploration has produced several mature routes, and engineering practice usually combines them:

    • Recent tokens are retained with a local window.
    • A small number of tokens act as global anchors and participate throughout.
    • Historical tokens participate selectively based on importance.

    4.2 Key Distinction: Saving Memory vs. Saving Bandwidth

    If attention only computes less but still stores all KV, memory pressure remains and million-token context is still not feasible.

    Only when selected tokens are loaded on demand and kept resident through paged management can context truly be “held up.”

    Therefore, sparse attention must be combined with paged KV management and prefix reuse.

    4.3 Required Quality Regression

    Sparse attention loses information. Its impact on long-context tasks must be evaluated specifically, including:

    • Needle-in-a-haystack retrieval
    • Multi-hop reasoning
    • Long-range instruction following

    Recommended practices:

    • Build a regression set covering long-document QA, cross-turn tool calls, and long-conversation summarization.
    • Compare against a dense-attention baseline and define acceptable metric fluctuation thresholds.
    • After launch, continuously monitor user feedback and error rates in long-context scenarios.

    5. Reading Load Tests: From Replay Data to Production Capacity

    5.1 Why Use Agent Replay Instead of Traditional Benchmarks?

    Traditional load tests often use fixed input and output lengths. These are difficult to reflect real agent workloads, which tend to involve long context, multi-turn calls, and inputs that are much larger than outputs.

    One inference engine publicly shared a set of measured results based on agent interaction replay:

    • Single node
    • 32 concurrent agents
    • 1M context
    • Continuous run for one hour
    • Default batch size: 12

    During that hour:

    • Average throughput: 28.9 requests per minute
    • Total tokens processed: about 3.57 million
    • p50 decode speed: 22.3 token/s

    5.2 How to Read the Numbers

    1. Total throughput
      About 3.57 million tokens / 3600 seconds ≈ 993 token/s.
      This includes both input (prefill) and output (decode), and reflects overall cluster production.

    2. Request rate
      28.9 requests per minute ≈ 0.48 requests per second.
      Based on this, the average request is about 2060 tokens.
      If output length is not provided, it is impossible to determine directly whether input dominates absolutely. Conclusions should be drawn together with the number of decoded output tokens.

    3. Decode speed
      p50 is 22.3 token/s. This is the direct experience of how fast a single user sees tokens generated.

    4. Comparison with the theoretical upper limit
      The earlier estimate put the single-sequence weight-read upper limit at about 1000 token/s. This is the same order of magnitude as the 993 token/s total throughput and can serve as a coarse capacity reference.
      However, to determine whether the system is truly close to the memory-bandwidth limit, further breakdown is needed: prefill, decode, KV reads, communication, and p50/p99 latency. Total throughput alone is not enough.

    5.3 Common Load-Testing Mistakes

    Mistake Consequence Recommended Practice
    Only looking at peak throughput Single-user experience is ignored Report p50/p90/p99 decode speed and time to first token
    Imbalanced input/output lengths Cannot reflect real cost Construct replay data based on actual business input/output ratios
    Ignoring the share of long-context traffic Overestimates capacity Report throughput and latency for long-context scenarios separately
    No warm-up First requests distort metrics Collect stable data after warm-up
    Lack of controls Cannot attribute bottlenecks Record batch size, context-length distribution, and cache hit rate together

    6. From Zero to Production: A Reusable Deployment Path

    6.1 Five-Step Process

    1. Prepare the environment
      Align accelerator-card drivers, firmware, and runtime versions. Write them into a container-image baseline to avoid “it runs locally but fails in production.”

    2. Build the image
      Fix the inference engine, dependencies, and kernel artifacts into the image. Distribute through an image repository.

    3. Prepare weights and compilation cache
      Complete weight-format conversion and FP8 quantization calibration. Precompile kernels for the target hardware and cache them to avoid recompiling on every startup.

    4. Start the service
      Load weights and cache, then expose an OpenAI-compatible API.

    5. Smoke test and load test
      First verify interface and output correctness, then run replay-based load tests.

    6.2 Example Configuration and Invocation

    1. # Illustrative configuration; field names should follow the specific engine documentation
    2. model:
    3. path: /models/large-moe-fp8
    4. dtype: fp8
    5. max_context_length: 1048576
    6. max_batch_size: 12
    7. kv_cache_dtype: fp8
    8. attention:
    9. mode: sparse # Sparse attention
    10. sliding_window: 4096 # Local window
    11. global_tokens: 512 # Global anchors
    12. server:
    13. host: 0.0.0.0
    14. port: 8080
    15. api_style: openai_compatible
    1. # Example invocation using the OpenAI-compatible API style
    2. curl -s http://127.0.0.1:8080/v1/chat/completions \
    3. -H "Content-Type: application/json" \
    4. -d '{
    5. "model": "large-moe-fp8",
    6. "messages": [{"role": "user", "content": "Explain in three sentences why MoE saves compute but not memory"}],
    7. "max_tokens": 512,
    8. "stream": true
    9. }'

    The value of a compatible API is that upper-layer applications can migrate without changes. This makes it easier to compare backends and perform gradual rollout.

    6.3 Production Checklist

    • Driver, firmware, and runtime versions are consistent and traceable.
    • Quantization calibration is complete and precision regression has passed.
    • Compilation cache has been generated and distributed with the image.
    • Service startup time and time to first token are within acceptable ranges.
    • API compatibility passes automated test cases.
    • Monitoring metrics such as throughput, p50/p99 latency, memory usage, and error rate are integrated into alerting.

    7. Supporting Practices: Observability, Capacity Planning, and Cloud Deployment

    7.1 Observability

    In addition to request volume and error rate, teams should monitor:

    • Memory usage and KV-cache utilization
    • Actual batch size
    • Cache hit rate
    • Per-card utilization
    • Kernel execution time distribution

    These signals can be connected to logging services and monitoring/alerting systems. Thresholds can be defined and linked to autoscaling.

    7.2 Capacity Planning

    • Pool scheduling by context length
      Group requests by context length. Long-context requests can be queued separately so they do not slow down short requests.

    • Separate prefill and decode
      Prefill is compute-intensive, while decode is memory-intensive. Separating their deployment is a common industry practice.

    • Prefix caching
      Fixed prefixes such as system prompts and tool definitions can be reused across requests, significantly reducing repeated prefill cost.

    7.3 Cloud Deployment

    If a team does not yet have its own heterogeneous compute infrastructure, it can choose cloud instances with heterogeneous accelerators on demand. It can also use:

    • Object storage for model weights and compilation cache
    • A container platform to host inference services
    • Monitoring and alerting to track service quality

    Major cloud providers typically offer general heterogeneous-compute and cloud-product capabilities that can serve as deployment references.

    One important note: weight files and compilation caches may contain sensitive information, and monitoring data may involve business metrics. Access controls and compliance auditing should be put in place.

    Regardless of deployment form, capacity validation should be based on replay data from the team’s own workload.


    8. Conclusion: Three Takeaways

    1. Calculate first, optimize second.
      Memory and bandwidth estimates for weights, activated parameters, and KV cache determine the upper bound of optimization. In memory-bound decode scenarios, kernel fusion can approach the limit, but it cannot break through it.

    2. Long context requires “store less + compute less.”
      Sparse attention must be combined with paged KV management to support million-token context, and quality regression must be performed at the same time.

    3. Read the right load-test metrics.
      Total throughput and single-user decode speed should be reported separately, and long-context and short-context scenarios should be distinguished. Only then can capacity decisions be supported properly.

    Returning to the data at the beginning: single node, million-token context, nearly 29 requests per minute, and p50 decode speed of 22.3 token/s. The value is not in any single number. It is in a reusable method: calculate first, fuse kernels, load-test correctly, and keep regression ongoing.

    As heterogeneous compute becomes the norm, this method is worth preserving more than optimization tricks tied to any single platform.