MoE LLMs for Inference: Balancing Speed and Cost
Inference is often the ongoing cost center for large language models, so inference efficiency directly affects a product's unit economics. This article explains how Mixture of Experts (MoE) models separate total parameters from active parameters, why decoding is often constrained by memory bandwidth, and how batching, KV cache usage, quantization, speculative decoding, continuous batching, chunked prefill, attention design, and prefix caching affect speed and cost. It also covers the hidden cost of chain-of-thought reasoning, how to interpret token pricing, and the practical constraints of edge and automotive deployment.
1. Why Inference Becomes the Cost Center
The cost of large models has two parts: training and inference. Training is a one-time investment, while inference continues as request volume grows. After a model goes live, serving cost usually grows roughly linearly with request volume, so inference efficiency directly determines the product’s unit economics.
From an engineering perspective, inference serving must satisfy three types of constraints at the same time:
- Response latency: whether users can see the first token within an acceptable time, and how long the full answer takes to complete.
- Unit cost: the compute, memory, and electricity cost per request or per successful task.
- Deployment location: whether the model runs on a cloud cluster, a phone, an automotive compute unit, or an edge server. Each has very different compute, memory, and power budgets.
These three constrain each other. Lowering latency usually requires more parallel resources, which raises cost. Reducing cost may sacrifice output quality or throughput. When selecting a model or deployment approach, first identify which constraint is hard, then make trade-offs across the remaining dimensions.
2. Mixture of Experts: Total Parameters vs. Active Parameters
The Mixture of Experts (MoE) architecture splits the feed-forward layer into multiple expert sub-networks. A router computes a score for each expert for every input token, and only the top-k experts are activated for computation.
Below is a simplified top-k routing example to illustrate the computation flow:
import numpy as npdef moe_forward(x, gate_w, experts, k=2):"""x: input vector (d,); gate_w: router weights (num_experts, d); experts: list of expert functions"""logits = gate_w @ xtopk = np.argsort(logits)[-k:] # Select the k experts with the highest scoresweights = np.exp(logits[topk] - logits[topk].max())weights /= weights.sum() # Normalize into combination weightsreturn sum(w * experts[i](x) for w, i in zip(weights, topk))
This structure has several direct effects:
- Compute is determined by active parameters. Each token passes through only k experts, so the single-step forward computation is approximately proportional to the active parameters. At the same total parameter scale, MoE uses far less computation per token than a dense model.
- Memory footprint is determined by total parameters. During inference, all expert weights must reside in GPU memory or system memory, even though a single token uses only a small portion of them. The larger the total parameter count, the higher the deployment barrier.
- Routing carries a load-imbalance risk. If most tokens are routed to a small number of experts, those experts become bottlenecks while the rest sit idle. Training usually uses load-balancing losses or bias adjustments to mitigate this.
- Distributed deployment introduces communication overhead. When experts are spread across multiple GPUs, tokens must be forwarded between devices through all-to-all communication. Network bandwidth and latency directly affect throughput.
3. Breaking Down Speed and Cost
3.1 Why Decoding Is Limited by Memory Bandwidth
Autoregressive decoding reads model weights once for every generated token. At small batch sizes, compute units spend most of their time waiting for data movement, so the bottleneck is memory bandwidth rather than compute.
A rough throughput upper bound can be estimated as follows:
def decode_tokens_per_sec_upper_bound(bandwidth_gb_s, active_params_b, bytes_per_param):"""Ignore KV cache reads and scheduling overhead; return the theoretical upper bound for single-request decoding speed"""bytes_per_token_gb = active_params_b * bytes_per_param # Weight bytes that must be read per token, in GBreturn bandwidth_gb_s / bytes_per_token_gb# Example: assume device bandwidth 1000 GB/s, model active parameters 10B, INT8 quantization (1 byte per parameter)print(decode_tokens_per_sec_upper_bound(1000, 10, 1)) # about 100 tokens/s
This result shows two things. First, reducing the number of bytes per parameter through quantization can directly raise the theoretical decoding speed upper bound. Second, fewer active parameters means less data read per token. MoE improves this by reducing active parameters.
3.2 Batch Size and Throughput
Merging multiple requests into one batch allows weight reads to be amortized across multiple tokens, which raises system throughput. The trade-off is that each request may wait longer, and first-token latency can also be affected by queueing.
For MoE models, the benefit of batch size saturates earlier. As the batch grows, different tokens activate more dispersed experts, so more expert weights must be read. The per-token amortized read volume therefore decreases more slowly than in dense models. As a result, the optimal batch size for MoE models must be determined empirically; dense-model rules of thumb cannot be applied directly.
3.3 KV Cache Memory Usage
Attention needs to cache the keys and values of historical tokens during decoding (the KV cache). Long contexts and high concurrency can make KV cache the dominant consumer of memory.
def kv_cache_bytes(layers, kv_heads, head_dim, seq_len, batch, bytes_per_elem=2):"""Estimate KV cache usage; the factor 2 corresponds to the K and V tensors"""return 2 * layers * kv_heads * head_dim * seq_len * batch * bytes_per_elem
This formula suggests three optimization directions: reduce the number of KV heads (grouped-query attention), shorten the effective context (sliding window), and reduce the number of bytes per element (KV cache quantization).
4. Common Inference Optimizations
Quantization. Compressing weights from 16-bit to 8-bit or 4-bit can significantly reduce memory usage and bandwidth requirements. The cost is precision loss, and different tasks are affected differently. Numerically sensitive tasks such as mathematical reasoning and code generation usually need separate validation.
Speculative decoding. A small model quickly generates candidate tokens, which a large model then verifies in parallel. With the standard rejection-sampling implementation, the output distribution matches the original model, making this a lossless acceleration method. The benefit depends on the acceptance rate of candidate tokens; when the acceptance rate is low, it can even become slower.
Continuous batching and chunked prefill. Continuous batching allows requests to join or leave the batch at any step, reducing GPU idle time. Chunked prefill splits long prompts into smaller chunks and interleaves them with decoding for other requests, reducing the impact of long inputs on other users’ latency.
Attention structure optimization. Grouped-query attention reduces the number of KV heads, and sliding-window attention limits the history visible to each token. Both change the model structure and must be determined during training; they cannot be obtained only through inference-side configuration.
Prefix caching. When multiple requests share the same system prompt or document prefix, the computed KV cache can be reused. For scenarios such as fixed-template customer service and retrieval-based question answering, the benefit is often clear.
These methods can often be combined, but the combined effect must be confirmed through end-to-end load testing. Individual gains cannot simply be multiplied.
5. Hidden Costs of Chain-of-Thought Reasoning
Reasoning models generate longer intermediate reasoning content before producing the final answer. This content is usually billed as output tokens and occupies decoding time.
The cost of a single request can be broken down as follows:
def request_cost(input_tokens, output_tokens, price_in_per_m, price_out_per_m):"""Prices are amounts per million tokens; return the cost of a single request"""return (input_tokens * price_in_per_m + output_tokens * price_out_per_m) / 1_000_000def decode_latency_seconds(output_tokens, tokens_per_sec):return output_tokens / tokens_per_sec
Consider a task with 1,000 input tokens. A direct answer requires 300 output tokens, while enabling deep reasoning produces an additional 3,000 output tokens. Output volume increases by about ten times, and cost and decoding time grow roughly ten times as well. If the output price is higher than the input price, the increase becomes even more pronounced.
Therefore, choosing a reasoning model cannot rely only on unit price. It is necessary to look at the average number of output tokens consumed to complete a task. Common control methods include:
- Setting a reasoning budget cap and forcing a conclusion after it is exceeded.
- Routing by question difficulty, using non-reasoning mode for simple questions and enabling deep reasoning only for complex ones.
- Specifying output length requirements in the prompt and validating on an evaluation set whether quality is affected.
6. How to Read a Price Sheet
Model prices are usually expressed per million tokens, with separate prices for input and output. When reading a price sheet, pay attention to the following:
- Input and output are asymmetric. Output tokens must be generated serially one by one, requiring more compute and memory usage, so most services price output higher than input.
- Cache hits may have discounts. For requests with repeated prefixes, some services offer a lower cached input price. Whether a hit occurs depends on whether the request structure is stable.
- Context length may be priced in tiers. Long-context requests often have a higher unit price.
- Unit price is not cost. What really determines spending is the cost per successful task.
Cost per successful task can be estimated as follows:
def cost_per_successful_task(avg_in, avg_out, price_in_per_m, price_out_per_m, calls_per_task, success_rate):per_call = request_cost(avg_in, avg_out, price_in_per_m, price_out_per_m)return per_call * calls_per_task / success_rate
Success rate is critical here. A model with a low unit price but a low success rate may require multiple retries, making the final unit cost higher. Evaluation should use task-level success rate, not the unit price of a single call.
7. Real-World Constraints for Edge and Automotive Deployment
Deploying models to phones or automotive devices faces constraints that are completely different from the cloud:
- Memory budget. The memory available to the model on a device is usually only a few GB, so quantization and pruning are needed to fit it. Even if an MoE model has few active parameters, all expert weights still need to reside in memory, which limits its applicability on edge devices.
- Power and heat. Sustained high-load inference causes chip throttling, and actual throughput falls below benchmark results.
- Deterministic latency. Automotive scenarios have hard requirements for response time, and tail latency (p95, p99) is more important than the average.
- Offline operation and privacy.
