DeepSeek Sparse Architecture: Balancing Inference Cost and Performance

    This article examines why LLM inference costs are difficult to reduce and how sparse architectures address the problem. It compares quantization, knowledge distillation, and speculative sampling, then explains sparse activation through Mixture of Experts routing, including load balancing, communication overhead, and memory usage. It also covers DeepSeek Sparse Attention for long-context computation, its limits, and the distinction between standard and Speciale post-training versions.

    1. Why Inference Costs Are Hard to Reduce

    Large language models improve as parameter scale and training data grow, but the computation required to generate each token also increases. For high-frequency enterprise applications such as customer service, data analysis, and coding assistance, inference costs are often a major long-term operating expense.

    There are three common approaches to reducing cost. Quantization compresses weights from higher precision to lower precision and can save GPU memory, but accuracy loss may appear in some tasks. Knowledge distillation uses a large model to train a smaller one; it has lower cost, but the smaller model has a limited capability ceiling. Speculative sampling first has a small model draft tokens and then a large model verifies them, which can accelerate generation; the benefit depends on the draft acceptance rate. These methods either operate after model training or accelerate the inference pipeline. They cannot structurally reduce the number of parameters involved in computing each token.

    Sparse architectures take a different path. They specify at design time that only part of the parameters are used in each computation, thereby reducing the computation per inference.

    This article is based on publicly available information; specific parameters and performance data were not individually verified during writing. For model versions, interfaces, and performance metrics, refer to DeepSeek’s official documentation and technical reports.

    2. Sparse Activation: Basic Principles of Expert Routing

    A dense model uses almost all parameters when processing each token. A Mixture of Experts (MoE) structure splits the feed-forward network into multiple expert sub-networks, and a router selects a small number of experts for each token. The selected experts participate in computation, while the others do not.

    The router outputs a set of scores for each token, selects the top-k experts with the highest scores, and combines their outputs by weight. The total number of model parameters can be large, but only a small fraction of them are actually activated for each token. This is what “large total parameters, small activated parameters” means.

    There are two common difficulties in routing design. The first is load imbalance. If a few experts are selected for a long time, other experts may not be trained sufficiently and may even remain idle. During training, a load-balancing loss is usually added, and an expert capacity limit is set to distribute tokens more evenly. The second is communication overhead. When experts are distributed across different GPUs, tokens need to be transferred across cards, and interconnection bandwidth during deployment directly affects latency. This point is often overlooked during model selection.

    It is important to note that MoE usually reduces the computation per token, not the weight footprint. All experts still need to be loaded into GPU memory or host memory, so memory requirements may not decrease proportionally with the activation ratio.

    3. Sparse Attention: Reducing Computation for Long Contexts

    Expert routing addresses computation in the feed-forward network. The cost of the attention part grows with context length. The longer the sequence, the more historical tokens each new token must attend to, and the overhead rises significantly.

    The idea behind sparse attention is to let each token attend to only part of the historical positions. In its public technical materials, DeepSeek has proposed a sparse attention scheme called DeepSeek Sparse Attention (DSA). According to public descriptions, it first uses a lightweight indexing module to select several relevant historical positions for the current token, and then performs full attention computation on those positions. The specific indexing method, computational overhead, and accuracy impact should be based on the official technical report.

    Sparse attention also has boundaries. Selecting the wrong historical positions can lose key information, so tasks that require precise comparison across the full text need separate evaluation. The benefit of selective attention is more obvious in long-context scenarios and may not be significant in short-text scenarios.

    Expert routing and sparse attention act on the feed-forward network and context access, respectively, and they solve different problems. They should be evaluated separately during model selection.

    4. Standard and Speciale Versions: Distinction and Verification

    Different post-training versions of the same model generation are often released with different priorities. A version such as Speciale is generally aimed at stronger reasoning tasks; a common approach is to allow the model to generate a longer reasoning process in exchange for higher answer accuracy. This trade-off increases per-request latency and token consumption.