How Ultra-Large Mixture-of-Experts Models Reduce Inference Costs

    Ultra-large language models are increasingly evaluated not only by parameter scale but by how much model capacity can be delivered at a given inference cost. This article explains how mixture-of-experts routing, gated state-space memory, sparse attention, gated residuals, and local-pattern embeddings change information flow to reduce per-token activation and long-context overhead. It also distinguishes total parameters from activated parameters and real serving cost, examines the trade-offs of each technique, and outlines a layered evaluation approach covering quality, latency, memory, throughput, and full-task cost.

    In recent years, competition in ultra-large language models has gradually shifted from simply increasing parameter count to the question of how to invoke larger model capacity at lower inference cost. At roughly the same total parameter scale, some new models use mixture-of-experts architectures, state-space memory, sparse attention, conditional embeddings, and related techniques to reduce the number of parameters actually activated per token and to lower the compute and memory overhead of long-context scenarios.

    Based on publicly available descriptions, the core idea behind these approaches is not simply compressing the model. Rather, it is redesigning how information flows through the network: some historical information is compressed into a fixed-size state, some context is selected through sparse indexing, and local token patterns are supplemented by low-cost embedding structures. Residual paths are also reorganized to improve training stability in deep networks.

    One point needs clarification first: total parameter count, activated parameters per token, actual compute, and final usage cost are not the same concept.

    1. Total Parameters Are Not the Same as Per-Inference Cost

    In a mixture-of-experts model, the model can have a very large total parameter count while routing each input to only a subset of experts. For example, a model may have trillions of total parameters, but a single token may activate only billions or tens of billions of parameters.

    This design has two direct consequences:

    1. The model can contain more experts and cover richer knowledge and task patterns.
    2. The matrix computation required per token does not have to grow in proportion to total parameter count.

    However, lower activated parameter count does not necessarily mean cost falls by the same proportion. Actual inference cost is also affected by:

    • the sequence length used in attention computation;
    • the size of the KV cache and its read/write bandwidth;
    • expert routing and communication overhead;
    • the difference between compute in the prefill and decode phases;
    • memory capacity, memory bandwidth, and parallelism strategy;
    • the quantization format and hardware support for the corresponding operators.

    Therefore, when someone claims that cost has fallen by more than a certain percentage, it is necessary to clarify whether this refers to theoretical FLOPs, per-token compute, the actual number of generated tokens, or the comprehensive cost of a complete request. If a stronger reasoning model reduces the number of output steps, the cost reduction may also come from the fact that the same task requires fewer tokens, not only from lower compute per token.

    2. Gated State Space: Reducing Dependence on Full Attention

    Traditional self-attention requires the current position to interact with a large number of tokens in the historical sequence. As context length grows, both compute and KV cache access pressure increase. The idea behind a gated state-space structure is to gradually compress historical information into a fixed-size state instead of rereading the full history at every position.

    This can be abstracted as:

    1. state_t = Update(state_{t-1}, token_t, gate_t)
    2. output_t = Read(state_t, token_t)

    Here, state_t stores a summary of history up to the current position, while gate_t determines which information is written into the state, which information is retained, and which information is forgotten.

    The advantage of this structure is that state size usually does not grow linearly with context length, making it especially suitable for long text, continuous dialogue, and streaming generation. Compared with full attention, it can reduce the frequency of reading historical information and reduce part of the KV cache pressure.

    But it also has clear limitations: a fixed-size state inevitably compresses information. If a task requires precise reference to a detail from far earlier in the context, relying only on state-space memory may not be enough. A more realistic approach is therefore not to remove attention entirely, but to alternate state-space layers with a small number of sparse attention layers.

    3. Sparse Attention: Retrieve Only Truly Relevant Context

    In an alternating structure, most layers maintain and update the global state at relatively low cost, while a small number of layers perform precise retrieval. Sparse attention typically does not compute over all historical tokens. Instead, a lightweight indexer first screens candidate regions, and then higher-precision attention is performed on those candidate blocks.

    The process can be summarized as follows:

    1. Divide the historical context into contiguous blocks.
    2. Generate a compact representation for each block.
    3. Compute block-level relevance using the current query.
    4. Select a small number of highly relevant blocks.
    5. Perform fine-grained attention only within those blocks.

    The key is not to make the model “remember everything,” but to enable it to find potentially relevant content at low cost when needed. For tasks such as long-document question answering, code repository analysis, and multi-turn dialogue, block-level retrieval is often more economical than full token-by-token attention.

    However, the effectiveness of sparse attention depends heavily on the quality of the indexer. If the candidate screening stage misses critical context, later attention cannot recover the filtered information, no matter how precisely it is computed. When evaluating such models, therefore, it is not enough to look only at average speed. It is also necessary to test long-distance references, cross-paragraph reasoning, and scenarios in which multiple similar entities appear at the same time.

    4. Gated Residuals: Reorganizing Cross-Layer Information Transfer

    Deep models usually rely on residual connections to ensure stable gradient propagation. Traditional residual paths are simple and effective, but in very deep networks and complex sparse structures, they may cause interference between layers or unbalanced update magnitudes.

    The basic idea of a gated residual is to split the residual stream into multiple parallel branches and use gating mechanisms to control reading from and writing to different branches. For example, a single residual stream can be expanded into multiple cooperating channels:

    1. branch_i' = branch_i + write_gate_i * transform_i(input)
    2. output = Σ read_gate_i * branch_i'

    The gating here does not necessarily mean that all branches simultaneously perform full computation. Instead, it allows the model to dynamically decide, based on the current token and current layer state, which path information should enter and which path it should be read from.

    This design may provide three benefits:

    • reducing interference between different functional modules;
    • improving gradient propagation in deep networks;
    • making it easier for the model to learn information representations at different levels.

    It is important to note that the value of gated residuals is reflected mainly in training stability and representational capability, and does not necessarily translate directly into lower inference FLOPs. It is usually an architectural optimization that supports the effective training of larger models, rather than a standalone cost-compression technique.

    5. Local Pattern Embeddings: Supplementing Short-Term Memory at Low Compute Cost

    In addition to the main backbone, some new architectures introduce conditional embedding structures to capture local token combinations, phrase patterns, or common fragments. The approach is similar to forming local patterns from adjacent tokens and then generating additional representations through lookup tables or lightweight transformations.

    The advantages of such structures include:

    • avoiding the need to run a full deep network for every local pattern;
    • quickly capturing high-frequency short-range dependencies;
    • expanding model capacity at relatively low computational cost.

    It is more like adding a “conditional memory” that specializes in local, repetitive, and easily summarized patterns. Compared with a large backbone network, lookup tables or lightweight mappings usually have lower computational overhead, but storage capacity, memory access bandwidth, and pattern generalization still require attention.

    If local pattern embeddings rely too heavily on high-frequency combinations in the training data, the model may perform well on common expressions but gain limited benefit on rare combinations or cross-domain text. Such modules therefore need to work together with backbone representation learning and cannot be treated as a replacement for general reasoning capability.

    6. Why the Same Task May Consume Fewer Tokens

    Lower model cost may also come from improved output efficiency. A more capable model may complete the same task with a shorter chain of thought, fewer repeated explanations, or more accurate tool calls. This reduces the number of generated tokens, but it is a different issue from reducing the per-token compute cost.

    A more complete cost model can be expressed as:

    1. total cost =
    2. input_tokens × per_input_token_cost
    3. + output_tokens × per_output_token_cost
    4. + KV cache and communication overhead
    5. + scheduling, storage, and hardware resource cost

    When evaluating a model, at minimum record the following separately:

    • input length;
    • output length;
    • time to first token;
    • subsequent token generation speed;
    • peak memory usage;
    • performance degradation under long context;
    • accuracy and retry count for the same task.

    Cost data is comparable only when task quality is roughly equivalent. If a model gives shorter answers but omits key information, or requires more human review, then apparent token savings may not translate into real business value.

    7. Compared with More Aggressive Information-Flow Restructuring

    Current technical routes differ mainly in what they prioritize.

    One class of approaches emphasizes expanding total capacity while reducing activated parameters per token. Typical methods include combinations of mixture-of-experts, gated state space, sparse attention, and conditional embeddings. This direction focuses more on balancing model capacity, inference efficiency, and training stability.

    Another class goes further by distinguishing input processing from output generation paths, redesigning the organization of the KV cache, and even attempting to reuse some key-value information across layers. This route compresses memory access more aggressively for long context and decoding, but implementation complexity, hardware adaptation difficulty, and training stability may also be higher.

    Neither direction is inherently superior. The former is often more likely to achieve balanced performance on general tasks, while the latter may deliver greater value in ultra-long context, extremely low latency, or high-concurrency serving. The final result still depends on training data, routing strategy, operator implementation, and deployment hardware.

    8. How to Evaluate the Model in Practice

    To determine whether a new model is truly “cheaper,” it is better to build a layered test suite than to rely on a few subjective conversations.

    The first layer is basic capability testing, including text understanding, structured output, code generation, and information extraction. The focus should be on accuracy and output length.

    The second layer is long-context testing, including cross-paragraph references, long-distance entity association, long-document summarization, and multi-turn information tracking. The key question is whether performance remains stable as context length increases.

    The third layer is efficiency testing, recording time to first token, sustained generation speed, peak memory usage, and concurrent throughput.

    The fourth layer is cost testing, comparing resource consumption per request under the same hardware, the same quantization configuration, and the same input and output requirements.

    The fifth layer is reliability testing, observing whether the model is prone to repetition, omission, hallucination, or frequent retries on complex tasks.

    Overall, the value of a new generation of ultra-large models should not be reduced simply to “more parameters” or “fewer activated parameters.” What matters is whether, through more reasonable information-flow design, the model can turn large-model capacity into engineering capability that is deployable, scalable, and sustainable to operate. Only by examining quality, latency, memory footprint, throughput, and full-task cost together can one form an objective judgment about so-called cost reduction.