How to Rationally Evaluate Trillion-Parameter MoE Multimodal Large Models

    This article takes a measured, engineering-oriented look at a class of trillion-parameter sparse MoE multimodal models—defined here by trillion-scale total parameters and hundred-billion-scale activated parameters—rather than any single product. It walks through six upfront judgments, then explains how to separate the compute account (activated parameters) from the memory account (total parameters), how attention is optimized for long context through latent attention and lower-bounded-decay linear attention, how MoE routing stays stable with latent-space experts and quantile-based load balancing, how multi-stage context extension and agent-oriented RL post-training shape real usability, and how to estimate training and serving cost. It closes with common misconceptions and a practical view: judge these systems by per-token cost, effective context length, and task accuracy rather than headline parameter counts.

    Note: This article discusses a class of technical approaches rather than any specific model. It uses “trillion-level total parameters, hundred-billion-level activated parameters” as the typical scale for this class of models; refer to the official public materials for specific values. All code is pseudocode, provided only to illustrate the computational logic.

    1. Conclusions First: Six Judgments

    1. Sparsification is the main line, not a party trick. The essence of sparse MoE is “conditional computation”: knowledge capacity is placed in a vast pool of parameters, while the compute cost of a single forward pass is confined to a small number of activated parameters. This decouples model capacity from inference cost, so the two no longer grow at the same rate.

    2. Attention is the main battlefield for efficiency. Once the context is stretched, attention imposes two costs at the same time: the quadratic compute of the attention matrix, and the KV cache that grows linearly with length. Rework such as low-rank compression, gating, and linearized decay all aim to “lighten the load” for long context.

    3. Cross-layer information reuse is coming back. Residual connections solved gradient backpropagation across layers, but deep representations can still be diluted as they pass through layer-by-layer transformation. A chunked residual based on attention outputs is a low-overhead approach to cross-layer fusion.

    4. Training stability depends on structural design. Routing balance, numerical precision, and expert granularity often matter more than the compute budget in determining whether a pretraining run converges smoothly.

    5. Post-training decides “usability.” With the same base capability, reinforcement learning oriented toward agentic tasks can open a clear gap in multi-step planning, tool calling, and long-horizon reasoning.

    6. Million-token context should be judged by “effective length,” not nominal length. For developers, the more meaningful questions are: What is the accuracy on long-context tasks? What is the time-to-first-token? How much memory does the KV cache occupy?

    2. Scale and Sparsity: Keep the Two Accounts Separate

    To evaluate an ultra-sparse MoE, you first need to separate two accounts.

    • The compute account is about activated parameters. In a standard Transformer, each token’s per-layer linear computation is roughly 2× the backbone forward pass, and backpropagation doubles it again, so the per-token training compute is about 6N, where N is the number of parameters involved in the computation. The attention portion grows with the square of the sequence length, and MoE adds routing and communication overhead on top.
    • The memory account is about total parameters. Using BF16 storage as an example, trillion-level total parameters correspond to about 2 TB of weights, while hundred-billion-level activated parameters correspond to about 200 GB. The activated portion governs compute; the total portion determines memory capacity and access pressure.

    The rule of thumb behind this design: a model’s factual knowledge capacity correlates strongly with total parameters, while the arithmetic intensity at inference time is governed more by activated parameters. Sparsification is like “putting books in a library and taking out only a few each time”—it expands knowledge capacity while keeping computation in check.

    But sparsity is not free:

    • The more experts there are, the greater the risk of routing noise and load imbalance; some experts may be under-trained, becoming parameters that “exist in name but are effectively inactive.”
    • In distributed inference, tokens must be sent across devices to their assigned experts, and communication overhead rises sharply.
    • During autoregressive decoding, each token activates only a few experts, so the matmul sizes are small and per-request hardware utilization is often low.

    Therefore, evaluating this class of models means looking at both total and activated parameters, as well as the number of experts, top-k routing, and shared-expert design. Looking at either “total parameters” or “activated parameters” alone tends to yield a one-sided conclusion.

    3. Attention: Lightening the Load for Long Context

    3.1 Three Kinds of Pressure

    1. Compute pressure: Standard attention computes a T×T attention matrix, and the cost grows quadratically with context length.
    2. Cache pressure: During decoding, every token’s key-value pairs must be cached. The cache size is roughly 2 × L × H_kv × d × B × S × b, where L is the number of layers, H_kv is the number of KV heads, d is the head dimension, B is the batch size, S is the sequence length, and b is the number of bytes per element. Double the context and the cache doubles too.
    3. Access pressure: During decoding, generating each token requires reading the entire KV cache. The longer the context, the more likely that a single token’s latency is bounded by memory bandwidth rather than compute.

    3.2 Latent Attention: Compressing the KV Cache into a Low-Dimensional Space

    The idea behind latent attention (MLA-style designs) is: first project the multi-head keys and values into a low-dimensional latent vector much smaller than the total head dimension, cache only the latent vector during inference, and up-project it back into per-head keys and values when needed; gating is also introduced to control how strongly context information is injected at each position. The intuitive benefits are:

    • The cache can shrink by several times or even an order of magnitude, with the exact factor depending on the ratio of the latent vector dimension to the total multi-head dimension;
    • The latent space is a controlled compression—too high a compression ratio causes information loss, so both the gating and the latent dimension need to be weighed in ablation experiments;
    • If the compressed representation is incompatible with positional encoding, a dedicated positional handling scheme must be designed, which is one of the main sources of implementation complexity for these structures.

    3.3 Linear Attention with Lower-Bounded Decay

    Linear attention (including gated Delta-rule variants) replaces the T×T matrix with a recurrent state: each token only updates a fixed-size state S, reducing complexity from quadratic to linear, and requiring no KV cache that grows with length during decoding.

    In practice, the first T tokens of causal attention are visible to every output. Linear attention approximates this by compressing history into a fixed-size state and then performing a single query. It trades a controllable loss of expressiveness for efficiency, so a common compromise is hierarchical mixing: some layers use standard attention for “precise retrieval,” while others use linear attention for “efficient memory.”

    Real deployments also face an often-overlooked numerical problem. When computing linear attention in tiles, the relative weights within a block are determined by the ratio of cumulative decay factors; when the log decay has no lower bound, this ratio can exceed the representable range of low-precision floating point and overflow or underflow. A common workaround is to write a separate per-position path for the “diagonal block,” which leaves two paths coexisting within the same layer—fast and slow. As a result, the matmul units cannot run at full load, and the operators are harder to optimize.

    A more thorough solution is lower-bounded decay parameterization: replace the unbounded softplus with “fixed lower bound × Sigmoid,” constraining the log decay to the interval [g_min, 0), where g_min is a negative constant (such as -5). The meaning of g_min is: for every token the information advances, the weight can decay by at most a factor of e^(-5) (about 0.0067). It serves two purposes at once:

    • It ensures the decay “does not vanish,” so that the decay ratio between any two positions within a block is bounded jointly by the block length and g_min;
    • It also ensures the decay is “fast enough,” so that historical information does not accumulate indiscriminately.

    Illustration of the parameterization:

    1. Original: log_decay = -softplus(a * z + b) # range (-∞, 0), lower bound not fixed
    2. New: log_decay = g_min * sigmoid(a * z + b) # range (g_min, 0), g_min = -5

    Both forms include 0 in their output range, but in the original the value 0 can only be approached asymptotically, whereas the new form has a closed value interval.

    Core code illustration:

    1. # Linear attention with lower-bounded log decay (pseudocode)
    2. def linear_attn(x, Wq, Wk, Wv, gate, g_min=-5.0, tile=16):
    3. q, k, v = Wq(x), Wk(x), Wv(x)
    4. # Per-position decay factor lies in (exp(g_min), 1)
    5. log_g = g_min * sigmoid(gate(x))
    6. S, outs = 0, []
    7. for start in range(0, len(x), tile):
    8. end = min(start + tile, len(x))
    9. log_gs = log_g[start:end] # In-block log decay
    10. # Cumulative decay log(G_i / G_j) between i and j (i >= j)
    11. D = rel_decay(log_gs)
    12. # Diagonal blocks also use a single dense matmul: exp(D) neither overflows nor underflows
    13. intra = exp(D) * (q[start:end] @ k[start:end].T)
    14. # Off-diagonal blocks: reuse the compressed state S from the end of the previous block
    15. inter = (q[start:end] @ S) * exp(rel_decay_to_start(log_gs))
    16. outs.append((intra + inter) @ v[start:end])
    17. # Cross-block state update: intra-block positions weighted by decay, inter-block inherits state
    18. G_end = exp(sum(log_gs))
    19. S = G_end * S + (k[start:end] * exp(cumsum(log_gs))[:, None]).T @ v[start:end]
    20. return concat(outs)

    The benefits appear in three areas:

    • Diagonal and off-diagonal blocks both go through dense matmul, removing the special branch, making the computation graph more regular and easier for low-level libraries to fuse, schedule, and quantize;
    • The throughput gain comes from a shorter execution path, not higher parallelism, so it can usually be adopted without changing the model’s main structure;
    • The numerical range is stable, making it more reliable in both training and long-sequence inference.

    It should be noted that the decay lower bound is essentially an inductive bias: the upper bound reflects how long the model “can remember,” while the lower bound reflects “how much it forgets at minimum.” g_min should be treated as a hyperparameter, determined by task type through ablation experiments, and should not be copied wholesale.

    3.4 Block-Level Attention Residuals: Cross-Layer Information Reuse

    In a deep Transformer, a key piece of information from a layer-1 attention output may have to pass through dozens of layers of transformation before it influences the final layer, and it can be diluted or overwritten along the way. The block-level attention residual can be summarized as “store, aggregate, inject”:

    1. After each attention block is computed, its output is not added directly to the backbone but is temporarily stored by “block”;
    2. When a preset position is reached, an aggregation operator (such as a learned-weight weighted sum) merges the stored per-block outputs;
    3. The merged result is injected into the backbone all at once.

    Illustrative code:

    1. # Block-level attention residual (pseudocode)
    2. class BlockAttnRes:
    3. def __init__(self, num_blocks, num_layers, dim):
    4. self.num_blocks = num_blocks
    5. self.blocks = [] # block id -> attention output of that block, shaped [L_block, T, dim]
    6. self.register(self._agg, shape=(num_blocks, num_layers, dim))
    7. def forward(self, x, block_id, attn_out):
    8. if (block_id + 1) % self.num_blocks == 0:
    9. self.blocks.append(attn_out)
    10. if len(self.blocks) == self.num_blocks:
    11. # Weighted aggregation over "block × layer × dim", then injected into the backbone at once
    12. x = x + einsum("bltd,blh->thd", stack(self.blocks), self._agg)
    13. self.blocks.clear()
    14. return x

    Compared with per-layer residuals, the value of this kind of scheme is that shallow-layer information participates in deep computation in a more direct way; the storage and aggregation are concentrated in a few positions, keeping memory and communication overhead controllable.

    4. MoE: Routing, Latent Space, and Balance

    Attention answers “how far to look,” while MoE answers “how much to hold.” The MoE design of ultra-sparse models usually revolves around three questions.

    4.1 Latent-Space Experts: Small Parameters Supporting High Sparsity

    When the number of experts grows to hundreds or even thousands, if each expert is a full hidden-dimension feed-forward network, total parameters balloon quickly and expert-parallel communication volume rises accordingly. The low-rank latent-space expert works as follows:

    • A shared input down-projection matrix W_down reduces the hidden vector x to dimension d_c;
    • Each expert computes only in the d_c-dimensional space;
    • A shared up-projection matrix W_up maps the result back to the hidden dimension.

    Change in parameters and compute (per expert):

    Scheme Parameters Compute (MACs)
    Standard expert 2×d×d_e 2×d×d_e
    Latent-space expert d×d_c + 2×d_c×d_e + d_c×d 2×d_c×d_e + 2×d_c×d

    When d_e is on the same order as d and d_c is much smaller than d, the per-expert cost shrinks roughly in proportion to d_c/d. This means: with the activated-parameter budget unchanged, the model can accommodate more experts, and sparsity and knowledge capacity rise accordingly. Sharing the down-projection and up-projection across all experts is the key to making this scheme work.

    The cost is equally clear: all experts’ information must squeeze through the same d_c-dimensional bottleneck, so the model’s expressiveness is, to some degree, limited by the capacity of the latent space. The down-projection dimension should not be set too small, or it easily becomes a performance bottleneck.

    4.2 Why MoE Needs Load Balancing

    The MoE router is “winner-takes-all.” Early in training, if a few experts happen to perform better, they get selected by more tokens, are trained more fully, and in turn attract even more tokens, forming a positive feedback loop. Load imbalance triggers a chain of consequences:

    • Compute waste: Under expert-parallel deployment, each step is gated by the most loaded device, and the compute of idle devices is wasted;
    • Model degradation: Unpopular experts receive too few gradients, so the effective capacity is below the nominal capacity;
    • Training instability: Once routing decisions become concentrated, expert specialization tends toward homogenization.

    4.3 Quantile Balancing: Stabilizing Without an Auxiliary Loss

    The traditional approach adds a load-balancing auxiliary loss that penalizes the variance of per-expert load, but it has two problems: the extra loss term competes with the main task objective for gradients, and its weight needs repeated tuning; the balance point between load balancing and model quality also relies on experience.

    Quantile balancing (a QB-style approach) instead “directly calibrates the routing bias rather than changing the loss function”:

    1. Count the number of tokens each expert actually processes within a sliding window;
    2. Take a target quantile within the window as the reference threshold;
    3. For experts above the threshold, lower the routing bias; for those below, raise it;
    4. The bias only affects “which expert is chosen,” does not enter the loss, and can be zeroed at inference time.

    Illustrative code:

    1. # Quantile-based routing bias update (pseudocode)
    2. def update_bias(load, bias, q=0.9, step=1e-3, beta=0.9):
    3. # load: number of tokens actually processed by each expert within the sliding window
    4. thr = quantile(load, q) # Use the quantile as the balancing reference
    5. delta = where(load < thr, 1.0, -1.0) # Underloaded -> up, overloaded -> down
    6. return clamp(beta * bias + (1 - beta) * step * delta, -1e2, 1e2)
    7. # During training: bias only participates in routing selection
    8. gate = softmax(router(x) + bias)

    The difference between the quantile method and the mean method lies in the reference standard. Using the median (q=0.5), the update target is roughly “let as many experts as possible not exceed the median,” but with many experts it is mathematically impossible for the majority to sit below the median at the same time. Choosing a higher quantile (such as q=0.9) acts as a “floor”: it mainly supports the least-loaded tail experts to ease starvation of unpopular experts, rather than forcibly flattening all differences. This is closer to “filling gaps on demand” than “cutting peaks to fill valleys.”

    Selection guidance:

    Aspect Auxiliary-loss balancing Quantile balancing
    Implementation change Loss term, gradient scaling, weight tuning Only maintain routing bias
    Coupling with main task Gradient competition exists No gradients, only affects routing
    Debugging cost Weight-sensitive, needs multiple experiments Few parameters, mainly q and update step
    Load uniformity Usually more uniform Focuses on tail support, allows some variance
    Inference deployment Generally no special handling Bias must be explicitly zeroed to prevent leftover config

    Whichever scheme is used, it should be paired with monitoring of each expert’s token share. Any expert whose share persistently deviates from the mean is a potential risk point.

    5. Multi-Stage Training: Context and Post-Training

    5.1 Curriculum-Style Context Extension

    A context window is hard to expand all at once, and the industry commonly uses curriculum-style training: first train the model’s language ability and basic reasoning on short contexts, then extend stage by stage to the 64K, 256K, and even 1M token level. Each stage needs to handle four things carefully:

    • Data mixture: Gradually raise the proportion of long samples such as long documents, repository-level code, and multi-turn dialogues, to avoid long-context ability that is “real in name only”;
    • Positional encoding adaptation: Adjust parameters such as the base frequency of rotary positional encoding so that the position representation fits the new length range;
    • Efficient structures: As sequences grow longer, computation gradually tilts toward linear attention and low-rank caches to control training cost;
    • Batch size and learning rate: A single step with long sequences processes many tokens, so the parallelism strategy (such as context parallelism) and optimizer hyperparameters need to be adjusted accordingly.

    Long-context ability relies especially on data engineering. Common techniques include:

    • Constructing “cross-paragraph retrieval” tasks from real long documents, such as cross-chapter question answering and evidence localization;
    • Concatenating same-repository code in file-dependency order and generating tasks that require cross-file reasoning;
    • Controlling where key information is inserted to alleviate the positional bias of “middle content being ignored.”

    5.2 Agent-Oriented RL Post-Training

    Pretraining teaches a model “knowledge and expression,” while post-training decides whether it can “get things done.” The typical agent work loop is “observe, think, call a tool, read the result, and decide again.” This loop can span dozens of steps, which places special demands on post-training:

    1. Long multi-turn trajectories: A single trajectory may contain dozens of tool calls, with a cumulative context far exceeding an ordinary dialogue, so the training framework needs to support ultra-long sequences and efficient sampling;
    2. Sparse rewards: Task success or failure often gives a signal only at the final step, so techniques such as advantage normalization and process rewards are needed to ease the credit-assignment problem;
    3. Verifiable rewards: Deterministic feedback such as code execution, unit tests, and data validation is more reliable and harder to “game”;
    4. Training-inference consistency: The inference implementation used during sampling should stay numerically consistent with the training implementation, or importance sampling will be distorted and could even cause training to collapse.

    6. From Paper to Production: Cost Estimation and Engineering

    6.1 Do the Math

    Memory estimation:

    1. Per-GPU weight memory ≈ total params × bytes per param / tensor parallel degree
    2. Per-GPU KV memory ≈ 2 × layers × cache elements per token × seq length × concurrency × bytes / context parallel degree

    Taking 576 cache elements per token, 64 layers, and BF16 storage as an example:

    • A single 1M-token request needs about 64 GB of cache, which a single GPU cannot hold, so context parallelism or cache offloading is required;
    • When the context expands to 8M tokens, the cache alone needs about 512 GB—equivalent to the full memory of eight 80 GB GPUs.

    These figures show that context length amplifies memory demand linearly, making cache compression and context parallelism nearly mandatory.

    Throughput and latency estimation:

    1. Per-GPU decode throughput ≈ (memory bandwidth × num GPUs × utilization) / bytes read per token
    2. Bytes read per token ≈ activated params per token × bytes + KV bytes per token

    Consider two typical scenarios:

    • Short context: With 100B activated parameters, reading one token takes about 200 GB (BF16); with 16 GPUs at a total bandwidth of 12 TB/s and 50% utilization, the theoretical throughput is about 30 token/s;
    • Million-level context: The KV portion is about 0.13 GB, only about 2 GB more than the activated weights, so throughput drops only slightly.

    This is exactly where low-rank compression schemes such as latent attention add value: they bring the decode cost of long context closer to that of short context, whereas standard multi-head attention in million-level contexts can have a per-token KV read of hundreds of GB or even TB scale.

    Training cost estimation:

    1. Training compute ≈ 6 × activated params × training tokens × (1 + attention and communication overhead factor)

    With 100 billion activated parameters and 15 trillion training tokens, the main term is about 9×10^24 floating-point operations. Assuming 1024 accelerator cards carry this, achieving about 90% compute utilization requires each card to reach roughly 600 TFLOPS of effective compute. This estimate ignores the overhead that grows with sequence length in attention and the loss from fault recomputation, so the actual value is usually higher. But it is enough to make one point: training ultra-sparse models places extremely high demands on “large-scale parallelism plus high effective compute,” and the cluster size and parallelism strategy are themselves part of the model design.

    6.2 Deployment and Cost Optimization

    • Separate prefill and decode: The prefill stage is compute-intensive, while the decode stage is memory-access-intensive. Deploying them in separate pools lets you configure parallelism strategies and batching parameters independently, improving resource utilization;
    • Increase the decode batch size: MoE decode compute utilization depends heavily on the number of tokens in a batch. Too small a batch leaves the expert matmuls “underfed”; with a large enough batch, the experts selected by different requests complement one another and hardware utilization rises noticeably;
    • Prefix cache reuse: Fixed prefixes such as system prompts, tool descriptions, and code repository context can be cached to avoid repeated prefilling. This is also one of the key cost-reduction levers in agentic scenarios;
    • Cache and weight quantization: Quantizing the KV cache and weights can significantly reduce memory and bandwidth usage, but evaluation is needed to confirm the accuracy loss is acceptable;
    • Observability: Metrics such as expert load, cache hit rate, time-to-first-token, and per-token output latency should be brought into monitoring and alerting so anomalies can be caught early.

    When deploying on the cloud, you can combine the following general capabilities to reduce the engineering cost of going from experiment to production:

    • Elastic compute resource pools to handle the different load profiles of training and inference;
    • A container platform for deploying inference services;
    • Object storage for saving model weights and training data;
    • Logging and monitoring/alerting to observe service quality and resource levels;
    • Model serving and development platforms for model hosting, fine-tuning, and application orchestration.

    7. Common Misconceptions and Risk Warnings

    1. “Bigger total parameters means stronger.” Knowledge capacity does correlate with total parameters, but capability gains also depend on data quality, the number of training tokens, and post-training methods. Activated parameters are the main determinant of inference cost.
    2. “Longer context is always better.” Nominal length is not effective length. Evaluation should cover cross-paragraph retrieval, long-document reasoning, and repository-level code understanding, and should observe behavior when information sits in the middle.
    3. “Linear attention can fully replace standard attention.” Linear attention loses expressiveness on precise-retrieval tasks and in practice often needs to be mixed with standard attention.
    4. “MoE is always cheaper.” In small-batch, low-concurrency scenarios, MoE’s memory-access and communication overhead can offset the benefits of sparsity. Whether it pays off should be judged by the actual throughput-latency curve of your business.
    5. Drawing conclusions from public benchmarks alone. Benchmark results are easily influenced by training data. A more reliable approach is to use a private evaluation set close to your own business, examining stability, reproducibility, and inference cost.
    6. Ignoring compliance and licensing. Open-source model licenses may carry commercial-use conditions or usage restrictions, so a compliance review should be completed before launch.

    8. Conclusion

    Back to the original question: how should we view trillion-parameter-scale MoE multimodal models? A more reasonable perspective is to treat them as a systems approach that “trades engineering constraints for a higher capability ceiling”:

    • Sparsification redistributes capacity and compute;
    • Latent attention and lower-bounded-decay linear attention cut the compute and access cost of long context;
    • Latent-space experts and quantile balancing support ultra-large-scale routing;
    • Block-level attention residuals improve information flow across deep layers;
    • Multi-stage training and agentic RL turn base capability into usability.

    For developers, focusing on parameter scale matters less than focusing on per-token cost, effective context length, and task accuracy. For technical teams, many of these structural designs have strong reusability. For example, the lower-bounded-decay linear attention rework and quantile-based routing balance both involve small changes with clear benefits, and are worth trying in your own training and inference pipelines. Looking at them rationally, evaluating them quantitatively, and selecting on demand is the right way to approach this class of models.