Trillion-Parameter LLM Architecture Explained: Sparse MoE, Hybrid Linear Attention, and Native Multimodality

    This article dissects the architecture behind the latest generation of trillion-parameter large language models, going beyond a single product launch to trace the underlying engineering threads. It explains how sparsity decouples total parameters from active parameters, why the KV cache—rather than parameter count—is the true bottleneck for long context, how hybrid linear attention trades a fixed-size state for cache growth, what attention residuals contribute across layers, and why flagship models are moving toward native multimodal design. It closes with a practical, multi-dimensional framework for evaluating these models and concrete recommendations for selecting and deploying them in production.

    Every time a new flagship large model ships, developer attention tends to cluster around three questions: did the parameter count double again, is there a new attention mechanism, and is long context actually usable? A recent flagship model from China’s frontier labs—referred to here as the “new-generation trillion-parameter model”—arrived with a combination of 2.8 trillion total parameters, hybrid linear attention, attention residuals, and native visual understanding. Its demo performance sparked debate about the overall capability ceiling of China’s large-model landscape.

    This article isn’t limited to that one release. Instead, it systematically lays out the technical threads behind this generation of flagship models: what actually supports trillion-parameter scale, where the cost of long context really sits, how hybrid architectures make their tradeoffs, why multimodality is moving toward “native,” and how enterprise teams should evaluate, select, and deploy these models.

    1. The Parameter Race: From Brute-Force Scale to Sparsity

    The impressive parameter count of the new-generation trillion-parameter model is only the surface. The real barrier is sparsification. These models generally adopt a design that decouples total parameters from active parameters. A simplified illustration makes the idea clear (using a routing-agnostic upper bound as an example; actual figures depend on the specific configuration):

    1. total params ≈ attention & shared-layer params + num_experts × params_per_expert
    2. active params ≈ attention & shared-layer params + routing_k × params_per_expert

    The two can differ by more than an order of magnitude, which creates a division of labor: knowledge capacity is measured by total parameters, while per-inference compute is measured by active parameters. This has four direct consequences:

    • Knowledge capacity is decoupled from inference cost: routing to different experts gives the model a large set of “optional knowledge bases,” but a single forward pass activates only a small fraction. Parameter count can keep growing while per-token compute stays roughly controlled.
    • The main cost variable shifts to memory capacity and bandwidth: the total weight footprint determines memory usage, and generating each token requires reading the relevant weights, which makes bandwidth the bottleneck. When deploying trillion-scale sparse models, memory capacity, inter-GPU interconnect bandwidth, and batching strategy often matter more than peak compute.
    • Training becomes significantly harder: routing balance, expert load distribution, and communication overhead all need dedicated design. Common training objectives (such as load-balancing loss) and initialization strategies are essentially trying to strike a balance between “letting experts do their own jobs” and “not overloading a few experts.”
    • “More parameters” does not necessarily mean “stronger intelligence”: improvements on tasks like long-chain reasoning and code generation within the same model generation come largely from the training recipe and post-training stages (supervised fine-tuning, preference alignment, long-chain reasoning reinforcement learning). Judging a model by parameter count alone easily misjudges its real capability.

    2. The Real Bottleneck for Long Context: KV Cache, Not Parameters

    When long context comes up, attention is usually the first thing mentioned, but what actually drives inference cost is often the KV cache.

    2.1 The Dual Overhead of Standard Attention

    Standard self-attention computes the dot product of the query Q with all keys K, then does a weighted sum, for a compute cost of O(n²). During autoregressive decoding, every step must read all historical keys and values. A simplified estimate shows the scale (with typical configurations for layers, heads, and head dimension):

    1. KV bytes per token per layer = 2 (K and V) × num_heads × head_dim × bytes_per_element
    2. = 2 × 64 × 128 × 2 = 32768 B ≈ 32 KiB
    3. Assuming 32 layers, BF16 precision, and no GQA/MQA compression:
    4. 32 KiB × 32 layers = 1 MiB / token
    5. A single 128K context ≈ 128 GiB
    6. A single 1M context ≈ 1000 GiB

    This exceeds the total memory of most single machines, and it doesn’t yet account for the stacking of multiple concurrent requests. The longer the context, the more the KV cache shifts from a “secondary overhead” to a “primary cost”—which is exactly why long-context serving has been slow to become mainstream.

    2.2 Compressing from Both the Algorithm and Engineering Ends

    On the algorithm side:

    • Grouped-query attention / multi-query attention (GQA/MQA): multiple query heads share a group of key-value heads, directly reducing cache size.
    • Linear attention and its variants: rewrite attention as “state accumulation,” turning a cache that grows linearly with length into one of fixed size.
    • Local windows and sparse attention: keep only neighboring or fixed-pattern attention connections.

    On the engineering side (at the inference-engine level):

    • Paged memory management: allocate the KV cache in blocks to reduce fragmentation and support dynamic growth.
    • Prefix cache reuse: compute identical prefixes—system prompts, long documents—only once and share them across requests.
    • KV cache quantization: compress the cache to INT8 or lower precision.
    • Sliding-window attention: stop retaining keys and values for history beyond the window.
    • Speculative decoding: use a small model to draft and a large model to verify in batches, reducing the overhead from decode steps.

    Algorithm and engineering are not substitutes: the algorithm changes how complexity grows, while engineering determines how many requests can run concurrently and how long a context can be supported on the same hardware. Combining the two is the complete answer to making long context practical.

    3. Hybrid Linear Attention: Trading “State” for “Cache”

    3.1 The Mathematical Essence of Linear Attention

    Starting from a slight rewrite of softmax attention:

    1. SoftmaxAttention(Q, K, V) = softmax(Q Kᵀ / √d) V
    2. LinearAttention(Q, K, V) = φ(Q) (φ(K)ᵀ V) / (φ(Q) φ(K)ᵀ 1)

    φ(·) is usually an element-wise activation function. The key is to use the associativity of matrix multiplication to reorder the computation: first compute φ(K)ᵀV—an “accumulated state” independent of sequence length—and then multiply it by each φ(Q). This drops the compute from O(n²d) to O(nd²), and at inference time you only need to maintain a fixed-size state matrix. The cost is losing softmax’s ability to “automatically screen out” irrelevant content.

    3.2 From Simple Summation to “Gating + Forgetting”

    Early linear attention’s simple state summation let old information accumulate indefinitely and interfere with itself. Later research introduced two kinds of improvements:

    • Gating: decide per channel whether to retain or decay information;
    • Decay-based recursive updates: linear attention with a forget gate can be written as a recurrence:
    1. S_t = A_t ⊙ S_{t-1} + β_t · v_t k_tᵀ
    2. o_t = q_t S_t

    A_t is a per-channel decay factor, and β_t controls write strength. Intuitively, important information is written in and irrelevant information gradually fades out, giving attention a form of “selective memory.” The new-generation trillion-parameter model uses a KDA-style linear attention built on delta-rule updates, which belongs to this lineage. State-space model (SSM) architectures follow the same “fixed state, linear recurrence” route for long-sequence modeling; the two are essentially equivalent in computational structure.

    3.3 Hybrid Architecture: A Division of Labor Between Global and Local

    Pure linear attention has two shortcomings: it performs poorly on tasks that require exact matching and verbatim retrieval, and its matrix form isn’t necessarily faster than highly optimized softmax kernels on short contexts. The common industry solution is to go hybrid:

    • Keep standard attention in a few layers (such as feed-forward layers, or every few layers) to handle global, precise retrieval;
    • Use linear attention in the remaining layers to handle the compression and propagation of long-range information.

    This way, the KV cache from global attention is tied to only a few layers, so the overall long-context cache drops substantially. If the officially cited estimate of about 1 MiB/token cache is roughly on par with a standard-attention model of equivalent configuration, that implies the proportion of global attention layers isn’t low—which is exactly the price paid for “global precise retrieval.” This is also the basis for the cost and selection discussion later.

    3.4 Training Side: Chunk-Wise Parallelism Is the Key Engineering Prerequisite

    The natural recursive form of linear attention is unfriendly to GPU parallelism. For this reason, training implementations commonly use chunk-wise parallelism: split the sequence into blocks, compute in parallel via matrix multiplication within each block, and pass and update state sequentially across blocks.

    1. # Illustrative pseudocode: chunk-wise parallel linear attention forward pass (simplified, no decay)
    2. def chunkwise_linear_attention(q, k, v, block_size=64):
    3. """
    4. q, k: [B, H, T, D] queries and keys (after feature mapping)
    5. v: [B, H, T, D] values
    6. returns: [B, H, T, D] output sequence; state memory is O(D²), independent of T
    7. """
    8. B, H, T, D = q.shape
    9. state = q.new_zeros(B, H, D, D) # cumulative state passed across blocks
    10. out = []
    11. for start in range(0, T, block_size): # sequential across blocks
    12. qb, kb, vb = q[:, :, start:start+block_size], \
    13. k[:, :, start:start+block_size], \
    14. v[:, :, start:start+block_size]
    15. intra = qb @ kb.transpose(-1, -2) @ vb # intra-block attention, parallelizable
    16. mask = torch.tril(torch.ones(qb.shape[-2], kb.shape[-2], device=q.device, dtype=torch.bool))
    17. inter = qb @ state # contribution from historical state
    18. out.append(intra * mask + inter)
    19. state = state + kb.transpose(-1, -2) @ vb # update state
    20. return torch.cat(out, dim=-2)

    Here the across-block loop length is only “sequence length ÷ block size,” so training throughput and memory usage are far friendlier than token-by-token recurrence. Combined with CUDA graphs, custom kernels, and operator fusion, training ultra-long context on trillion-scale models finally becomes feasible. Teams building their own inference services can borrow from this too: in scenarios like long-document summarization and contract review, evaluating layer by layer whether to introduce linear attention or state-space modules is often more cost-effective than simply scaling up hardware.

    4. Attention Residuals: Giving Information a Shortcut Across Layers

    Public material offers limited implementation detail on attention residuals. Judging by the name and technical lineage, the idea is close to “introducing cross-layer/cross-block residual connections into the attention computation.” Common forms include:

    • Using attention to compute a weighted aggregation of the outputs of earlier layers, then feeding it as input to the current layer;
    • Adding a learnable residual gate between the attention output and the main path;
    • Passing some activations across blocks to relieve the information bottleneck in deep networks.

    You can think of it as adding a “cross-layer express lane” for attention on top of the main residual connection. Ordinary residuals accumulate layer outputs at a fixed ratio, whereas attention residuals let the weights be dynamically assigned based on content. There are three main benefits:

    • Improved information flow in deep layers: gradients can back-propagate along multiple paths, making deep training more stable;
    • Stronger inter-layer collaboration: shallow local features combine more easily with deep global representations;
    • Mitigating “representation collapse”: when deep-layer outputs become too similar, dynamic aggregation helps preserve inter-layer differences.

    Three things to watch when implementing it: the initialization of residual weights (borrowing from the LayerScale idea—initialize to small values close to zero so training starts near an identity mapping); memory and bandwidth overhead; and how it works with normalization layers and the routing mechanism.

    5. Native Multimodality: From “Bolted-On Vision” to “Unified Modeling”

    “Native support for visual understanding” deserves its own discussion. Two common routes exist for multimodal models in the industry:

    • Post-hoc concatenation: train a text LLM first, then attach a visual encoder and align it through a projection layer or adapter, with training focused on alignment and instruction tuning;
    • Native: vision and text are jointly modeled within the same pretraining framework, even sharing tokenization and sequence representations.

    Native design is favored by flagship models for three reasons:

    • Earlier, more thorough alignment: image-text data participates in pretraining, so visual semantics and language ability are shaped together;
    • Unified handling of mixed image-text content: interleaved content in web pages, documents, and presentations can be fed in as a unified sequence;
    • Better support for downstream vision tasks: document parsing, chart understanding, and interface understanding can all be done within the same model.

    The challenges are equally clear: visual tokens are often numerous (a high-resolution image easily reaches a thousand-plus tokens), and stacked on top of a long document, the sequence balloons quickly. This is where the advantage of controllable long-sequence cost from hybrid linear attention becomes apparent. For evaluation, it’s better to use test sets that include Chinese-language scenarios, multi-turn image-text dialogue, and real documents, rather than fixating on a single benchmark score.

    6. A Rational View: An Evaluation System Matters More Than a Single Score

    Two extremes are common in these discussions: one holds that “the biggest model is the strongest,” and the other that “it’s just another routine iteration.” A more reasonable stance is to build a multi-dimensional evaluation system:

    Dimension Key Question Suggested Validation
    General capability Are reasoning, coding, and writing balanced? Regression on a self-built business test set
    Long context How long is the effective context? Needle-in-a-haystack, multi-document QA
    Multimodality Are chart, document, and interface understanding reliable? Real business image-text samples
    Engineering performance What are first-token latency, throughput, and concurrency? Load testing on a stress-test platform
    Cost Is per-token cost acceptable? Extrapolated from projected business volume
    Stability Are rate limiting and multi-AZ failover in place? Gradual rollout plus monitoring and alerts

    The “effective context” problem deserves special caution. Nominal support for a million-token context doesn’t mean the model can accurately recall key information at the 900,000th token. It’s worth inserting target information at positions like 0, 25%, 50%, 75%, and 95%, then examining how the recall rate changes with position before deciding on a long-document processing strategy.

    As for whether this can “reignite former glory,” the answer depends on the industry’s overall engineering and commercialization capability. The model is only the starting point; what truly determines value is whether it can serve massive request volumes reliably at reasonable cost, whether enterprises can integrate it into existing systems, and whether it can create differentiation in vertical scenarios. Chinese large models have entered the global first tier in architectural innovation, but there’s still a great deal of work to do in compute supply, inference cost optimization, and the application ecosystem.

    7. Practical Deployment Recommendations

    1. Abstract the model access layer: wrap authentication, rate limiting, routing, and caching behind a unified gateway so business code isn’t tied to a specific model. A model upgrade then only requires adjusting routing configuration.
    2. Build a regression test set: cover real business scenarios and run it after every model or prompt change—decide with data, not gut feeling.
    3. Manage context carefully: retrieve-then-inject for long documents, summarize and compress conversation history, and reserve long context for tasks that genuinely need it.
    4. Do capacity planning for on-premises deployment: estimate memory as weights + KV cache + activations, and in high-concurrency scenarios prioritize evaluating quantization schemes and multi-instance horizontal scaling.
    5. Make the whole pipeline observable: log input/output token counts, first-token latency, timeouts, and error rates for each request, and connect them to logging services and a monitoring/alerting system.

    8. Common Misconceptions

    • Parameter count above all: active parameters, training-data quality, and post-training methods matter just as much—parameter count alone doesn’t determine intelligence.
    • Context length above all: length is a capacity ceiling; effective recall rate is the real usability metric.
    • Demo above all: carefully curated demos differ greatly from real business distributions, so you must validate with your own data.
    • Ignoring total cost of ownership: long-term resource consumption during inference often far exceeds the one-time onboarding cost.
    • Treating linear attention as a cure-all: on short contexts, traditional attention kernels are still efficient, so choose by scenario.

    9. Conclusion

    The technical combination shown by the new-generation trillion-parameter model—sparse parameters, hybrid linear attention, attention residuals, and native multimodality—maps neatly onto four main threads of large-model evolution today: trading scale for capacity, architecture for efficiency, structure for trainability, and modality for scenario coverage. For China’s large-model community, this is a substantive step forward, but whether it “reignites former glory” ultimately comes down to real-world delivery.

    For developers, rather than chasing the emotion of a single launch, it’s better to focus on reusable engineering capabilities: a unified model access layer, a quantifiable evaluation system, a controllable cost structure, and an observable runtime pipeline. Master these, and no matter how the next generation iterates, you can quickly evaluate and integrate it. Baidu AI Cloud’s Qianfan platform provides unified access and fine-tuning for a range of mainstream large models and can serve as a practical foundation for enterprises building these capabilities.