LLM Architecture Evolution: Aggressive vs. Conservative Choices Explained

    This article examines how flagship large language model (LLM) architectures have converged on a common skeleton — hybrid linear/sparse attention, multi-path gated residuals, and explicit external memory. It argues that "aggressive" and "conservative" cannot be judged on the same level, and shows why the real differences lie in module origin, validation depth, scope of change, training stage, and how modules are assembled. It breaks down the five cost factors that make a conservative approach rational, offers a layered framework for classifying a team's choices, and translates the conservative stance into concrete engineering practices: module admission criteria, staged change management, inference-side optimization, and a pre-launch evaluation checklist.

    LLM Architecture Evolution: Aggressive vs. Conservative Choices Explained

    Over the past year, an unexpected pattern has emerged in flagship models: the publicly disclosed attention backbones are looking more and more alike. Linear attention and sparse attention are stacked in roughly a 3:1 ratio, residuals have grown from a single trunk into multi-path gated connections, some models add a lookup-table memory, and even the layer ratios are converging. If everyone is converging, why do some teams seem to “tinker” far more than others? How should “aggressive” versus “conservative” be defined and judged? That is the core question this article sets out to answer.

    1. First, look at the phenomenon clearly: the skeleton really is converging

    Compiling the flagship architectures disclosed by different teams, they can be summarized in the table below. “Team A/B/…” is used only to distinguish technical routes and does not point to specific vendors.

    Team Attention backbone Residual/connection Memory module Layers & ratio Total / active params
    A (latest-gen Flash-like) Asymmetric attention: CSA2 + sliding window attention (SWA) Bounded-replay single-pass mHC Engram lookup-table memory, ~196B — ~552B total; ~8B active on input, ~16B on output; KV cache in FP4
    B (Flash-like) Linear attention + sparse attention (DSA) Hybrid residual — 45 layers = 34 linear + 11 DSA, ~3:1 320B/18B; index-pool compression 4:1
    C (flagship Pro-like) Gated linear attention (KDA) + gated sparse attention (Gated DSA) Attention residual (AttnRes) + stable latent MoE — 93 layers = 69 KDA + 24 Gated DSA 2.8T/104B
    D (Flash-Next-like) Gated linear attention (GDN) + sparse attention (QSA) Gated residual N-gram lookup-table memory 48 layers = 36 GDN + 12 QSA, 3:1 125B active/6B, plus 51B lookup params
    E (preview) Gated sparse attention + IndexCache Four-path residual + multi-token prediction (MTP) — 78 backbone layers 770B/49B
    F (paper only, unreleased) High-sparsity attention (HySparse2) KV bridging + KV reuse No standalone indexer 49 layers in paper; 80B-A3B in experiments —

    Three common trends stand out:

    1. Attention is shifting from a single type to a hybrid stack. Pure softmax full attention is no longer the only option: linear attention handles long-sequence modeling for most layers, while sparse attention restores precise token-level retrieval in a few layers. A 3:1 ratio has become a recurring pattern across the industry.
    2. Residuals are shifting from single-path addition to multi-path gating. Multi-path residuals, gated residuals, and attention residuals are all variations on the same idea: letting information from different depths flow back with learnable weights to ease gradient and representation degradation in deep layers.
    3. Memory is shifting from implicit-within-parameters to explicit-outside-parameters. N-gram lookups, Engram-style embedding-level memory, index caches, and KV reuse all attach an addressable store outside the model. Lookup-table memory can be substantial in size — reaching 51B parameters in one case.

    So the surface-level judgment that “everyone is slowly converging” holds. What really needs discussion is where the strategic differences between teams lie within that convergence.

    2. Two calibrations: don’t rush to a conclusion

    2.1 Official statements and config field names are not the same thing

    One team’s official blog mentioned only “linear attention.” The KDA field that appears in the config file is simply that team’s name for its own linear-attention scheme. Treating a config field name as the official technical term makes it easy to draw wrong conclusions.

    2.2 A flagship Pro is not a base model that was re-pretrained

    One team’s flagship Pro model card states that it is a post-training iteration of the previous flagship, and one of the changes is swapping 24 layers of gated latent attention (Gated MLA) for gated sparse attention (Gated DSA). It is not a base model retrained from scratch.

    This distinction determines every judgment that follows: replacing some layers on an already-trained model and adopting a new architecture directly during pre-training have completely different engineering implications. The former is low-cost and controllable; the latter is what truly counts as “aggressive.” When discussing “who is more aggressive,” these two types of action should be viewed separately.

    3. Tracing the modules: the repeatedly borrowed modules have all passed scale validation

    Aligning each module with its first public appearance reveals a clear evolution chain:

    Module First publicly proposed Later cited by
    MLA (multi-head latent attention) An open-source team, 2024-05 Sparse attention and later schemes use it as a baseline; some teams added gating on top
    KDA (gated linear attention) A smart-assistant team, 2025-10 So far only used by that team’s own flagship Pro
    DSA (fine-grained sparse attention) An open-source team, 2025-09 Named in official materials by Zhipu, Moonshot, and Tencent; a team’s flagship Flash-like and Pro-like both adopted it
    mHC (hybrid residual connection) A team’s paper, 2025-12 That team’s official blog, plus the official docs and model cards of two other teams
    Engram (lookup-table memory) A team in collaboration with a university, 2026-01 A team’s latest flagship official docs, plus official materials from two other teams

    Three points are worth noting:

    • Propagation is now measured in months. From first proposal to integration by peers, the cycle has compressed to a few months.
    • They all come from projects validated at scale. The authors of the cited modules completed large-scale validation on their own flagship models, which lowers the risk for those who borrow them.
    • Only KDA is “used in-house only.” All other modules have been adopted across teams. This doesn’t mean KDA is inferior — it means its breadth of validation is still at an early stage.

    4. The core answer: aggressive and conservative must be examined layer by layer

    Let me start with a conclusion: “aggressive” and “conservative” are not being compared on the same level.

    4.1 Conservative in module selection, aggressive in integration method

    The modules different teams choose overlap heavily; the differences mainly show up in how they are assembled:

    • Ratios: all use a “linear + sparse” hybrid, but some run about 3:1, some nearly 3:1 (69:24); some introduce gating in certain layers, others cache the index of the sparse branch.
    • Asymmetric design: the attention backbone can be written as “asymmetric CSA2 + sliding window attention.” Linear attention is good at compressing history and controlling memory; sliding window attention is good at close-range local precision. Dividing the labor between them is itself a structural innovation.
    • Residual form: single-path, multi-path, gated, and attention residuals are all in use, and how the weights are initialized and normalized directly affects stability.
    • Memory form: some put memory at the embedding layer as an N-gram lookup, others introduce a standalone Engram module; some teams use index caching, KV bridging, and KV reuse as alternatives that eliminate a standalone indexer.
    • Sparsity and decode precision: the same idea comes in “high-sparsity” and “medium-sparsity” tiers, and KV precision also offers different choices such as BF16, FP8, and FP4.

    4.2 Conservative in scope of change, aggressive in timing of change

    One team directly swapped out a quarter of its attention layers between flagship generations — a classic case of “post-training aggressiveness.” Another team introduced asymmetric attention, a new residual form, and lookup-table memory all at once in its latest flagship — “generational aggressiveness.” A team that adopted many in-house structures early and now favors mature modules in its last two generations is being “conservative at the base-model level.”

    The same team can behave in opposite ways at different stages. Pulling out a single version of a team and calling it “conservative” or “aggressive” is not a robust conclusion.

    4.3 There is also “formal conservatism”: replicating a public configuration

    Another case is more subtle: if a new model reproduces a public paper’s configuration layer by layer — for example, the experimental setup of 49 layers and 80B total / 3B active parameters — then its “conservatism” is not just at the selection level; the entire configuration path is constrained by public material. Fast formal following does not necessarily equal low engineering risk.

    5. Why “tinkering less” is rational: five cost factors

    5.1 The cost and irreversibility of base-model pre-training

    A single frontier-scale pre-training run — from data preparation and compute scheduling to training stability — makes any problem hard to roll back locally. The mainstream approach is to run ablation studies at small and medium scale first, verify the scaling trend, and only then scale up to flagship size. This means the decision points are concentrated in the few months before training starts, with a narrow fault-tolerance window.

    5.2 The hidden constraints on the inference side

    This is the most easily overlooked item:

    • Linear attention turns the KV cache into a fixed-size recurrent state. Memory usage becomes predictable, but this summary-style memory cannot precisely backtrack to early tokens, so exact retrieval has to fall back on a few sparse attention layers.
    • Sparse attention performs index computation at every layer; some schemes use 4:1 index-pool compression to reduce the overhead. The indexer itself needs extra compute and memory.
    • After mixing linear and sparse attention, each layer’s compute and memory-access characteristics differ, so operator fusion, parallel partitioning, and batching strategies all have to be redesigned.
    • Lookup-table memory brings sparse memory access, so bandwidth utilization and batching efficiency become new tuning targets.
    • 4-bit KV cache, MTP, and index caching all require dedicated support in the inference engine.
    1. # Illustration: KV cache memory estimation (for order-of-magnitude understanding only;
    2. # ignores indexer, paging, and framework implementation differences)
    3. def kv_cache_bytes(n_layers, n_kv_heads, head_dim, seq_len, batch, bits):
    4. # One copy each for K and V per layer
    5. return n_layers * 2 * n_kv_heads * head_dim * seq_len * batch * bits / 8
    6. for name, kv_layers, bits in [("dense attention", 45, 16), ("3:1 hybrid", 11, 16), ("3:1 hybrid + FP4", 11, 4)]:
    7. gb = kv_cache_bytes(kv_layers, 8, 128, 131072, 1, bits) / 1024 ** 3
    8. print(f"{name:<16} single-request 128K context KV cache ≈ {gb:.2f} GiB")
    9. # dense attention: single-request 128K context KV cache ≈ 22.50 GiB
    10. # 3:1 hybrid: single-request 128K context KV cache ≈ 5.50 GiB
    11. # 3:1 hybrid + FP4: single-request 128K context KV cache ≈ 1.38 GiB

    These few lines of illustrative code explain why the industry converges: on top of a 3:1 hybrid, simply dropping the KV cache of the sparse layers from 16 bits to 4 bits saves about another three-quarters. The evolution of attention architecture is largely driven by inference cost.

    5.3 Ecosystem adaptation and compatibility

    New architectures require the inference engine, quantization schemes, fine-tuning frameworks, and evaluation baselines to adapt in sync. The more aggressive the architecture, the larger the ecosystem gap at launch. Adopting modules that have already been validated lets you reuse a large amount of existing optimization and experience, with significantly lower launch risk.

    5.4 Training stability and reproducibility

    Gated residuals, hybrid layers, and lookup-table memory all introduce new trainable parameters and numerical paths. Their sensitivity to initialization, learning rate, gradient clipping, and precision strategy needs to be recalibrated. Borrowing a public configuration is equivalent to inheriting hyperparameters the original authors have already tuned.

    5.5 Difficulty of evaluation and attribution

    When an architecture changes in several places at once, gains are hard to attribute to a single module. The incremental “swap a little in each layer, add a little everywhere” approach often makes it harder to explain where the gains come from in public evaluations. Teams therefore tend to follow modules that have been validated in isolation, concentrating innovation on one or two points they can independently justify.

    6. A layered framework for judging aggressive vs. conservative

    Rather than argue about “who is more aggressive,” it’s better to break the question into a few checkable dimensions:

    Dimension Key question Leans conservative Leans aggressive
    Module origin Does it use modules already publicly validated by others? Yes, with attribution In-house new operators/modules
    Validation depth Only small/medium experiments, or flagship-scale validation? Stays in papers and ablations Landed at flagship scale
    Scope of change New layers, replaced layers, or a swapped backbone? Adding or local replacement Replacing the attention backbone
    Training stage Adopted in pre-training, or replaced in post-training? Post-training iteration Built directly into the base-model pre-training
    Integration method Does layer order, ratio, gating, and precision follow a public config? Follows it Redesigned
    Inference ecosystem Does the inference engine need heavy customization? Reuses existing optimization Requires deep customization

    Applying this framework to the phenomenon at the start:

    • Post-training replacement of a quarter of the attention layers is aggressive on the “scope of change” and “training stage” rows, but conservative on “module origin” and “validation depth.”
    • Introducing asymmetric attention, a new residual, and lookup-table memory across a generation is aggressive on several dimensions, but the modules it relies on have all been publicly validated.
    • A team that built many in-house modules early and has shifted to mature ones in its last two generations is trending conservative on “module origin” while still exploring on “integration method.”

    7. Turning “conservative” into executable engineering practice

    7.1 Selecting modules: four admission criteria

    1. Validated at flagship scale: it has at least demonstrated its scaling trend at billion- to hundred-billion-parameter scale, rather than working only on small models.
    2. A reusable implementation exists: the inference engine and training framework already have a corresponding implementation, or a clearly portable reference implementation.
    3. Gains can be isolated: it can be A/B tested against a baseline under fixed data and budget, with gains attributable on their own.
    4. There is an exit mechanism: it can roll back to the previous generation’s architecture when something goes wrong, without blocking the release cadence.

    7.2 Controlling the pace: single-point changes, staged validation

    1. # Illustration: hybrid stacking construction logic (pseudocode)
    2. # In the config, both the ratio (RATIO) and whether to gate (GATED) are variables,
    3. # which makes single-point ablation easier
    4. def build_layers(n_layers, ratio=3, gated=False):
    5. layers = []
    6. for i in range(n_layers):
    7. if i % (ratio + 1) == ratio: # Insert one sparse attention layer every ratio+1 layers
    8. layers.append(SparseAttnLayer(top_k=2048, index_top_k=512))
    9. else:
    10. layers.append(LinearAttnLayer(state_dim=128))
    11. layers.append(GatedResidual(gated) if gated else Residual())
    12. return layers
    13. # Change order: ratio 1→3→7 to observe the quality/latency curve, then enable gating
    14. # separately — change only one variable at a time

    7.3 Inference side: spend the saved memory where it matters most

    The KV memory saved by hybrid attention can be converted into longer context, higher concurrency, or larger batch sizes. But note:

    • Prefix cache hit rates will change. Whether the recurrent state of linear attention can be safely reused depends on whether the implementation supports state snapshots.
    • The stability of sparse indexing. Index-selection jitter can cause output fluctuations for the same request, so the tail latency of time-to-first-token (TTFT) needs to be tracked.
    • The precision vs. long-context tradeoff. Low-bit KV cache quantization is usually paired with per-channel scaling and outlier retention; long-context tasks need to focus on validating “recall of middle-of-context information.”
    • Load testing must cover real workloads. When short and long requests are mixed, the utilization difference between linear and sparse layers is amplified.

    7.4 Before launch: a checkable evaluation checklist

    • Structural attribution: comparison against the baseline with the same data, the same token budget, and the same tuning budget.
    • Long context: retrieval-style tasks at the 32K/128K level, such as multi-needle retrieval and cross-document aggregation.
    • Serving metrics: throughput, TTFT P99, time per output token (TPOT), peak memory, prefix cache hit rate.
    • Stability: perplexity difference before and after quantization, special-token handling, truncation behavior for over-long inputs.
    • Rollback: model versions, inference config, and business routing support a one-click switch back.

    7.5 Team capability: turn structural experiments into a pipeline

    Capabilities worth borrowing, by layer:

    • Training side: distributed training platform, elastic compute scheduling, experiment tracking and config management, checkpoint management.
    • Data side: data version management, deduplication and quality filtering, data-mix experiments.
    • Inference side: a unified inference serving layer, dynamic batching, quantization and cache management.
    • Ops side: monitoring and alerting, logging services, canary releases and fast rollback.

    Common cloud capabilities — object storage, container platforms, message queues, logging services, and monitoring/alerting — can be combined into an end-to-end experiment-and-launch pipeline.

    8. Conclusion: to read the trend, look at the “layers”; to read the strategy, look at the “timing”

    • Surface convergence: hybrid linear/sparse attention, multi-path gated residuals, and explicit memory modules are becoming common choices for flagship models.
    • The real differences: not in whether a module is used, but in the module’s origin, validation depth, scope of change, training stage, and integration method.
    • “Conservative” does not mean behind: following validated solutions in module selection and concentrating innovation on one or two attributable points is currently the most cost-effective evolution strategy.
    • “Aggressive” does not mean advanced either: without isolable validation and a rollback mechanism, aggressive changes transmit risk to the entire product.

    For individual developers and small and medium teams, the more valuable approach is not to chase every new module, but to build your own “structural admission checklist” and “layered validation process,” so that under a controllable cost, every structural change becomes an experiment that is measurable, rollback-able, and explainable.