KDA, AttnRes, and LatentMoE: Understanding Three Emerging LLM Architecture Designs
This article examines KDA, AttnRes, and LatentMoE from three perspectives: design motivation, implementation principles, and engineering trade-offs. It explains how dynamic block-sparse attention, cross-layer attention residuals, and latent-space mixture-of-experts architectures address attention cost, deep information flow, and model capacity. It also provides practical evaluation criteria and deployment considerations without drawing performance conclusions about any specific model.
Recently, several open-source large language models with hundreds of billions of parameters have attracted significant attention. Their weights have been downloaded widely, and inference frameworks have begun adding support for them. Beyond the publicity, three architectural ideas are particularly worth examining: KDA, AttnRes, and LatentMoE.
This article analyzes each design from three perspectives: why it is needed, how it works, and what engineering costs it introduces. It also proposes a practical evaluation framework. The discussion focuses on architectural ideas rather than the performance of any specific product.
1. Why New Architecture Designs Are Needed
Traditional dense Transformers face two major bottlenecks.
First, standard multi-head attention has computational and memory costs that grow approximately quadratically with sequence length. In long-context scenarios, the KV cache can consume a substantial amount of GPU memory.
Second, as parameter counts increase, training costs generally increase as well, while the capability gains from simply adding more parameters gradually diminish.
The underlying idea behind these newer designs can be summarized in two principles:
- Concentrate computation where it is actually needed.
- Store parameters in a more reusable form.
KDA, AttnRes, and LatentMoE represent three different approaches to attention computation, deep information flow, and feed-forward network capacity.
2. KDA: Dynamic Block-Sparse Attention
2.1 Motivation
Standard attention calculates a similarity score for every query-key pair. In real text, many token pairs have weak relationships, so the value of computing all these scores is limited.
KDA first uses a lightweight module to estimate which key blocks are likely to be useful. It then performs full attention only over the selected blocks.
2.2 How It Works
A typical KDA implementation contains three components:
- Importance estimation: A low-rank projection estimates the score of each key block for every query.
- Top-k selection: Each query retains only the top-k blocks, while the remaining blocks are masked.
- Different handling during training and inference: Training commonly uses a differentiable approximation so that gradients can propagate. Inference uses hard selection to obtain actual computational savings.
The following pseudocode illustrates the data flow only. It is not production-ready code.
# Take the mean of each key block as its representative vector.# Shape: [num_blocks, d]block_repr = k.view(num_blocks, block_size, d).mean(dim=1)scores = low_rank_predict(q, block_repr) # Lightweight scoringidx = topk(scores, k=sparse_budget) # Select k blocks per queryattn = softmax(q @ k[idx].T / sqrt(d)) @ v[idx]
A practical implementation must also handle block size, masking, data types, and causal constraints.
Sparse attention can reduce the number of KV entries involved in attention, but the scoring process itself introduces overhead. If blocks are too small, this overhead may offset the gains from sparsification.
2.3 Engineering Impact
Sparse attention primarily reduces attention computation for long sequences and the amount of KV data that must be read from memory at each step.
When combined with mechanisms that move cold KV entries to system memory or storage, it may also reduce memory consumption during long-context inference. However, whether memory usage decreases, and by how much, depends on sequence length, block size, estimator accuracy, and hardware bandwidth. These effects must be confirmed through measurement.
3. AttnRes: Attention Residual Connections
3.1 Background
In deep Transformers, attention outputs are repeatedly mixed across layers. Fine-grained features from shallow layers may be smoothed out during propagation. In some cases, attention distributions in deeper layers become more uniform and token representations become increasingly similar. This phenomenon is often described as representation collapse.
3.2 Mechanism
In addition to the standard residual connection, AttnRes introduces a cross-layer path. The current layer can access not only the previous layer but also attention outputs from several earlier layers.
One possible formulation is:
h_l = Attn(h_{l-1}) + h_{l-1} + α · Attn(h_{l-d})
Here:
αis a learnable gating coefficient.dis the cross-layer offset.
Unlike an ordinary residual connection, this path carries attention outputs—the results of modeling relationships between tokens—rather than only the original hidden states.
A common strategy is to initialize α to zero. This makes the network behave similarly to a standard residual architecture at the beginning of training. The model can then gradually learn whether cross-layer information is useful.
3.3 Benefits and Risks
- Potentially reduces representation collapse: Deeper layers can directly access attention information from shallower layers, which may help model long-range dependencies.
- May improve gradient flow: The additional path provides a more direct gradient route to deeper layers. Whether it actually improves training stability must be verified through training curves.
- Increases memory usage: Several historical attention outputs must be cached. The resulting activation memory cost can be significant in deep models.
- Requires tuning: If the offset
dis too large, the information may become stale. Ifαbecomes too large, it may interfere with the original signal.
4. LatentMoE: Mixture of Experts in a Latent Space
4.1 From Standard MoE to LatentMoE
In a standard Mixture-of-Experts architecture, each expert stores a complete feed-forward network. A router activates only a small number of experts for each token, using sparse activation to provide greater total capacity.
However, as the number of experts increases, total parameter counts and the cost of distributing and loading weights can grow rapidly.
LatentMoE takes a different approach. Instead of storing a complete independent feed-forward network for every expert, it gives all experts a shared set of low-rank bases. Each expert stores only the coefficients used to combine those bases. The effective expert weights are constructed from the bases and coefficients when needed.
4.2 Structural Overview
The following pseudocode illustrates how the down-projection and up-projection weights can be materialized. Here:
Bis the number of bases.ris the low-rank dimension.Eis the number of experts.dis the model dimension.fis the intermediate FFN dimension.
# Shared latent-space bases: B low-rank basesbasis_down = Parameter(B, d, r)basis_up = Parameter(B, r, f)# Each expert stores only B combination coefficients.# One set is used for the down projection and one for the up projection.coeff_down = Parameter(E, B)coeff_up = Parameter(E, B)W_down = einsum('eb,bdr->edr', coeff_down, basis_down)W_up = einsum('eb,brf->erf', coeff_up, basis_up)# Forward pass:# y = W_up[e] · act(W_down[e] · x)
Under this structure, the parameter count for the experts decreases from approximately:
E · 2 · d · f
in a standard MoE to approximately:
B · r · (d + f) + 2 · E · B
When B is much smaller than E, and r is significantly smaller than both d and f, the parameter savings can be substantial.
Conversely, if the number of bases approaches the number of experts, or if the low-rank dimension is too large, the compression benefit quickly weakens. The cost of materializing the effective weights also increases.
4.3 Routing and Deployment
LatentMoE still has to address the classic challenges of MoE systems:
- Load balancing: Auxiliary losses or dynamic bias adjustment can be used to prevent a small number of experts from receiving too many tokens.
- Expert specialization: Shared bases may make experts more similar to one another. Router temperature and the number of experts need to be tuned together.
- Deployment and loading: Expert weights can potentially be loaded on demand from slower storage. This requires the inference framework to support expert paging and caching. Otherwise, frequent swapping can significantly increase latency.
5. Combining the Three Designs
| Dimension | Technology | Target problem | Main cost |
|---|---|---|---|
| Attention computation | KDA | Long-sequence attention overhead | Scoring overhead and block-selection errors |
| Information flow | AttnRes | Deep representation collapse and training stability | Historical activation memory and tuning complexity |
| Feed-forward capacity | LatentMoE | Parameter scale and deployment cost | Weight materialization and increased expert similarity |
The intended combination is as follows:
- KDA reduces attention computation in each layer.
- AttnRes may reduce information loss caused by sparsification and deep propagation.
- LatentMoE uses a more compact parameter representation to expand model capacity.
These are testable hypotheses, not established conclusions. To determine whether the combination works in practice, evaluations should compare models with and without each component while keeping the number of active parameters consistent. Quality, latency, and memory usage should all be measured.
6. Suitable and Unsuitable Scenarios
These designs are not appropriate for every use case. Extra caution is needed in the following situations:
- Applications dominated by short inputs: When sequences are short, the cost of sparse scoring may exceed the savings from reducing attention computation.
- Online services with extremely strict latency requirements: On-demand expert loading can introduce tail latency. Popular experts may need to be warmed up or kept resident.
- Small training teams: Both AttnRes and LatentMoE require careful hyperparameter tuning, which can increase the cost of building and maintaining a training pipeline.
- Research requiring strict reproducibility: If sparse kernels and quantization formats are not yet stably supported by the inference framework, results may vary across framework versions.
7. Evaluation Method and Selection Checklist
Public data often lacks a consistent evaluation methodology. Download counts, community leaderboards, and one-off demonstrations are not sufficient to support a reliable conclusion.
A practical evaluation process should include the following steps:
- Build an internal evaluation set that reflects real production inputs and includes long-context samples.
- Record throughput, time to first token, and peak memory at multiple sequence lengths rather than looking only at averages.
- Check refusal behavior, factual errors, and code hallucinations. These issues are often invisible in single-point benchmark results.
- Verify whether the selected inference framework version, sparse kernels, and expert-loading mechanisms are officially supported rather than community patches.
- Deploy through a gradual rollout with explicit rollback conditions. For example, automatically switch back to the original model if error rates, latency percentiles, or cost exceed defined thresholds.
- Keep the quantization format, batch size, and hardware fixed when comparing results. Otherwise, configuration differences may be mistaken for architectural differences.
8. Practical Recommendations by Role
- Architecture researchers: Start by reproducing the LatentMoE basis decomposition on a small model. Examine changes in expert similarity and routing quality before applying the approach at larger scale.
- Inference engineers: Prioritize measuring the actual benefits of KV-cache offloading and expert paging. Record latency distributions for each strategy rather than relying only on average latency.
- Application developers: Use internal evaluation data as the primary decision criterion instead of relying on a single community benchmark. Pay particular attention to refusal behavior and code correctness.
- Enterprise users: Existing container platforms, object storage, and monitoring systems can be used to build an inference service. Until framework support becomes mature, keep the previous model available as a fallback.
Conclusion
KDA, AttnRes, and LatentMoE address three different challenges: attention computation, deep information flow, and expert parameter scale.
Their value is not determined by the results of one particular model version. Instead, they provide a set of architectural ideas that can be separated, tested, and evaluated.
Whether a design is suitable for a specific workload ultimately depends on sequence length, hardware conditions, and quality requirements. The answer should come from measurements conducted under the target deployment conditions.
