LLM Architecture Evolution: From Hybrid Attention to Native Multimodality

    Recent LLM releases have made architecture choices a central topic: hybrid linear attention, attention residual connections, and native multimodal support. This article explains the engineering problems behind these directions, outlines their tradeoffs, and provides practical guidance for evaluation, model selection, deployment, and regression testing.

    Introduction

    The pace of LLM releases has accelerated significantly over the past two years. With each new model, the community tends to discuss similar questions: How large is the model? What changed in the architecture? How impressive are the demos? Can it renew momentum for Chinese LLM development?

    The attention these discussions receive shows that Chinese LLM teams have built strong technical and engineering capabilities. However, from the perspective of developers and enterprise adoption, a more useful question is: what engineering problems does a new architecture actually solve, and how should teams evaluate and select it?

    Enthusiasm can be a source of confidence, but technical judgment should remain clear and grounded.

    This article examines three architectural directions disclosed in recent model releases:

    1. Hybrid linear attention
    2. Attention residual connections
    3. Native multimodal support

    For each direction, it analyzes the design motivation, engineering value, practical limitations, and evaluation considerations.


    1. Beyond Parameter Scale: Architecture Efficiency and Inference Cost

    The industry often describes capability gains from larger parameter counts, more compute, and more data as brute-force scaling. Scaling laws describe approximate power-law relationships between model performance and parameters, data, and compute, providing a theoretical basis for large-scale pre-training.

    However, scaling laws mainly describe the relationship between performance and training investment. They do not directly determine inference efficiency. If total parameter count increases but decoding still relies on full attention with quadratic complexity, latency and memory overhead in long-context scenarios can quickly become difficult to control.

    Large-model deployment commonly faces three tensions:

    • Effect versus cost: Pushing capability boundaries often comes with high inference cost, and marginal returns may diminish.
    • Long context versus memory usage: As context length doubles, full-attention computation and KV Cache usage both increase noticeably.
    • Unified understanding and generation: Deciding whether to handle visual, document, and other non-text inputs through external modules or native modeling directly affects system complexity.

    Hybrid linear attention, attention residual connections, and native multimodality correspond closely to these three tensions.

    It is important to emphasize that architectural innovation does not negate the value of scale. Regardless of the architecture, parameter count, data quality, and compute investment remain fundamental to the capability ceiling.

    A reasonable approach is to evaluate scale and architecture efficiency separately:

    • Scale determines the upper limit of capability.
    • Architecture efficiency determines whether that capability is affordable and practical to run.

    2. Hybrid Linear Attention: Balancing Expressiveness and Complexity

    2.1 The Bottleneck of Standard Attention

    Standard attention uses the similarity between queries and keys to aggregate values across positions. For a sequence length of n and hidden dimension d, its time complexity is approximately O(n²d).

    During autoregressive decoding, each new token requires reading all historical keys and values. As the context grows, memory-bandwidth pressure during decoding increases significantly. This helps explain why long-context inference can feel slow after the first token, with generation throughput becoming limited.

    2.2 Linear Attention: From Retrieving History to Updating State

    Linear attention approximates softmax with a kernel function. By leveraging the associativity of matrix multiplication, it rewrites per-token attention into a process that maintains a fixed-size state matrix and updates it incrementally as new tokens arrive.

    Its complexity is approximately linear with respect to sequence length, and cache usage during decoding is largely decoupled from context length.

    The tradeoff is that the fixed-size state has limited capacity. The model may struggle to recall a large number of discrete facts accurately, reducing retrieval precision.

    Research in this area commonly introduces two improvements:

    • Gating mechanisms: Allow the state to dynamically determine how much information to retain or forget during updates, enabling more controlled memory management.
    • Differential update terms: Use state self-interference, similar to the Delta Rule, to replace old information with new information more precisely and improve associative recall.

    Both improvements belong to the broader technical family of gated linear attention, which is an important foundation for the state layers used in hybrid designs.

    2.3 Hybrid Stacking: Efficient Layers Plus a Few Precise Layers

    The hybrid approach combines two types of sequence layers in the same model:

    • Most layers use linear attention to model long-range dependencies efficiently.
    • A smaller number of layers retain full attention to handle precise retrieval.
    • MLP or feed-forward layers continue to store a large amount of knowledge.

    Industry practice suggests that a small number of key full-attention layers can be enough to compensate for the recall limitations of purely linear structures. As a result, periodic interleaving of gated linear attention and full attention has become one of the mainstream approaches.

    2.4 Practical Benefits

    • Long-context cost grows closer to linearly: Scenarios such as long-document summarization and repository-level code analysis become more manageable in marginal cost.
    • Better high-concurrency behavior: More stable cache reads and writes during decoding can improve serving throughput.
    • Stronger small-model competitiveness: Under the same compute budget, architectural efficiency can partially offset parameter disadvantages. This is especially useful for edge and private deployment scenarios.

    2.5 Usage Considerations

    • In hybrid models, different sequence layers may be executed with different parallelism strategies. When choosing an inference framework or tuning batching parameters, developers should follow the model provider’s official deployment guidance.
    • The practical benefits of linear attention layers depend heavily on context length, concurrency level, and hardware bandwidth. In short-context or low-concurrency scenarios, performance gains may be limited. Measurement is therefore essential.

    3. Attention Residual: Dynamically Routing Information Across Layers

    Traditional residual connections use a fixed identity mapping, adding the current layer’s output to the previous layer’s output. This design has been highly effective for training deep networks, but its routing mechanism is static. Signals crossing multiple layers can only be combined with fixed coefficients.

    The core idea of attention residual is to borrow the query-key matching mechanism from attention. Each layer uses its own query vector to dynamically aggregate outputs from selected historical layers.

    The design motivation can be summarized in three points:

    • Mitigating deep-network degradation: Problems such as gradient propagation difficulty and representation collapse can be alleviated by adaptively selecting information sources.
    • Improving reuse across hierarchy: Lower layers tend to capture local features, while higher layers focus on abstract semantics. Allowing higher layers to “look back” at lower-layer information can reduce information loss during layer-by-layer propagation.
    • Parameter efficiency: Dynamic routing mainly adds lightweight projection and scoring computation, without necessarily requiring a proportional increase in parameters.

    From an engineering perspective, two points deserve attention:

    • Cross-layer residual connections require caching outputs from multiple layers, which can amplify memory and bandwidth pressure. When the model has many layers, this should be evaluated together with parallelism strategies.
    • In small-scale experiments, “whether to use attention-style residual connections” and “the aggregation range of residual connections” can be treated as ablation variables and compared against standard residual connections.

    The above describes the general design pattern of this class of methods. For a specific model, details such as layer ratios, state dimensions, and routing scope should be verified against publicly available technical documentation.


    4. Native Multimodality: From Bolt-On Vision to Unified Representation

    Early multimodal solutions often used a concatenated architecture: vision encoder + projection module + language model.

    In this design, images are first processed by a vision encoder to extract features, then projected into the language space and inserted into the input sequence as additional tokens.

    This approach is engineering-mature, but it has clear limitations:

    • Visual and text representations come from different spaces, and fine-grained alignment often depends on later training to compensate.
    • The concatenated structure may generalize poorly across different tasks.
    • Handling images, charts, document layouts, and video frames simultaneously often requires stitching together multiple preprocessing pipelines.

    Native multimodality takes a different approach. During pre-training, visual tokens and text tokens learn together within the same attention mechanism and parameter set, allowing both modalities to share a unified representation space.

    Its engineering value includes:

    • More natural cross-modal reasoning: The model can jointly understand image and text information within the same context.
    • More stable understanding of complex visual objects: Structured visual content such as charts, formulas, diagrams, and mixed layouts can be parsed more reliably.
    • Simpler integration paths: A single interface can handle mixed image-and-text input, reducing the development cost of multimodal applications.

    For developers, native multimodality means that tasks such as “viewing images, reading tables, and understanding documents” can be orchestrated in the same system as text-only tasks. Examples include contract comparison, invoice information extraction, and Q&A over instructional materials.


    5. From “National Model Pride” to Reproducible Evaluation

    Whenever a Chinese model is released, discussions often focus on whether it can reignite momentum for Chinese LLM development. If this is understood as Chinese models gradually closing the gap with international frontier models, the expectation is reasonable.

    For developers, however, emotional judgment should not replace technical judgment. To objectively evaluate a model, it is recommended to build a reproducible and comparable evaluation system.

    5.1 Capability Dimensions Should Cover Real Tasks

    A single benchmark score can be affected by evaluation-set leakage, prompt-template differences, and other factors. A practical approach is to use private evaluation sets as the primary source and public benchmarks as supplementary support.

    Coverage should include task categories such as:

    • General question answering
    • Code generation and repair
    • Mathematical reasoning
    • Long-document understanding
    • Multimodal understanding
    • Tool calling

    5.2 Efficiency Should Distinguish First-Token Latency from Token Generation Speed

    Time to first token is mainly determined by the prefill stage. Output speed depends mainly on the decoding stage.

    These two metrics affect different user experiences: the first relates to perceived waiting time, while the second affects reading speed and interaction smoothness. They should be recorded separately.

    5.3 Robustness and Stability Should Be Based on Repeated Experiments

    Key test cases should be run multiple times. Success rate and variance should be recorded.

    LLM outputs are stochastic, so a single result is not sufficient as a basis for model selection.

    5.4 Cost and Substitutability Should Be Designed in Advance

    When comparing costs, teams should unify the measurement scope, including:

    • Input token price
    • Output token price
    • Context length
    • Cache hit behavior
    • Retry counts

    It is also advisable to keep an abstraction layer that allows switching between multiple models, avoiding lock-in to a single service.

    5.5 Automated Testing Should Be Embedded in Daily Workflows

    Model quality tests should be integrated into continuous integration workflows. Whenever prompt templates or model parameters change, automated regression tests can help detect performance regressions early.

    The following example shows a runnable evaluation framework skeleton. It defines a unified model-calling interface and task cases, making it easier to compare different models horizontally.

    1. """Lightweight model evaluation framework example: unified interface for comparing different models."""
    2. import hashlib
    3. import json
    4. import statistics
    5. import time
    6. from dataclasses import dataclass, field
    7. from pathlib import Path
    8. from typing import Callable
    9. class ModelAdapter:
    10. """Model adapter: implement complete when integrating a new model."""
    11. name: str = "base"
    12. def complete(
    13. self, prompt: str, system: str = "", max_tokens: int = 1024, temperature: float = 0.0
    14. ) -> dict:
    15. # Example: {"text": ..., "usage": {"prompt_tokens": 0, "completion_tokens": 0}}
    16. raise NotImplementedError
    17. @dataclass
    18. class EvalCase:
    19. id: str
    20. category: str # Task category, such as code / math / long_context / multimodal
    21. prompt: str
    22. system: str = ""
    23. judge: Callable[[str], tuple[bool, str]] = None # Judge function: (passed, reason)
    24. repeat: int = 3 # Repeat count, used to observe stability
    25. @dataclass
    26. class CaseResult:
    27. passed: int = 0
    28. total: int = 0
    29. first_token_ms: list = field(default_factory=list)
    30. total_ms: list = field(default_factory=list)
    31. @property
    32. def pass_rate(self) -> float:
    33. return self.passed / self.total if self.total else 0.0
    34. @property
    35. def p95_latency_ms(self) -> float:
    36. # p95: return the maximum when the sample size is small to avoid interpolation errors
    37. return max(self.total_ms) if len(self.total_ms) < 20 else sorted(self.total_ms)[int(len(self.total_ms) * 0.95)]
    38. def run_eval(model: ModelAdapter, cases: list[EvalCase], max_tokens: int = 1024) -> dict:
    39. results: dict[str, CaseResult] = {}
    40. for case in cases:
    41. stat = results.setdefault(case.id, CaseResult())
    42. for _ in range(case.repeat):
    43. t0 = time.perf_counter()
    44. first_token_ms, chunk_count = 0.0, 0
    45. for _chunk in model.stream(case.prompt, case.system, max_tokens):
    46. if chunk_count == 0:
    47. first_token_ms = (time.perf_counter() - t0) * 1000
    48. chunk_count += 1
    49. stat.total_ms.append((time.perf_counter() - t0) * 1000)
    50. stat.first_token_ms.append(first_token_ms)
    51. # After streaming ends, judge using the full text returned by stream
    52. ok, _reason = case.judge(model.last_full_text) if case.judge else (True, "")
    53. stat.passed += int(ok)
    54. stat.total += 1
    55. return {k: {
    56. "pass_rate": round(v.pass_rate, 3),
    57. "p95_total_ms": round(v.p95_latency_ms, 1),
    58. "avg_first_token_ms": round(statistics.mean(v.first_token_ms), 1) if v.first_token_ms else None,
    59. } for k, v in results.items()}
    60. if __name__ == "__main__":
    61. # Placeholder: replace with a concrete model adapter implementation in actual use
    62. model = ModelAdapter()
    63. # Evaluation cases should be desensitized samples from real business tickets and maintained long-term
    64. cases: list[EvalCase] = []
    65. report = run_eval(model, cases)
    66. Path("eval_report.json").write_text(json.dumps(report, ensure_ascii=False, indent=2)

    When using this framework, teams need to complete the streaming-call logic and full-text retrieval logic in the model adapter.

    The framework follows the interface-isolation principle: adding a new model only requires implementing one adapter. After that, the model can be compared horizontally using the same set of evaluation cases.


    6. From Evaluation to Deployment: Making New Architectures Work in Practice

    6.1 Long Context: Being Able to Read Does Not Mean Being Able to Use

    A larger context window does not guarantee that the model can fully utilize all content. A common issue is lost-in-the-middle behavior, where critical information located in the middle of the context is overlooked.

    A layered strategy is recommended:

    • Retrieval and localization
    • Structured summarization
    • Segmented questioning

    This allows deeper reasoning only over the fragments that are truly relevant.

    6.2 Multimodality: Preprocessing Quality Determines the Upper Limit

    Native multimodality reduces system complexity, but image clarity, resolution, and table-splitting quality remain critical factors affecting results.

    For scanned documents or documents with complex layouts, layout analysis should be performed first, followed by region-based processing before sending content to the model.

    6.3 Performance Tuning: Turning Architectural Features into Observable Metrics

    When deploying LLM services on cloud platforms such as Baidu AI Cloud, teams can use object storage to archive evaluation datasets and inference logs. Log services and monitoring alerts can be used to establish baselines and set threshold alerts for:

    • Time to first token
    • Output speed
    • Error rate

    Using the same evaluation framework before and after an upgrade makes the benefits of architectural improvements measurable.

    6.4 Data Security: Clear Boundaries and Responsibilities

    Enterprises should prioritize service offerings that provide data isolation and compliance commitments. Key questions include:

    • Whether data is used for training
    • How long data is retained
    • Whether access and usage can be audited

    For sensitive data, private deployment or dedicated data channels may be appropriate.


    7. Conclusion: Measurable Progress Matters More Than Hype

    From hybrid linear attention to attention residual connections, and further to native multimodality, architectural exploration by Chinese LLM teams is pushing domestic models toward closer competition with international frontier models.

    For developers, evaluating the value of new architectures ultimately comes down to three things:

    • Use evaluation as the basis for decisions: Rely on real-task success rate, latency, and cost.
    • Match architecture to scenario: Long context, multimodality, and low latency impose different requirements.
    • Use engineering to sustain capability: Even the strongest model still needs retrieval, orchestration, monitoring, and regression testing.

    Technical competition is not only about who has larger parameters. It is about who can run capability reliably in real business environments at a reasonable cost.

    When more teams incorporate evaluation systems, engineering pipelines, and cost awareness into model selection, technical progress in Chinese LLMs can be translated into real business value. That may be the most pragmatic answer to the discussion around Chinese model progress.