LLM Architecture Evolution: The Divergence of Attention Mechanisms
Since the Transformer architecture was introduced in "Attention Is All You Need" (2017), improvements have never stopped, but their pace has clearly diverged across two core modules. Position encoding at the input, the FFN at the output, and the attention mechanism—which dominates both compute and memory cost—each face different constraints. After years of engineering practice, the industry has gradually reached a conclusion: the FFN module has converged, with different teams landing on broadly similar answers, while the attention module keeps generating disagreement, with each vendor pursuing a different technical path. Understanding this "one converging, one diverging" pattern is key to grasping the current evolution of LLM architecture.
This article examines how two core Transformer modules have evolved in opposite directions: the FFN layer has converged on fine-grained Mixture-of-Experts (MoE), while the attention mechanism has branched into multiple parallel routes—linear, sparse, compression/latent representation, and hybrid stacking. It explains the bottlenecks of standard softmax attention, the cost structure of each route across compute, memory, and global recall, and offers practical engineering guidance for choosing architectures under real-world constraints on context length, throughput, memory, and accuracy.
