RNN Explained: The Starting Point of Sequence Modeling and a Guide to Choosing Models for LLMs
This article explains why Recurrent Neural Networks (RNNs) are the starting point of sequence modeling and modern large language models (LLMs). It covers why language cannot be treated as a "bag of words," how RNNs pass information step by step through a hidden state, and why that same design causes problems with vanishing/exploding gradients, lack of parallelism, and long-range dependencies. It then traces two improvement paths—fixing memory (LSTM/GRU) and changing the paradigm (attention and the Transformer)—and includes a comparison table, a practical selection checklist, common mistakes, and an FAQ to help you choose between RNN/LSTM, GRU, and Transformer architectures for real projects.
1. Why Is It Hard for a Machine to “Read” a Sentence?
For humans, understanding a sentence is almost instinctive: when you read a word, you automatically connect it to the preceding context and can anticipate what comes next. But for a machine, language is first a sequence of discrete symbols. To truly “understand” it, three problems must be solved: how to represent word meaning, how to model word order, and how to preserve long-range dependencies.
The takeaway up front: The RNN is the starting point of sequence modeling and of today’s large language models (LLMs). Its three defining characteristics—memory, serial processing, and gradients—largely determined the entire evolution path of sequence models over the following decade-plus: those who improved on it targeted memory, and those who replaced it targeted parallelism. Without this line of exploration, there would be no ability for AI to read and understand a sentence.
2. Why Can’t Language Be Treated as a “Bag of Words”?
Unlike images, language has a strong temporal (sequential) nature. “I love you” and “You love me” are made of exactly the same words, yet their meanings are opposite. Early bag-of-words (Bag-of-Words) models treat text as an unordered collection of words and discard order information, so they cannot distinguish these kinds of differences.
The core thing a sequence model must capture is the dependency between elements: the current state depends not only on the current input but also on all the information read so far. The RNN was the first architecture to systematically introduce the concept of “memory.”
3. Why Must an RNN “Pass Information Step by Step”?
The key design of an RNN is the hidden state. It reads the input one time step at a time, and the output at each step depends on the hidden state carried over from the previous step—much like a person reading an article word by word, holding the key information in mind as they go.
# Conceptual pseudocode: RNN forward pass# x[:, t, :]: [batch, input_dim] word vector at the current time step (one-hot or embedding output)# W_x: [input_dim, hidden_dim] W_h: [hidden_dim, hidden_dim] b: [hidden_dim]hidden = torch.zeros(batch, hidden_dim) # initial memory statefor t in range(seq_len): # read one time step at a timex_t = x[:, t, :]hidden = torch.tanh(x_t @ W_x + hidden @ W_h + b) # update memoryy_t = hidden @ W_o # produce output based on the current memory
W_x, W_h, and W_o are shared across different time steps. This brings an important advantage: the number of parameters does not balloon as sentence length grows, so the model can naturally handle variable-length sequences—a 10-word sentence and a 100-word sentence use the same set of weights.
If you want to try it hands-on, the minimal runnable example in PyTorch is just three lines:
import torch, torch.nn as nnrnn = nn.RNN(input_size=16, hidden_size=32, batch_first=True)x = torch.randn(4, 10, 16) # [batch=4, seq_len=10, input_size=16]out, h_n = rnn(x)print(out.shape, h_n.shape) # torch.Size([4, 10, 32]) torch.Size([1, 4, 32])
h_n is the final memory after the entire sentence has been processed—whether the earliest information is still inside it depends on the bottlenecks discussed in the next three sections.
4. Why Are RNNs Hard to Parallelize and Bad at Long Dependencies?
The “step-by-step passing” design is exactly what plants the hidden risks, and it is the direct reason RNNs have been replaced by Transformers in most mainstream scenarios.
- Vanishing and exploding gradients: During backpropagation, gradients must be propagated back through every time step. After many multiplications, they decay or grow exponentially, making it hard for the model to learn long-range dependencies. In a sentence of a few dozen words, information from the beginning may be nearly gone by the time it reaches the end.
- No parallel computation: Step t must wait for step t-1 to finish, forcing computation to be serial. On hardware built for parallelism, such as GPUs, RNNs struggle to fully use the available compute, limiting both training and inference speed.
- Long-dependency information bottleneck: The hidden state is the only carrier of memory and has a fixed dimension. As the sequence gets longer, early information is repeatedly “overwritten and diluted” within this limited state space.
5. Two Improvement Paths: First Fix Memory, Then Change the Paradigm
Path one: redesign the memory unit. Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) introduce gating mechanisms that let the model learn to “selectively remember and forget,” significantly mitigating the vanishing-gradient problem and enabling RNN-family models (both LSTM and GRU belong to the RNN family) to capture longer dependencies. But the fundamental flaw of serial computation remains.
Path two: introduce the attention mechanism. Attention lets the model, when processing the current position, directly “look back” at every other position in the sequence and assign different weights to each. No matter how far apart two words are, a direct connection can be established, bypassing the information decay of step-by-step passing. Building on this, the Seq2Seq framework was widely adopted for tasks like translation and summarization, and attention gradually became a standard component.
| Dimension | RNN (vanilla) | LSTM / GRU | Transformer |
|---|---|---|---|
| Time-step parallelism | No | No | Yes (positions can be computed simultaneously within the sequence) |
| Long-dependency ability | Weak, gradients vanish easily | Medium, mitigated by gating | Strong, any two positions reach each other in one step |
| Do parameters grow with sequence length | No | No | No, but attention compute grows roughly with the square of length |
| Inference latency characteristics | Low per-step latency, streamable | Slightly higher per-step latency, streamable | High first-token latency, large batch throughput |
| Typical use cases | Minimal online models, sensor time series | Medium-length time series, resource-constrained edge | Large-scale pretraining, long-text understanding and generation |
6. The Transformer: Self-Attention at the Core, Not “Attention Only”
The landmark 2017 paper that introduced the Transformer architecture changed the research paradigm of natural language processing (NLP). Its central claim is this: model sequences with self-attention at the core, rather than with attention alone—a complete Transformer block also includes a feed-forward network (FFN), residual connections, layer normalization (LayerNorm), and positional encoding. Attention is responsible for “exchanging information across positions,” the FFN is responsible for “position-wise nonlinear transformation,” and positional encoding restores the order information that recurrent structures originally carried intrinsically.
This shift brings three decisive benefits:
- Full parallelism: The representation at any position can be computed simultaneously, greatly improving training efficiency and making large-scale pretraining feasible from an engineering standpoint.
- Stronger long-dependency modeling: Any two positions reach each other in one step, unaffected by distance-based decay.
- Easy to scale: The regular, uniform structure grows fairly stably with parameter and data scale—this is the engineering prerequisite for today’s large language models.
Placed on a longer timeline: in 1950, Turing proposed the classic test idea for judging machine intelligence; in 1956, the Dartmouth conference formally established the field of artificial intelligence. Since then, expert systems attempted early commercialization, the backpropagation algorithm solved the training problem for multi-layer networks, deep belief networks reignited deep learning research, AlexNet (2012) sharply improved error rates in a large-scale image recognition competition, and Word2Vec (2013) mapped words into dense vectors. Together these milestones form a single main thread: making machines increasingly capable of processing sequential data. The RNN is the pivotal, bridging link on this thread.
7. Does the RNN Still Have a Place? Being Clear About Its Boundaries
“Being replaced” does not mean “being phased out.” In streaming and real-time scenarios (speech streams, sensors, word-by-word online incremental inference), in low-resource and edge deployments (small parameter count, low memory footprint, no need to cache the entire history), and in tasks that involve very short sequences with strict latency requirements, RNN/LSTM still holds practical advantages: its computational complexity is roughly linear in sequence length, and the state size stays constant, whereas self-attention’s compute and memory overhead grows roughly with the square of length. Recently emerged approaches such as state space models (SSMs) are, in essence, continuing to explore along this same idea of “constant state, linear complexity.”
8. Selection Checklist: Which One to Use and When?
- Short sequences + online incremental inference + a strict latency budget → prefer RNN / LSTM / GRU; switch to LSTM when you need longer memory.
- Edge or memory-constrained deployment where the model must be very small → GRU or a single-layer LSTM, deployed after quantization.
- Need parallel training, long-range dependencies, or large-scale pretraining → Transformer / attention architectures.
- Very long text with tight GPU memory → consider sparse attention, linear attention, SSMs, and other schemes that reduce length complexity.
- Already running an RNN service in production: first quantify the gains (accuracy, P99 latency, memory), then decide whether to migrate—don’t rewrite simply because a technique is “newer.”
There are three common mistakes: handing a sequence task to a bag-of-words model and losing word order; forcing a vanilla RNN to handle long dependencies so that training fails to converge, then misattributing it to “not enough data”; and abandoning streaming output just to use a Transformer, pushing first-token latency from tens of milliseconds up to hundreds of milliseconds.
9. Frequently Asked Questions
How do RNNs and CNNs differ on sequence tasks? A CNN extracts local patterns in parallel within a fixed window via 1D convolution; it is naturally parallelizable, but its receptive field is limited by kernel size and the number of layers. An RNN recurses time step by time step, with a receptive field that theoretically covers the whole sequence, but it cannot be parallelized.
Is LSTM always better than a vanilla RNN? On long-dependency tasks, usually yes, but it has more parameters and heavier per-step computation. For short sequences or extremely tight compute, a simple RNN may offer better cost-effectiveness.
When should you migrate from RNN to Transformer? When one of three signals appears: training cannot fully utilize the GPU, long-range dependencies consistently fail to be learned, or you need to unify onto a large-scale pretraining technology stack.
10. Summary
The RNN was the first to give machines the ability to “read and remember as they go,” establishing the fundamental paradigm of sequence modeling. Constrained by serial computation and gradient problems, it has been replaced by the Transformer in most mainstream scenarios, yet it remains an unavoidable starting point. LSTM/GRU fixed the memory weakness, the attention mechanism opened up the ideas of long dependencies and parallelism, and the Transformer brought these explorations together into a bridge toward large language models.
Understanding the RNN is not merely a look back at technical history—it is understanding why today’s AI can read and comprehend the sentence you say. It inherited the modeling intuition of “step-by-step passing,” and then completed a paradigm shift built on top of that intuition’s limitations.
