From One Sentence to Context: How Recurrent Neural Networks Work and How to Use Them
Recurrent neural networks (RNNs) process sequential data by carrying a hidden state from one time step to the next. This article explains their core equations, common sequence-to-output patterns, training with Backpropagation Through Time (BPTT), vanishing and exploding gradients, gated architectures such as LSTM and GRU, input preparation, evaluation, deployment, and practical criteria for deciding when an RNN is appropriate.
1. Why Understanding a Sentence Requires More Than Individual Words
“This feature is very useful” and “This feature is not very useful” share many of the same words, but their attitudes are completely different. If a model only counts word occurrences without considering position, negation, and semantic composition, it may produce the wrong conclusion.
Word order matters as well. “The customer canceled the order” and “The order canceled the customer” use similar words, but they do not have the same meaning or even the same semantic plausibility. Language is not an unordered collection of words. It is a sequence with ordering, compositional rules, and context dependencies.
The same issue appears outside language. Determining whether a device is currently malfunctioning may require earlier sensor readings. Deciding whether an operation is suspicious may depend on a series of previous actions. Recognizing the current speech segment may require neighboring audio signals.
These examples point to a common requirement: a model must process the current input while also using historical information.
A Recurrent Neural Network (RNN) is a classic model for this purpose. It is not a prerequisite for every language-understanding system, nor is it the only approach to sequence modeling. However, its simple recurrent structure clearly demonstrates how historical information can be incorporated into the current computation.
All technical abbreviations in this article are written in full when they first appear. The abbreviations are used afterward.
2. The Core Idea of an RNN: Bringing History into the Current Computation
2.1 From Independent Computation to Recurrent Computation
A standard feed-forward structure can map an input to an output. However, if every position in a sequence is processed independently without additional positional or contextual information, the model cannot naturally represent dependencies between earlier and later positions.
An RNN introduces a hidden state. At time step (t), the model receives the current input (xt) and the hidden state from the previous time step, (h{t-1}), and produces a new hidden state:
h_t = tanh(W_x x_t + W_h h_{t-1} + b_h)
If the model must produce an output at the current time step, it can compute:
z_t = W_y h_t + b_y
Where:
- (x_t) is the current input vector;
- (h_t) is the representation of historical information up to the current position;
- (W_x) is the weight matrix from the input to the hidden state;
- (W_h) is the recurrent weight matrix between hidden states;
- (W_y) is the weight matrix from the hidden state to the output;
- (b_h) and (b_y) are bias terms;
tanhis an element-wise nonlinear function.
The meaning of (z_t) depends on the task. A classification task can convert it into class probabilities, while a regression task can apply an appropriate output mapping to obtain a prediction.
2.2 A Hidden State Is Not a Verbatim Copy of the Input
A hidden state can be viewed as a continuously updated summary. After reading new content, the model combines the current information with the previous summary to produce the next one.
This analogy has clear limits. A hidden state is not a summary that humans can directly read, and it does not preserve every historical word in full. It is a numerical representation shaped by the training objective. Because it normally has a fixed dimensionality, its information capacity is limited.
For example, when processing “The service responds quickly, but the results are inaccurate,” the model may need to preserve the evaluation in the first clause and adjust the overall judgment after recognizing the contrast and the second clause. Whether the model has actually learned this relationship must be verified through data and testing. It cannot be inferred from the architecture alone.
2.3 What Parameter Sharing Means
The same RNN layer reuses the same parameters at every time step. As a result, increasing the sequence length does not increase the layer’s parameter count in proportion to the number of time steps.
However, a constant parameter count does not mean constant computational cost. Longer sequences require more recurrent steps, and training usually requires storing more intermediate states.
3. How RNNs Are Used for Different Tasks
3.1 Sequence to a Single Result
Sentiment classification, behavioral risk detection, and device-state recognition can all map an entire sequence to one result.
A common approach is to use the hidden state at the last valid time step as the representation of the whole sequence, then pass it to a classification or regression module. Another option is to aggregate the states from multiple valid positions.
The word “valid” is important. When placeholder elements are added to make sequences in a batch the same length, the final padded position must not be treated as the true end of the sequence.
3.2 Sequence to Per-Position Results
Some tasks require an output at every position, such as assigning a label to each word in a text or predicting device status at every time point.
In this case, an output layer can be connected to every (h_t). The training loss is usually summed or averaged over valid positions, while padded positions must be excluded.
3.3 Sequence to Another Sequence
When the input and output have different lengths, one recurrent structure can encode the input and another can generate the output step by step.
This approach separates encoding from generation. However, if the entire input must be passed through a single fixed-size vector, long sequences can create an information bottleneck. An attention mechanism can allow the generation process to access multiple input positions as needed, which helps reduce this limitation.
4. How RNNs Learn: Training Through Time
An RNN appears to contain a loop in its structural diagram. When processing a finite sequence, however, it can be unrolled over time into a computation chain:
x_1 → h_1 → h_2 → … → h_T
Each hidden state also receives the input from its corresponding time step.
During training, the model first performs a forward pass. It then updates its parameters according to the loss between its predictions and the target values. Because later hidden states depend on earlier hidden states, gradients must also be propagated backward through time.
This process is called Backpropagation Through Time (BPTT).
4.1 Why Long-Range Dependencies Are Difficult to Learn
When the loss must influence a state far back in the sequence, the gradient passes through many recurrent steps. Each step involves local derivatives and matrix multiplication. Repeated multiplication can cause the gradient to shrink rapidly or grow excessively.
Therefore, a theoretical connection from the past to the present does not guarantee that training can effectively use information from the distant past.
4.2 Vanishing Gradients
With vanishing gradients, the training signal reaching early positions becomes extremely weak. The model then struggles to learn dependencies across many time steps.
For example, a key condition may appear at the beginning of a long document while the final prediction depends on it. A standard RNN may rely more heavily on local clues near the end and overlook the distant information.
4.3 Exploding Gradients
Exploding gradients can cause excessively large parameter updates. Typical symptoms include sharply fluctuating loss, numerical overflow, and unstable training.
Common responses include gradient clipping, appropriate initialization, learning-rate adjustment, and checking the numerical range of the inputs. Gradient clipping can limit excessive gradients, but it does not directly solve the problem of insufficient long-term memory.
4.4 The Trade-Off of Truncated Training
For very long continuous sequences, truncated BPTT can divide the sequence into segments and limit the time range through which gradients are propagated during each update.
This reduces training cost, but it also shortens the direct training signal across segment boundaries. Whether hidden states should be carried between segments depends on whether the samples are continuous, whether the task permits this behavior, and how the data boundaries are defined.
5. How Gated Structures Improve Memory Handling
A standard RNN directly mixes historical information with the current input. It has no explicit mechanism for controlling what should be retained or updated. Gated structures introduce learnable controls to improve this process.
5.1 Long Short-Term Memory
Long Short-Term Memory (LSTM) networks introduce a cell state and use input, forget, and output gates to control information flow.
The basic idea is to:
- decide how much old information to retain;
- decide how much new information to write;
- decide how much internal state to expose externally.
The additive update path of the cell state helps reduce vanishing-gradient problems in standard RNNs. However, an LSTM does not have unlimited memory, nor can it guarantee stable performance on sequences of arbitrary length.
5.2 Gated Recurrent Unit
A Gated Recurrent Unit (GRU) uses update and reset gates to regulate the combination of historical information and a candidate state. Unlike a typical LSTM, it does not maintain a separate cell state.
With the same input and hidden dimensions, a standard GRU usually has fewer parameters than a standard LSTM. In practice, however, speed and accuracy also depend on the implementation, hardware, sequence length, and task characteristics. It is not safe to claim that one structure is always better.
5.3 Boundaries of Bidirectional Structures
A bidirectional structure processes the input in both forward and backward directions, then combines the two states. For complete-text classification and offline sequence labeling, it can use information from both sides of the current position.
Real-time prediction requires caution. If future inputs are unavailable at prediction time, the model must not use them during training or evaluation. Otherwise, information leakage will produce results that cannot be reproduced in production.
6. From Raw Data to Model Input
6.1 Text Must First Be Converted into Numerical Representations
Text usually needs to be tokenized, mapped to indices, and converted into vectors before it can enter a model.
The tokenization unit may be a character, word, or smaller subword segment. This choice affects sequence length, vocabulary size, and the handling of unknown expressions. An embedding layer maps discrete indices to continuous vectors, which can be trained together with the task.
Tokenization should not be treated as an unimportant preprocessing detail. Under different tokenization granularities, the dependency spans that the model must learn can change substantially.
6.2 Padding and Masking Must Be Used Together
Batch training usually requires all samples in a batch to have the same tensor length, so shorter sequences must be padded.
Padding positions are not real content. Their influence should be excluded using valid lengths or masks. In particular, check whether:
- per-position losses ignore padding positions;
- aggregation operations process only valid states;
- the final state corresponds to the true end of the sequence;
- results remain consistent across samples with different lengths.
Simply setting padded inputs to zero does not always solve the problem, because the hidden state may still change during subsequent recurrent steps.
6.3 Time-Series Data Requires Attention to Time Itself
When processing sensor readings or business metrics, each input may contain several numerical features. Common preprocessing steps include scale normalization, missing-value indicators, and time-interval encoding.
Normalization statistics should be estimated only from the training set and then fixed for validation and testing. Estimating them from the full dataset can leak information from the future.
For irregularly sampled data, adjacent positions should not automatically be assumed to have equal time intervals. The time difference can be added explicitly, or a modeling approach designed for the specific setting can be used.
7. A Tool-Agnostic Training Example
The following pseudocode describes a sequence-to-single-classification workflow:
Initialize model parametersFor each training batch:Read inputs, true labels, and valid lengthsInitialize hidden states to zeroFor each time position:Compute candidate hidden statesUpdate only the states of samples that are still valid at this positionUse the final valid states to compute classification scoresCompute the loss from the scores and true labelsClear gradients from the previous batchCompute current gradients through backpropagationApply gradient clipping when neededUpdate parameters
Two engineering details are easy to overlook.
First, hidden states should normally be reset between unrelated samples. Passing the state from one user request to an unrelated request introduces an incorrect context dependency and may mix information across sessions.
Second, hidden states should be carried across batches only when the data is genuinely continuous, the ordering is reliable, and the training design explicitly allows it. The computation-graph boundary must also be controlled correctly to avoid unintentionally retaining an excessively long backpropagation chain.
8. Evaluation Should Not Rely on a Single Overall Score
8.1 Data Splits Must Match Real Usage
Time-series forecasting should generally split data according to time order so that the training process does not see future information.
For text tasks, inspect duplicate content, shared templates, and highly similar samples from the same entity. If such samples are distributed across the training and test sets, evaluation metrics may be artificially inflated.
8.2 Metrics Should Match the Business Objective
In addition to accuracy, classification tasks can use precision, recall, and their harmonic mean, the F1 score. With imbalanced classes, overall accuracy may hide failures on minority classes.
Regression and forecasting tasks should be evaluated using error magnitude, sensitivity to anomalies, and performance across different time horizons. Metrics should support real decisions rather than simply favoring the easiest number to optimize.
8.3 Break Down Testing by Sequence Length and Semantic Phenomenon
At a minimum, evaluate short, medium-length, and long sequences separately.
For text, it is also useful to create focused test samples involving negation, contrast, long-range dependencies, and changes in word order. These tests help determine whether the model is genuinely using context instead of relying on a few frequent words or accidental correlations.
9. Managing State and Cost in Production
Because RNNs process inputs recurrently over time, they are suitable for data that arrives incrementally. In an online setting, a service can preserve the current session state and update it whenever new input arrives, rather than recomputing the entire history each time.
However, stateful services introduce additional management complexity:
- state ownership must be clearly associated with a session or device;
- inputs from the same sequence must remain in the correct order;
- timeouts, reconnections, and state resets must be handled;
- states must not be incorrectly reused between users;
- a strategy is needed for migrating or clearing state after a model upgrade.
Within a single sequence, time steps depend on one another, which limits parallel computation. Deployment cost should therefore not be judged only by parameter count. Measure response time, throughput, and state-storage overhead under realistic sequence lengths.
10. How to Decide Whether to Use an RNN
RNNs and their gated variants remain worth evaluating when a task emphasizes streaming input, incremental updates, and a relatively compact representation of historical context.
If a task depends on interactions across very long ranges or requires strong sequence-level parallelism during training, other sequence-modeling approaches, including attention mechanisms, should also be considered.
A practical selection process should begin with a simple baseline and add complexity gradually. If a model without explicit temporal structure already meets the target, verify that the benefits of a more complex model justify its additional training, deployment, and maintenance costs.
The most important lesson of RNNs is not that a sentence cannot be understood without them. It is that they reveal the basic logic of sequence modeling: the meaning of the current input often depends on what happened earlier.
By understanding hidden states, recurrent computation, gradient propagation, and the limits of information compression, developers can more accurately analyze why a model works, why it fails, and whether the next improvement should target the data, the training process, or the model architecture.
