Large Language Models Explained: From Tokenization to Text Generation
Learn how large language models transform text into tokens and vectors, use positional information and Transformer attention to model context, train through next-token prediction, and generate responses step by step. This guide also explains alignment, hallucinations, inference parameters, and the practical limits of LLMs.
Large language models, commonly called LLMs, can chat, write, translate, code, and answer questions. At the most fundamental level, however, their core task is straightforward:
Given the text that has already appeared, predict what is most likely to come next.
This simple description involves several stages, including data representation, neural networks, context modeling, probability calculation, and generation strategies. Understanding these stages provides a more accurate mental model of how LLMs work.
1. What Does a Language Model Actually Do?
Consider the following input:
The weather today is very
The model must estimate what could appear next. Possible candidates might include “good,” “pleasant,” “hot,” or “cold,” and each candidate receives a probability.
The internal distribution might look something like this:
good 0.42pleasant 0.25hot 0.12cold 0.08terrible 0.03other 0.10
The model does not simply retrieve a fixed answer from a database. Instead, it calculates a probability distribution for the next token based on the current context and then selects a result according to a generation strategy.
After generating “good,” the model adds the new text to the context and predicts the next token. A complete response can therefore be viewed as a sequence of repeated predictions:
Existing context↓Predict the next token↓Append the result to the context↓Predict the next token again↓Repeat until generation ends
This process is called autoregressive generation. In simple terms, later content depends on content that has already been generated.
However, predicting the next token is not merely a basic word-association game. To make reasonable predictions across different topics and writing styles, the model must learn relationships between words, sentence structures, connections between concepts, and patterns associated with different tasks.
2. Step One: Converting Text into Numbers
Computers cannot directly process concepts such as “weather,” “program,” or “database” as language. Before text enters the model, it is usually split into tokens and encoded as numbers.
What Is a Token?
A token is a basic unit used by a model to process text. It does not necessarily represent one character, one Chinese character, or one complete English word. The exact segmentation depends on the tokenizer.
For example, the sentence:
I enjoy learning artificial intelligence
might be split into:
I / enjoy / learning / artificial / intelligence
It could also be divided into smaller pieces:
I / en / joy / learn / ing / arti / ficial / intelli / gence
English words, numbers, punctuation marks, and source code are also divided into corresponding pieces. Subword tokenization generally balances vocabulary size and expressive power: common words can be represented as complete units, while rare words can be decomposed into more frequent pieces.
Token IDs
After tokenization, each token is mapped to an integer ID:
Text: I like programmingIDs: [314, 52, 871]
These IDs have no inherent meaning. ID 314 does not naturally mean “I,” and ID 52 does not inherently mean “like.” They are simply indexes in the model’s vocabulary.
The semantic information is carried by the vector representation that comes next.
Embeddings: Turning IDs into Vectors
The model maps each token ID to a vector of floating-point numbers:
Token: likeVector: [0.18, -0.42, 0.76, ..., 0.09]
This is commonly called a token embedding. It can be viewed as a set of coordinates inside the model’s internal representation space. Words with similar meanings, usage patterns, or contextual relationships often appear relatively close to one another in this space.
For example, “car” and “vehicle” may be more closely related than “car” and “apple.” The word “run” may also receive different contextual representations in a software context and in an everyday conversation.
Modern models do not rely only on one fixed vector for each token. After entering the network, a token’s representation is combined with its position and surrounding context, producing increasingly rich representations across multiple layers.
3. Positional Information: How Does the Model Know Word Order?
If tokens were converted only into vectors, the model could not reliably distinguish between these two sentences:
The cat chases the dog.The dog chases the cat.
Both sentences contain the same words, but their meanings are different. The model must know where each token appears in the sequence.
For this reason, the input representation generally includes positional information in addition to token embeddings. Position information helps the model determine:
- Which token comes before or after another;
- How far apart two tokens are;
- Whether a token appears at the beginning, middle, or end of a sentence;
- How order, brackets, indentation, and other structures work in source code.
Positional information allows the model to interpret text as an ordered sequence rather than as an unordered collection of tokens.
4. The Core of the Transformer: Attention
The Transformer architecture is central to modern LLMs, and attention is one of its most important mechanisms.
What Problem Does Attention Solve?
Consider this sentence:
Alex put the book in the drawer because it was heavy.
The pronoun “it” may refer to the book, although the correct interpretation depends on context. The model must examine earlier words to determine the likely reference.
Similarly:
Although the code is short, it implements complex functionality.
Here, “it” is more likely to refer to the code than to the functionality. The model must dynamically evaluate relationships between words.
Attention can be summarized as follows:
- Treat the current token as a query;
- Examine other tokens in the context;
- Calculate how strongly they are related to the current token;
- Aggregate their information using weighted values;
- Produce a new representation that includes relevant context.
Query, Key, and Value
Attention calculations usually transform input representations into three types of vectors:
- Query: What information the current token is looking for;
- Key: What clues each contextual token can provide;
- Value: The actual information carried by each contextual token.
This can be compared to searching through documents:
- The Query is the search question;
- The Key is a document title or index;
- The Value is the content of the document.
The better a Query matches a Key, the greater the weight usually assigned to the corresponding Value. After weighted aggregation, the current token can absorb the contextual information most relevant to it.
Why Is Multi-Head Attention Needed?
A single attention mechanism might capture only one type of relationship, while language contains many kinds of relationships, including:
- Subject–verb relationships;
- Pronoun–reference relationships;
- Modifiers and the nouns they describe;
- Variable definitions and uses in code;
- Long-range dependencies;
- Relationships involving tone, style, and topic.
Multi-head attention runs several attention operations in parallel, allowing different heads to focus on different types of relationships. The resulting information is then combined into a more complete contextual representation.
5. What Happens Inside the Model?
A typical generation process can be simplified into the following stages.
Step 1: Tokenization
The input text is split into tokens and converted into token IDs.
Step 2: Vectorization
The token IDs are mapped to vectors, and positional information is added.
Step 3: Processing Through Multiple Transformer Layers
The vectors pass through many network layers. Each layer generally includes an attention module and a feed-forward network:
- The attention module combines contextual information;
- The feed-forward network applies nonlinear transformations at each position;
- Residual connections help information move reliably through deep networks;
- Normalization improves training stability.
As the number of layers increases, representations gradually move beyond surface-level token information toward syntax, semantic relationships, task patterns, and more abstract features.
Step 4: Producing Probabilities
The final-layer representation is mapped to the model’s entire vocabulary, producing a score for every candidate token. After normalization, these scores become a probability distribution for the next token.
Step 5: Selecting a Token and Continuing
The model selects a token according to the generation strategy, appends it to the existing text, and repeats the process.
6. How Are LLMs Trained?
The basic training of an LLM generally uses self-supervised learning. Instead of manually labeling every sentence with a correct answer, the model creates training targets directly from raw text.
For example, given:
Artificial intelligence is changing
the model may be trained to predict the next token, such as “the world” or “the way we work.” During training, the model compares its predicted distribution with the actual next token and adjusts its internal parameters through backpropagation.
The process is repeated many times:
- Provide large amounts of text;
- Sample or organize training sequences;
- Predict the next token at each position;
- Calculate the difference between the prediction and the actual result;
- Update the parameters using gradient descent;
- Repeat over many training iterations.
An LLM may contain a very large number of parameters. These parameters are not a collection of individually stored facts. They are numerical relationships formed during training. Through these relationships, the model compresses and represents statistical patterns, structural regularities, and connections between concepts found in language.
7. Why Can Next-Token Prediction Produce Many Capabilities?
Predicting the next token may seem like a narrow objective, but successful prediction requires many underlying abilities.
To continue a program correctly, the model must learn syntax, variable relationships, and common algorithmic patterns. To translate text, it must learn correspondences between languages. To answer a question, it must identify the question type and organize relevant information from the context. To summarize content, it must determine which details are important and which can be omitted.
As a result, capabilities such as translation, summarization, and coding do not necessarily exist as separate components called “translation modules” or “summary modules.” They can gradually emerge under a unified prediction objective.
When model size, training data, and computational resources reach a sufficient scale, many local patterns can combine into more complex capabilities. This phenomenon is often described as emergence. It does not mean that the model suddenly develops a mysterious form of consciousness. Rather, large-scale learning allows many simple patterns to combine into stronger overall behavior.
8. Why Is Alignment Needed After Training?
A model trained only on raw text may be good at language continuation but not necessarily good at following user instructions. It might produce poorly organized answers, fail to address the question, or repeat inappropriate content.
Practical models therefore typically undergo additional alignment training so that their behavior better matches human expectations. Common goals include:
- Understanding instructions more accurately;
- Organizing answers more clearly;
- Following requested formats;
- Expressing uncertainty when information is insufficient;
- Reducing harmful, illegal, or dangerous content;
- Maintaining a consistent tone or role across a conversation.
Alignment does not change the model’s fundamental mechanism of predicting the next token from context. Instead, additional training teaches the model to choose more appropriate expressions in interactive settings.
9. Why Do Models Sometimes Hallucinate?
An LLM generates text that is probable in context. It is not automatically connected to a verified factual database or a retrieval system. As a result, it can produce content that is grammatically correct and logically fluent but factually inaccurate.
Common causes include:
Errors or conflicts in the training data
The source text learned by the model may itself be inaccurate or contradictory.Questions beyond the model’s knowledge
If relevant information is missing, the model may still attempt to generate a complete-looking answer.Weakening of information in long contexts
When the input is very long, some details may receive insufficient attention.Probability generation is not fact verification
The model is better at generating text that resembles an answer than at automatically proving that the answer is correct.Ambiguous questions
When a user’s wording is unclear, the model may fill in unstated assumptions on its own.
In high-risk areas such as healthcare, law, finance, and engineering configuration, model output should therefore be checked against authoritative sources, retrieval systems, programmatic validation, and human review.
10. Do LLMs Truly Understand?
The meaning of “understanding” depends on the level being discussed.
From an engineering perspective, an LLM can establish contextual relationships, identify language structures, perform multi-step tasks, and generate coherent results according to instructions. In this functional sense, it demonstrates a strong form of understanding.
From a stricter cognitive perspective, however, it does not necessarily possess human feelings, consciousness, real-world experience, or a stable world model. The model performs calculations using its parameters and current context, then generates text that best fits the conditions.
A more precise description is:
An LLM develops powerful language-modeling and task-execution capabilities by learning statistical patterns and structural relationships from large amounts of text. Its output can resemble human understanding, but its internal mechanism is not equivalent to human consciousness or experience.
11. Important Inference-Time Parameters
Generation is also influenced by several decoding strategies.
Temperature
Temperature controls how concentrated the probability distribution is. With a lower temperature, the model is more likely to select high-probability results, producing more predictable output. With a higher temperature, the probability distribution becomes more dispersed, which can produce more varied language but may also increase deviation from the most likely answer.
Top-K and Top-P
These methods restrict the candidate set and select tokens only from a portion of the highest-probability options. They can reduce abrupt results caused by extremely unlikely tokens while preserving some randomness.
Maximum Generation Length
This determines how many tokens the model can generate at most. If the limit is too short, the response may be incomplete. If it is too long, costs increase and the risk of repetition or topic drift may also rise.
These parameters do not change what the model learned. They affect how the model selects tokens from the probability distribution during generation.
12. A Complete Mental Model
The overall process can be summarized as follows:
Raw text↓Tokenization and token IDs↓Embeddings and positional information↓Multiple Transformer layers↓Contextual representation↓Probability distribution for the next token↓Sampling or selection↓Iterative generation of a complete response
During training, the model repeatedly predicts the correct next token and learns language, knowledge, and task patterns. During inference, it generates output step by step based on the input context. During alignment, it learns to respond in ways that better match human expectations.
At its core, an LLM is neither a simple search engine nor a knowledge base containing every answer in advance. It is better understood as a large-scale probabilistic generation system. It processes context through layers of neural networks, calculates the likelihood of different possible continuations, and combines these local predictions into coherent long-form text.
This perspective makes both the strengths and limitations of LLMs easier to understand. They can demonstrate combined capabilities in translation, writing, programming, summarization, and reasoning, but reliable use in real-world applications still requires fact checking, contextual constraints, and supporting engineering systems.
