LLM Core Principles: Tokenization, Sampling, and Alignment Explained
This guide explains how large language models (LLMs) work under the hood, from tokenization and embeddings to Transformer-based context modeling, autoregressive generation, and sampling strategies. It covers why sampling goes beyond greedy decoding, how temperature, Top-K, and Top-P shape model output, and why pre-training and alignment are both essential. A FAQ section and a practical deployment checklist help developers use LLMs reliably.
1. Introduction: What Large Language Models Actually Do
When people think of large language models (LLMs), they usually picture capabilities like conversation, writing, code generation, and question answering. But at the core mechanism level, the most fundamental thing an LLM does is actually quite simple: given the existing content, predict the next most reasonable fragment.
This “fragment” is usually a Token. Rather than searching existing documents for answers the way a search engine does, a model computes the probability distribution of the next Token based on the language patterns, world knowledge, and contextual clues it learned during training. It then selects one Token and appends it to the context. This process repeats, eventually producing a complete answer.
Understanding this chain of steps helps explain why LLMs “continue writing,” why they hallucinate, why outputs differ significantly across different parameters, and why both pre-training and alignment are indispensable.
2. Tokenization: The Model’s View of Text
Large models cannot process raw strings directly, so the first step is usually tokenization.
A Token is the smallest unit of text that a model processes. It is not necessarily a complete word — it may be a subword, a punctuation mark, a space, a number, or even a Chinese character. For example:
Word: unkillableTokenization: un / kill / able
A long, rare, or compound word may be split into affixes, roots, or byte fragments. There are three main benefits to doing this:
- Controlling vocabulary size: There is no need to index every possible word, which lowers storage and computation costs.
- Improving generalization: When encountering out-of-vocabulary words, the model can represent them by combining known subwords.
- Supporting multiple languages: Different languages can be handled through a unified subword mechanism.
For Chinese text, common segmentation approaches include:
| Segmentation method | Description | Characteristics |
|---|---|---|
| Character-level | Split by individual Chinese character or symbol | Small vocabulary, longer sequences, fine-grained |
| Word-level | Split by word or phrase | More complete semantics, but relies on a dictionary or tokenizer |
| BPE / WordPiece | Learn common fragments through subword merging | Balances vocabulary size and generalization, widely used |
The core idea behind both BPE and WordPiece is to start from common fragments and gradually learn high-frequency combinations. For Chinese, different models may adopt character-level, word-level, or mixed segmentation. For mixed-language text — such as Chinese-English mixed content, code identifiers, rare names, or product names — the text may be split into multiple Tokens. This affects not only comprehension but also context length, inference cost, and generation latency.
3. Embedding and Transformer: Turning Tokens into Computable Representations
After tokenization, the text becomes a sequence of Token IDs, for example:
Token ID sequence: [273, 8891, 1502, ...]
But a Token ID is just an index, and the model cannot perform semantic computation on it directly. So it must be mapped to a dense vector through an Embedding (embedding layer):
Token ID sequence: [273, 8891, 1502, ...]↓ Embedding lookup mappingVector representation: [[0.12, -0.4, 0.7, ...],[0.33, 0.9, -0.2, ...],...]
These vectors are the basic representation that a neural network can process. Typically, semantically similar words are also closer in the vector space. For example, there may be some semantic similarity between “king” and “queen,” or between “doctor” and “nurse.”
However, note that: the semantic relationships and context modeling after vectorization depend on the Transformer / self-attention. The model performs context modeling and prediction in the vector space, rather than merely mapping each word to a vector in isolation.
The same word can carry entirely different meanings in different contexts. For example, “bank” refers to a financial institution in a financial context but a riverbank in a river context; “apple” may refer to a company in a tech context but a fruit in a food context. What truly lets the model distinguish these meanings is the self-attention mechanism in the Transformer. It makes the current Token attend to other Tokens in the context and aggregate information according to attention weights, thereby forming a representation that is relevant to the current context.
In addition, the Transformer usually needs mechanisms such as positional encoding or rotary positional encoding, because self-attention itself does not directly perceive word order. Overall, Embedding provides the basic semantic entry point, while the Transformer handles context modeling; together they support the model’s prediction capability.
4. Word Chain: Probabilistic Prediction and Autoregressive Generation
The core process by which a large model generates text can be understood as a game of “word chain.”
Given the current context, the model outputs a probability distribution over the entire vocabulary. This distribution indicates “what each next Token could be, and how likely each one is.” The process is as follows:
- Input the current context, i.e., the Token sequence already generated or provided by the user;
- The model computes the logits for the next Token;
- Functions such as softmax convert the logits into a probability distribution;
- Based on the probability distribution and the sampling strategy, select the next Token;
- Append the new Token to the context;
- Repeat the above until a stopping condition is met.
This mechanism is called autoregressive generation. Its key feature is that every step depends on what has already been generated, and subsequent Tokens are influenced by all preceding Tokens.
In engineering, this mechanism also introduces several practical constraints:
- Limited context length: The number of Tokens a model can handle is bounded by the context window.
- Generation cost grows with length: The longer the output, the greater the computation and memory pressure.
- Inference needs caching optimizations: A common approach is KV Cache, used to reduce repeated computation.
- Batching affects throughput: In online serving, batching is often used to improve GPU utilization, but it adds scheduling complexity.
Therefore, LLM generation is not only an “algorithmic problem” but also an inference-engineering and performance-optimization problem.
5. Sampling and Decoding: Why Not Just Take the Highest Probability
Many people assume that the model picks the highest-probability Token at every step. In reality, if greedy decoding is always used — always selecting the most probable candidate — the output tends to become monotonous and rigid, and can even fall into repetitive loops. For example:
I thinkI thinkI thinkI think
Real systems usually introduce random sampling, making a stochastic choice over the probability distribution to enhance diversity and naturalness.
Common sampling strategies are as follows:
| Strategy | Effect | Common understanding | Risk |
|---|---|---|---|
| Temperature | Adjusts how steep the probability distribution is | Higher temperature is more divergent, lower is more conservative | Too high may cause loss of semantic control |
| Top-K | Sample only from the K highest-probability candidates | Fixed number of candidates | Too small may lose valid expressions |
| Top-P | Sample from the smallest candidate set whose cumulative probability reaches P | Dynamic candidate set | Too low may be overly narrow |
| Repetition penalty | Lower the probability of previously appearing Tokens being chosen again | Mitigates repetition | Too strong may damage semantics |
Temperature can be understood as a scaling factor applied to the logits. When the temperature is higher, the probability distribution is flatter and the output is more creative; when the temperature is lower, the distribution is sharper and the output is more deterministic.
Top-K keeps only a fixed number of the top K candidates. Top-P is more dynamic: it keeps the smallest candidate set whose cumulative probability reaches the threshold P. For example, top_p=0.9 means sampling from the candidates that cumulatively cover 90% of the probability.
The pseudocode below is provided only to illustrate the logic:
# Pseudocode: temperature + Top-K + Top-P sampling# Production implementations also need to handle NaN, EOS, batch dimensions, dtype, and other casesdef sample_next_token(logits, temperature=0.8, top_k=None, top_p=0.9):if temperature <= 0:raise ValueError("temperature must be greater than 0; use greedy decoding for deterministic output")if top_p is not None and not (0 < top_p <= 1):raise ValueError("top_p must be within (0, 1]")scaled_logits = logits / temperatureprobs = softmax(scaled_logits)if top_k is not None:probs = filter_top_k(probs, top_k)sorted_idx, sorted_probs = sort_desc(probs)if top_p is not None:cumulative = cumsum(sorted_probs)cut = len(sorted_probs)for i, value in enumerate(cumulative):if value >= top_p:cut = i + 1 # the first index that meets the threshold, plus onebreaksorted_idx = sorted_idx[:cut]sorted_probs = sorted_probs[:cut]total = sum(sorted_probs)if total <= 0 or not isfinite(total):return int(np.argmax(logits)) # fall back when the candidate set is empty or the values are abnormalnormalized = sorted_probs / totalchosen_idx = np.random.choice(sorted_idx, p=normalized)return int(chosen_idx)
Note that temperature should be greater than 0. Some systems use temperature=0 to indicate greedy decoding, but from a mathematical implementation standpoint, it is generally recommended to go directly through the deterministic decoding path to avoid division by zero. top_p is usually set between 0 and 1, with common values such as 0.8 to 0.95. If the candidate set is empty or the probabilities are abnormal, you should fall back to a deterministic strategy, such as selecting the Token with the largest original logits.
Precisely because sampling retains some randomness, the same prompt can produce different answers at different times. This is one reason LLMs feel “alive,” but in business scenarios that require stable reproducibility, you can reduce the variability by fixing the random seed, lowering the temperature, or constraining decoding.
6. Pre-training: The First-Stage “General Education”
The first stage of a large language model is usually pre-training.
The core task of pre-training is still “next-Token prediction,” but the training data is enormous, covering many kinds of text such as web pages, books, code, papers, and conversations. By continuously predicting the next Token, the model learns language structure, factual knowledge, reasoning patterns, and code regularities.
The pre-training stage mainly focuses on two dimensions:
- Model parameter size: More parameters generally mean stronger expressive power, but also higher training and inference costs.
- Data size and quality: The quantity, diversity, and degree of cleaning of the data all affect model performance.
It is worth emphasizing that pre-training is not simply “the more data, the better.” A high-quality data pipeline usually also needs:
- Data cleaning: Remove noise, garbled text, ads, and duplicate content;
- Deduplication: Avoid overly strong memorization caused by high-frequency repetition;
- Compliance filtering: Exclude illegal content, privacy leaks, and harmful material;
- Safety filtering: Reduce the impact of high-risk content in the training data on model behavior;
- Quality mixing: Balance data types such as code, math, natural language, and multiple languages.
Pre-training gives the model a “general education,” letting it master a large amount of knowledge and language patterns. But it does not naturally make the model “obedient.” A model that has only been pre-trained may:
- fail to follow complex instructions;
- answer off-topic;
- output harmful content;
- not know when it should refuse to answer;
- struggle to produce output in a fixed format.
So pre-training determines the model’s upper bound of capability, but a subsequent alignment stage is still needed to make the model more consistent with human usage habits and safety requirements.
7. Alignment: Making the Model “Useful and Safe”
After pre-training, an alignment stage is usually still required.
The goal of alignment is to turn the model from “being able to predict” into being “useful, controllable, and safe.” Common methods include:
| Method | Core idea | Effect |
|---|---|---|
| SFT | Supervised fine-tuning with high-quality instruction–answer samples | Teaches the model to follow instructions and conversational formats |
| RLHF | Train a reward model from human feedback, then apply reinforcement learning | Optimizes usefulness, naturalness, and safety |
| DPO | Directly optimize model outputs based on preference data | Relatively simple to implement; commonly used for preference alignment |
| Constitutional AI | Improve outputs through principles and model self-correction | Emphasizes a rule-based, interpretable alignment approach |
| Safety alignment | Specialized training for harmful, unauthorized, and non-compliant scenarios | Reduces the probability of high-risk outputs |
Pre-training and alignment can be summarized as:
- Pre-training handles “learning knowledge”: it determines what the model knows and what it can reason about;
- Alignment handles “learning to behave”: it determines how the model expresses itself, whether it is obedient, and whether it is safe.
However, alignment cannot guarantee zero risk. Even after SFT, RLHF, or safety training, a model may still produce errors, omissions, or inappropriate answers in edge cases. For this reason, production systems usually also need evaluation, red-teaming, content filtering, access control, and human review mechanisms.
8. Frequently Asked Questions
1. Why do LLMs repeat themselves?
Common causes include greedy decoding, too low a temperature, overly long context, duplicate training data, and insufficient repetition penalty. You can try raising the temperature, enabling top-p, adding a repetition penalty, shortening the context, or adjusting the stopping condition.
2. How do I choose the temperature?
If you prioritize factual accuracy, code generation, or structured output, it is recommended to use a low temperature, such as 0.1 to 0.3, or even greedy decoding; for copywriting or brainstorming, you can raise it to around 0.7 to 1.0; too high a temperature increases the risk of nonsense and broken formatting.
3. What is the difference between Top-P and Top-K?
Top-K keeps a fixed number of the K highest-probability candidates; Top-P dynamically keeps the smallest candidate set whose cumulative probability reaches the threshold P. Generally, Top-P is more flexible and Top-K is more intuitive. In engineering, you can use just one, or combine them — for example, top-k=40 first, then top-p=0.9.
4. Why does the same prompt produce different outputs each time?
Because randomness exists in the sampling stage. If you need stable reproducibility, you can lower the temperature, fix the random seed, restrict the candidate space, or add caching and result constraints at the business layer.
5. Is longer context always better?
No. Longer context increases memory, latency, and cost, and may also dilute key information. In practice, you should choose an appropriate length based on the task and improve the density of effective information through approaches such as summarization, retrieval augmentation, and placing key information up front.
9. Summary: A Learning Path from Beginner to Advanced
The core chain of a large language model can be summarized in six steps:
- Tokenization: Split text into Tokens to obtain discrete indices;
- Vectorization: Map the indices to computable vectors via Embedding;
- Context modeling: Understand context through the Transformer and self-attention;
- Probabilistic prediction: Generate Token by Token in an autoregressive manner, outputting a probability distribution;
- Sampling and decoding: Balance determinism and diversity through strategies such as temperature, Top-K, and Top-P;
- Pre-training and alignment: Pre-training learns knowledge, while alignment improves usefulness and safety.
If you want to use or deploy LLMs in a real system, it is recommended to focus on the following verifiable items:
- Tokenizer consistency: Training, inference, and the business layer must use the same vocabulary and chat template;
- Sampling parameters: Clearly specify temperature, top_k, top_p, repetition penalty, maximum generation length, and stop words;
- Inference resources: Assess GPU memory, KV Cache, batch size, throughput, and latency;
- Cost control: Pay attention to Token consumption, context length, retry strategy, and cache hit rate;
- Monitoring metrics: Track timeout rate, error rate, refusal rate, generation length, user feedback, and safety hit rate;
- Quality evaluation: Build evaluation sets for factual accuracy, instruction following, format stability, hallucination rate, and safety.
Master this main line, then dig deeper step by step into each stage — tokenization, sampling, context, pre-training, and alignment — to form a complete knowledge framework that takes you from beginner to advanced.
