Token and Embedding Explained: The Underlying Logic of LLM and RAG Data Processing
This article walks through the data pipeline that powers Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG): text to tokens to vectors to context. It explains why models never actually "read" text, how tokenizers convert strings into integer IDs through subword segmentation, how embeddings turn discrete IDs into continuous vectors, and how both mechanisms work together across the offline and online stages of a RAG system. You will also find practical guidance on token budgeting, chunking strategies, retrieval consistency, reranking, and a summary of common mistakes and engineering best practices that explain issues like unexpected context overflow, mismatched token counts, and poor retrieval quality.
Introduction: Why Models Don’t Actually “Read” Text
Large Language Models (LLMs) appear to understand and generate text fluently, but at the computational level, a model has no concept of language. The input it receives is a sequence of integers, and the output it produces is also a sequence of integers. Text is just the human-facing “shell” wrapped around those numbers. Two components bridge the gap between numbers and text: the tokenizer, which converts between text and IDs, and the embedding, which gives those IDs semantic meaning and enables the semantic retrieval that Retrieval-Augmented Generation (RAG) depends on.
Many developers run into the same confusions when building LLM applications:
- Why does a document under a thousand words trigger an “exceeds context length” error?
- Why do token counts from two different tools differ so much?
- Why can’t a RAG system retrieve content that clearly exists in the knowledge base?
- Why does retrieval quality drop noticeably after swapping the embedding model?
Most of these answers trace back to a single data pipeline: text → token → vector → context. This article breaks that pipeline down layer by layer, helping you build a complete mental model in a short time.
1. Token: The First Encoding Step for Text
1.1 Two Intuitive Approaches: Character-Level and Word-Level
At the hardware level, computers store sequences of bytes, not the “characters” we see with our eyes. Take Chinese as an example: a common character usually takes 3 bytes after UTF-8 encoding. A single byte is just a discrete ID with a limited range of values and carries no semantics on its own. If a model treats bytes or characters as its smallest processing unit, it pays three costs at once:
- Overly long sequences: A few hundred Chinese characters often map to thousands of bytes, and the cost of self-attention grows quadratically with sequence length, so the cost of long text is amplified dramatically;
- No semantic units: After “artificial intelligence” is split into 4 characters or even 12 bytes, the model must spend extra parameters and compute to learn the “byte → character → word” composition rules from scratch, and these low-level patterns consume model capacity that could otherwise be used for higher-level semantic understanding;
- Hard to unify the vocabulary: Writing systems differ enormously across languages, so a character-level vocabulary is either incomplete or extremely sparse.
So is it feasible to use whole words as the unit? That approach has its own clear bottlenecks:
- Vocabulary explosion: A language’s vocabulary is nearly open-ended, with many spelling variants and proper nouns, making it impossible to enumerate exhaustively;
- Out-of-vocabulary (OOV) problem: Words outside the vocabulary can only be mapped to a generic “unknown word,” losing their specific meaning entirely;
- Difficult segmentation: In many languages, there are no natural delimiters between words when writing, so word boundaries themselves are ambiguous.
1.2 The Compromise: Subword Tokenization
The mainstream approach in the industry is subword tokenization: common words are kept whole as a single token, while rare words are split into smaller pieces. This balances vocabulary size against sequence length:
- Controllable vocabulary: The vocabulary typically keeps only tens of thousands of high-frequency pieces;
- Any text is encodable: Content not seen in advance can still be assembled from known pieces;
- Shared statistical patterns: The same root appearing in different words recurs repeatedly, so training signals can be shared. For example, if “unbelievable” is not directly in the vocabulary, it may be split into pieces such as “un,” “believ,” and “able.”
For this reason, token counts from different models can be either higher or lower than expected. The differences in token density are especially pronounced for content like Chinese, code, and JSON.
Definition of a token: A token is a piece (a character, word, or word fragment) cut from text by a tokenizer, along with its corresponding integer ID in the vocabulary.
1.3 Three Key Properties of Tokens
| Property | Meaning | Engineering value |
|---|---|---|
| Determinism | The same text and vocabulary always produce the same ID sequence | Reproducible results, useful for caching, counting, and troubleshooting |
| Reversibility | The ID sequence can be restored to text, except for OOV pieces | Verify the encoding pipeline and diagnose truncation or garbled output |
| Completeness | Any input can be cut into a finite number of tokens | Guarantees the system won’t fail because something “can’t be represented” |
1.4 Vocabulary: The Model’s “Dictionary”
Every LLM comes with a vocabulary, which is essentially a mapping table from “text pieces → integer IDs,” usually tens of thousands of entries in size. The vocabulary is typically tied to the model version: IDs are not interchangeable across models, and the token count for the same text can differ between models. Therefore, any logic involving token counting, length limits, and cost estimation must stay consistent with the model actually being called.
Sequences also include a few special tokens:
- Beginning-of-sequence (BOS) and end-of-sequence (EOS) markers;
- Padding (PAD) markers used for batch alignment;
- Unknown (UNK) markers for out-of-vocabulary content;
- Markers that separate multiple text segments.
These markers play a structural role inside the model but contribute nothing to the final text shown to the user.
2. Tokenizer: From String to ID Sequence
2.1 Two Phases: Training and Inference
A tokenizer’s work falls into two phases:
- Training phase: Statistics are gathered on adjacent character or byte co-occurrences in the corpus, and the most frequent adjacent pairs are merged repeatedly to build a “piece → ID” vocabulary, while freezing the priority order of each piece;
- Inference phase: The frozen vocabulary is applied to segment the input text, which passes through normalization (such as case folding and Unicode form unification), pre-tokenization (initial splitting by spaces and punctuation), and subword tokenization, after which each piece is looked up to produce the ID sequence.
Understanding this helps avoid a common misconception: a tokenizer does not “analyze” text on the fly each time. It simply looks things up according to fixed rules.
2.2 Example: Encoding and Decoding with a Simplified Tokenizer
The following code demonstrates the core flow using greedy longest-match. It is only meant to illustrate the principle and is not a real tokenization algorithm:
# Assume the vocabulary has been built through training; this maps pieces to IDsvocab = {"<unk>": 0, "人工": 1, "智能": 2, "技术": 3, "AI": 4, ",": 5}def segment(text):"""Starting from the current position, match the longest piece present in the vocabulary"""pieces, i = [], 0while i < len(text):for n in range(len(text) - i, 0, -1):if text[i:i + n] in vocab:pieces.append(text[i:i + n])i += nbreakelse:pieces.append("<unk>") # fallback for out-of-vocabulary contenti += 1return piecesdef encode(text):return [vocab[p] for p in segment(text)]def decode(ids):reverse = {i: p for p, i in vocab.items()}return "".join(reverse.get(i, "<unk>") for i in ids)text = "人工智能技术,AI"ids = encode(text)print(ids) # [1, 2, 3, 5, 4]; the actual result depends on the vocabulary contentsprint(decode(ids) == text) # True; encoding is reversible
The efficiency gained from tokenization is easy to see with a single English word:
- Character-level: Split into “u n b e l i e v a b l e,” 12 tokens in total;
- Word-level: If the word is not in the vocabulary, it can only be encoded as a single “unknown word,” and its meaning is completely lost;
- Subword-level: Split into 2–3 pieces such as “un,” “believ,” and “able,” preserving recognizable semantic cues while keeping the vocabulary size under control.
2.3 Engineering Notes
- Use matching pairs: The tokenizer must be paired with the target model. Token counts obtained by mixing different vocabularies are not comparable, which is one reason counts vary so widely between tools;
- Watch special content: Numbers, URLs, code, and emoji may be segmented differently than you’d expect. For example, in some models’ tokenizers, a long number may be split digit by digit, which places higher demands on precise numeric computation;
- Empty tokens should not be billed: In practice, text is usually preprocessed to trim leading and trailing whitespace and collapse consecutive whitespace. This avoids producing “whitespace-only” empty tokens and reduces meaningless token overhead.
3. Embedding: From Discrete IDs to Continuous Vectors
3.1 Why IDs Alone Aren’t Enough
An ID is essentially a discrete symbol. It can only express “same” or “different,” not “similar.” ID 3 and ID 4 being adjacent does not mean “artificial” and “intelligent” are semantically closer than ID 3 and ID 800. If a model relied only on IDs, it would have to memorize the co-occurrence patterns of each ID one by one from the training data, which limits generalization.
The core problem embeddings solve is this: to place tokens with similar meaning or function near each other in a numerical space.
3.2 Token Embedding: A Lookup Inside the Model
Internally, a model usually maintains an embedding matrix shaped as “vocabulary size × vector dimension,” with the dimension typically ranging from a few hundred to a few thousand. Each row of the matrix corresponds to one token’s ID in the vocabulary. For the token sequence [1, 2, 3], the lookup produces:
Embedding matrix (illustrative):Row 0: [ 0.12, -0.03, 0.05, 0.01] ← <unk>Row 1: [ 0.21, 0.08, -0.02, 0.04] ← "人工"Row 2: [-0.05, 0.33, 0.07, -0.06] ← "智能"Row 3: [ 0.09, -0.11, 0.28, 0.02] ← "技术"Output of the lookup for the token sequence [1, 2, 3]:[[ 0.21, 0.08, -0.02, 0.04],[-0.05, 0.33, 0.07, -0.06],[ 0.09, -0.11, 0.28, 0.02]]
This table is essentially a set of model parameters that keeps updating during training. After training, for a given token, the lookup is basically a memory read whose cost is far smaller than the subsequent attention computation. The reason is that attention requires pairwise interaction between tokens, so its cost grows quadratically with sequence length, while the lookup cost grows only linearly with the number of tokens.
The order of a sequence is usually provided separately by positional encoding. You can think of it as a division of labor: the embedding answers “what is this token,” and positional encoding answers “where does it appear.”
3.3 Text Vectors: The Foundation of RAG Retrieval
The target of RAG retrieval is not an individual token but an entire passage of text. The common approach uses an embedding model to convert a passage of text into a fixed-length vector:
- Tokenization: Use the tokenizer that ships with the model to split the text into tokens;
- Vector computation: Obtain the vector representation of each token;
- Pooling: Aggregate them into a single vector through methods such as average pooling or weighted pooling.
The difference between text vectors and keyword retrieval can be summarized as:
| Dimension | Keyword retrieval | Vector retrieval |
|---|---|---|
| Matching basis | Literal term overlap | Distance in vector space |
| Typical strengths | Exact matches for model numbers, IDs, abbreviations | Paraphrases, colloquial expressions, cross-language semantic matches |
| Main limitation | No match if the wording differs | Weak at character-level exact matching |
For example, if a user asks “how do I get my money back” while the knowledge base says “order refund process,” the two share no literal terms, but in semantic space their vectors are close, so vector retrieval can still find a match. Conversely, for exact product model numbers or error codes, keyword retrieval is often more reliable. For this reason, mature RAG systems typically adopt a hybrid retrieval strategy that combines vector and keyword retrieval.
4. How Token and Embedding Work Together to Support RAG
4.1 The Overall RAG Flow
The core idea of RAG is “retrieve first, then generate”: first find content relevant to the question from an external knowledge base, then hand it to the LLM as context to generate an answer. The goals are twofold: to make the model answer based on the latest, most accurate knowledge (mitigating stale knowledge and hallucination), and to make answers traceable and verifiable. The whole pipeline splits into an offline and an online stage:
- Offline stage: document ingestion → parsing and cleaning → chunking → embedding → writing to the vector store;
- Online stage: query understanding → query embedding → hybrid retrieval → reranking → context assembly → answer generation.
Tokens and embeddings run through both stages:
- Tokens handle “segmentation, counting, and length control,” deciding which content enters the model;
- Embeddings handle “representation and matching,” deciding which content can be found.
4.2 Offline Stage: Building the Knowledge Base
- Data ingestion and parsing: Documents are usually first stored in object storage, then read in batch and parsed into structured plain text. Parsing should preserve title hierarchy, lists, tables, and code-block boundaries as much as possible;
- Chunking: Split long documents into pieces suitable for retrieval. Tokens play two roles here: as a length unit for measuring chunk size, and as a boundary reference to avoid cutting a token sequence in half;
- Embedding: Each chunk is turned into a vector by the embedding model and stored together with the original text and metadata such as source, section, and last-updated time;
- Writing and indexing: Vectors are written into a vector database and indexed, while the original text and metadata can live in a document or relational database, linked by a unique identifier to support both retrieval and traceability;
- Incremental updates: When a document changes, only the chunks whose content actually changed need to be re-embedded, avoiding the wasted compute and index inconsistency caused by full rebuilds.
4.3 Online Stage: Retrieval and Generation
- Query understanding: Resolve references, rewrite colloquial phrasing, and split compound questions to improve retrieval hit rates;
- Query embedding: Process the query with exactly the same embedding model used to build the index;
- Hybrid retrieval: Retrieve candidate chunks separately via vector and keyword retrieval, then merge and deduplicate;
- Reranking: Use a reranker to finely score “question–chunk” pairs one by one and pick the most relevant handful of chunks;
- Context assembly: Assemble the instructions, retrieved chunks, and user question into a prompt under a token budget;
- Generation and attribution: The LLM generates an answer based on the context and cites its sources.
4.4 Key Details
(1) Chunking: balancing semantic completeness against token budget
- Chunks that are too small lack context, and the prerequisites needed for an answer may be scattered;
- Chunks that are too large dilute relevant information and consume precious context space.
A common practice is to primarily use natural semantic boundaries such as titles and paragraphs, combine them with overlap windows (adjacent chunks sharing some content) to avoid losing information that crosses boundaries, and set a maximum length measured in tokens.
(2) Consistency: the embedding model must be the same for indexing and querying
Text vectors are only comparable within the same vector space. If indexing and querying use different models, different pooling methods, or even different tokenization configurations, similarity becomes distorted even when each model performs well on its own. Switching the embedding model usually requires a full index rebuild.
(3) Reranking: making up for the limits of dual-encoder models with cross-encoders
Vector retrieval typically encodes the question and the document separately, computing distance only at the final step. This is fast but struggles to model fine-grained interactions between the two. A reranker concatenates the question and the chunk and feeds them in together, capturing fine-grained correspondences through cross-attention. It is more accurate but more expensive, making it well suited to finely ranking a small set of candidates.
(4) Context assembly: using tokens as the unified “budget unit”
Context-length limits are measured in tokens, and there is no fixed conversion ratio between characters in the knowledge base and token counts. System instructions, tool definitions, conversation history, retrieved chunks, and reserved output all share the same budget. Without token-level control, you can end up in a situation where the input fills the entire budget and leaves the model no room to answer.
4.5 Example: Context Assembly Under a Token Budget
def build_prompt(instruction, history, chunks, user_query,budget, reserved_output):"""Assemble a prompt within the token budget; chunks must be sorted by relevance, descending"""parts = [instruction]used = count_tokens(instruction)for msg in history[-3:]: # keep only the most recent few turns of historyneed = count_tokens(msg)if used + need > budget - reserved_output:breakparts.append(msg)used += needfor c in chunks:block = f"[{c.source}] {c.text}"need = count_tokens(block)if used + need > budget - reserved_output:break # out of budget; skip subsequent, less relevant chunksparts.append(block)used += needparts.append(user_query)return "\n\n".join(parts)
5. Common Pitfalls and Engineering Advice
| Common pitfall | Potential consequence | Improvement |
|---|---|---|
| Estimating tokens from character counts or “roughly” | Large length-estimation errors; requests rejected or answers truncated | Count precisely with the tokenizer that matches the model |
| Using different embedding models for indexing and querying | Distorted similarity and degraded retrieval quality | Pin the model version and rebuild the index fully on changes |
| Relying only on vector retrieval | Missing exact information such as model numbers, error codes, and abbreviations | Adopt hybrid retrieval combining vectors and keywords |
| Mechanical fixed-length chunking | Truncated semantics and broken tables or code structure | Split along semantic boundaries and set an overlap window |
| Not syncing the index after document updates | Retrieving stale content and distorted answers | Update the index incrementally based on content changes |
| Not citing sources in generated results | Answers that are hard to verify and a higher hallucination risk | Attach metadata to chunks and citations to answers |
Two additional points are worth noting:
- Diverse deployment options: Beyond building your own vector store and retrieval pipeline, many teams adopt turnkey knowledge-base solutions from cloud platforms. These solutions usually wrap chunking, embedding, retrieval, and reranking internally. Whichever form you choose, you should ensure that core metrics such as chunk granularity, retrieval quality, and token usage are observable;
- Build an evaluation system: Continuously tracking a fixed evaluation set and metrics such as recall and answer accuracy is the basis for judging whether an optimization actually works, so you avoid tuning based on subjective impressions alone.
Conclusion
Token and embedding may seem like two basic concepts, but they actually determine two categories of core constraints for LLMs and RAG:
- Tokens determine “how much fits”: They define the model’s input and output unit, affecting context capacity, response speed, and API cost;
- Embeddings determine “how accurately things are found”: They define the semantic space, affecting retrieval quality and answer accuracy.
Linked together, they form the data pipeline of “text → token → vector → context.” Understanding this chain not only explains common phenomena such as “why is it too long” and “why can’t it be found,” but also helps developers optimize chunking strategies, retrieval approaches, and context organization more purposefully, ultimately building LLM applications that are more reliable and more trustworthy.
