How LLM Tokenization Works: Why Tokenizers Split Chinese Text More Finely Than English
Large language models do not read text the way people do. They process sequences of integer token IDs produced by a tokenizer. This article explains what tokens are, how byte pair encoding (BPE) merges frequent character pairs into tokens, and why English and Chinese receive very different treatment. It also covers the practical effects of over-fragmented tokenization, including higher token counts and inference costs, shorter effective context, and error-prone semantic boundaries. Finally, it offers practical ways to mitigate these issues, from prompt wording and long-document handling to model selection and structured input. Read this to understand why a model sometimes seems to lose track of a Chinese sentence, and what you can do about it.
1. When a Machine “Reads” Text: You See Meaning, the Model Sees Numbers
When you read a passage, your brain naturally groups characters into words and words into meaning, taking in the logic and the emotion behind them. For a large language model (LLM), however, text is not something it perceives directly. Inside the model, a passage of text is first converted into a sequence of integer IDs, and it is these IDs that the model actually “sees” and uses in its computations.
The component responsible for this conversion is the tokenizer. It cuts a continuous stream of characters into discrete units and assigns each unit a unique ID. Understanding how a model splits text is the first step toward understanding why it sometimes gives off-target answers, and why Chinese text often seems to give it more trouble.
2. Tokens: The Smallest Unit of a Model’s Perception
A token is the atomic unit in which a large language model processes information. It is neither strictly a character nor quite a word, but a subword fragment that falls somewhere in between. Everything the model computes, from the attention mechanism to the probability distribution over the vocabulary, is expressed at the token level.
A useful way to picture this: a model does not directly “know” Chinese characters or words in the human sense. It knows a finite set of pre-defined tokens listed in its vocabulary, which typically contains tens of thousands of entries. Much of what the model can express, how precisely it can express it, and how much it costs to run is therefore constrained by this limited vocabulary.
3. BPE: Letting Statistics Guess Where Words Begin and End
Most mainstream LLMs today use Byte Pair Encoding (BPE) or similar subword algorithms to split text. The core idea is not mysterious. At its heart, BPE is a set of frequency-based statistical merging rules:
- Frequent combinations: characters or fragments that often appear next to each other in the corpus are merged into a single token.
- Infrequent combinations: fragments that appear rarely, or never appeared in the training data, are split into smaller units.
The BPE training process can be summarized with the following illustrative code:
# BPE training flow (illustrative)def train_bpe(corpus, num_merges):# 1. Initial vocabulary: all single charactersvocab = set(get_characters(corpus))# 2. Count the frequency of all adjacent symbol pairspair_freq = count_pairs(corpus)for _ in range(num_merges):# 3. Pick the most frequent adjacent pairbest = max(pair_freq, key=pair_freq.get)new_token = "".join(best)vocab.add(new_token) # 4. The merged fragment enters the vocabularycorpus = replace_all(corpus, best, new_token) # 5. Replace the fragment throughout the corpuspair_freq = count_pairs(corpus)return vocab
This mechanism has two defining characteristics:
- It is strongly frequency-driven. Whether a fragment ends up as a single token depends on how often it occurs in the training corpus, not on whether it is a “complete word” in the linguistic sense.
- It is strongly language- and domain-dependent. Different languages and domains have different distributions, so tokenization results differ as well. A tokenizer trained mainly on one language’s corpus often does not handle other languages as naturally.
At its core, BPE seeks a balance between vocabulary size and semantic efficiency. A larger vocabulary lets each token carry richer meaning and shortens sequences, but raises the model’s compute and storage costs. A smaller vocabulary does the opposite.
4. English vs. Chinese: Two Very Different Experiences
English has a built-in advantage: the space. It is a boundary signal that requires no statistics to be recognized and is almost perfectly reliable. Longer words such as information or streaming are usually preserved as single tokens thanks to how frequently they appear.
Chinese has no such guardrail. Characters are written tightly together with no explicit separator, so BPE must rely entirely on statistical frequency to guess where one word ends and the next begins. This leads to two typical outcomes:
- High-frequency words get the “merge treatment.” Common phrases such as “人工智能” (artificial intelligence) can end up as a single token.
- Low-frequency words get chopped up. A rare personal name, place name, or technical term that never appeared frequently in the corpus is often split into individual characters, or even finer byte-level IDs.
A classic ambiguity is the string “研究生命起源.” It can be read as “研究 / 生命 / 起源” (“research / life / origin,” meaning research into the origin of life), but it can also be misread as “研究生 / 命 / 起源” (“graduate student / life / origin”). In English, spaces rule out this kind of ambiguity automatically. In Chinese, the model has to guess.
5. Three Real-World Effects of Tokenization Differences
Tokenization may seem like a minor preprocessing step, but it has a direct, practical impact on model performance and engineering costs.
- Token counts and cost. The same piece of information often requires more tokens in Chinese. Because context window limits and token-based inference pricing are measured in tokens, Chinese content reaches the context limit more easily and each API call tends to cost more.
- Misaligned semantic boundaries. Tokenizing too finely makes it harder for the model to grasp complete semantic units, especially with long entities, technical terms, and personal names. Once a boundary is cut in the wrong place, everything built on top of it can drift.
- A gap in compression efficiency. The more fragmented the tokenization, the longer the sequence, and the less useful information the same context window can hold. This directly reduces the effective context available for long-document processing and retrieval-augmented generation (RAG).
6. Working Around Tokenization Drawbacks: Engineering Strategies
Tokenization is largely fixed during a model’s pre-training, and individual developers cannot change the underlying vocabulary. At the usage level, however, there is still plenty of room to optimize:
- Write prompts more deliberately. Prefer standard, common phrasing, and avoid obscure abbreviations and heavily custom terms. When a proper noun is unavoidable, add a brief explanation.
- Handle long documents with care. Split long Chinese materials into sections and summarize them before feeding them to the model to ease the pressure on the context window.
- Consider Chinese support when choosing a model. Pay attention to how well different options handle Chinese corpora. They genuinely differ in tokenization efficiency for Chinese.
- Structure your input. Use clear separators for key-value pairs, tables, and other structured fields to reduce the guesswork the model must do about fuzzy boundaries.
7. Conclusion
Behind the simple question “would you dare to let an AI write your summary?” lies a fact that is often overlooked: LLMs are not “reading” text. They are processing token sequences. BPE splits text using statistics, which gives English a head start thanks to spaces. Chinese, running “with no guardrails,” pays the price of more fragmented tokenization, higher costs, and boundaries that are more likely to be wrong. Once you understand how tokenization works, you can assess a model’s highlights and failures more rationally, and make better decisions about prompt design, document handling, and model selection.
