Why LLMs Hallucinate: Tokenization and the Root Causes of Factual Errors

    Large language models can produce fluent, confident answers that turn out to contain fabricated papers, numbers, citations, or legal clauses. This article explains why that happens. It clarifies a common misconception about hallucination, introduces tokens as the basic unit of text processing, describes how BPE (Byte Pair Encoding) merges frequent character pairs, and examines how tokenization behaves differently across English, Chinese, and Korean. It also maps out the real causal chain behind factual hallucination and provides a practical, engineering-focused checklist: retrieval-augmented generation, mandatory source output, manual review of rare entities, dedicated checks for numbers and dates, low-confidence fallback rules, and a troubleshooting table. Developers will learn why fluency alone is not reliability, and how to constrain probabilistic generation into a verifiable workflow.

    1. A Common Symptom: Confident Answers That Are Wrong

    In practice, you will often run into this: you ask a large model a specialized question, and it returns an answer that is well structured, dense with terminology, and delivered in a confident tone. It reads like a carefully compiled analysis. Yet when you check it point by point, you discover that the papers, data, cases, or legal clauses it cites may not actually exist.

    This is commonly called factual hallucination. It helps to clear up a widespread misunderstanding first: this is not simply a case of “the more a model knows, the more nonsense it makes up.” A more accurate explanation is that when the training data is insufficient to support a specific fact, and the application does not supply reliable context, the model still generates the next token by probability to keep the text coherent. Its strength is making language look natural, complete, and professional. But at the fact-checking level, it does not automatically possess the ability to judge whether “this piece of information is true and trustworthy.”

    So understanding hallucination is not just about asking “is the model smart enough.” You also have to look at how it receives text, how it splits that text, and how it continues writing text within a probability space.

    2. What Is a Token: The Basic Input Unit for Text

    When people first work with large models, many assume the model “reads sentences” the way a human does. Technically, though, the model does not directly understand words and sentences in the human sense. Instead, it converts input text into a set of tokens.

    A token is the basic input unit the model uses to process text. It can be a character, a subword, a full word, or a special symbol. It is worth stressing that a token is not the smallest unit in a linguistic sense. It is more like an encoding result oriented toward model computation: each token usually corresponds to a numeric ID, which is further mapped to a vector so the model can use it in attention computation and probability prediction.

    When humans read, we focus on meaning, grammar, reference, and logical relationships. When the model processes text, it first deals with token IDs, positional relationships, and contextual distributions. For example, when the input is “人工智能正在发展” (“artificial intelligence is developing”), different models may split it into different numbers of tokens, depending on the tokenizer, the vocabulary, and the model version. Tokenization affects how the model perceives word boundaries, and it also affects its ability to track entities and references across long contexts.

    3. BPE Tokenization: Frequency-Based Merge Rules

    Most mainstream large language models use BPE, or Byte Pair Encoding, as one of their base tokenization methods. The core idea of BPE is not linguistic analysis but a statistical rule: it repeatedly searches for high-frequency adjacent units in the training corpus and merges them into new token units, until the vocabulary reaches a preset target size.

    You can understand it through this simplified pseudocode:

    1. def train_bpe(corpus, num_merges):
    2. vocab = get_initial_chars(corpus)
    3. for _ in range(num_merges):
    4. pair_freq = count_adjacent_pairs(corpus)
    5. best_pair = max(pair_freq, key=lambda p: pair_freq[p])
    6. vocab.add(best_pair)
    7. corpus = replace_all(corpus, best_pair)
    8. return vocab

    It is worth adding that real engineering implementations do not simply perform a global string replacement. An actual system needs to maintain a symbol sequence over words or byte sequences, along with merge rules, encoding tables, and the handling of special tokens. In other words, a more accurate description of BPE is a set of “encoding and merge rules” rather than a simple text-replacement script.

    The direct result of BPE is this: high-frequency combinations are more likely to be merged into fuller token units, while low-frequency combinations may be broken into smaller character or byte units. For instance, common English words or frequent Chinese words may be kept whole more easily, whereas obscure proper nouns, rare technical terms, and unusual personal names may be split into finer pieces because the training corpus does not cover them well.

    4. Chinese and Multilingual Tokenization: No Spaces Does Not Mean Guaranteed Errors

    A lot of discussion focuses on the fact that Chinese has no spaces. It is true that English typically relies on spaces to form natural boundaries, while Chinese has no explicit separator like spaces. Without pre-tokenization by spaces, Chinese depends more heavily on statistical rules, character frequency, vocabulary coverage, and the distribution of the training corpus.

    That said, we should avoid drawing absolute conclusions, such as claiming Chinese tokenization is necessarily worse, or that a particular language is bound to produce more errors. The real outcome depends on several factors:

    1. The type and version of the tokenizer used;
    2. The vocabulary size and coverage;
    3. The proportion of that language in the training corpus;
    4. Whether the specific vocabulary consists of high-frequency words, technical terms, or low-frequency entities;
    5. Whether the application introduces retrieval augmentation, context constraints, and result verification.

    So when judging tokenization quality, you should specify a concrete model, tokenizer, and version, rather than abstractly claiming how a given language must behave.

    The comparison table below illustrates this:

    Language Example sentence Expected tokenization tendency Main risk points Notes
    English Artificial intelligence is continually evolving. Common words may become fairly complete tokens Low-frequency terms, brand names, and model names may still be split Spaces provide boundaries, but it depends on the tokenizer version
    Chinese 人工智能正在持续进化。 High-frequency words may merge; other parts may be split by character or subword Low-frequency proper nouns, terms, and number boundaries rely more on vocabulary coverage Different Chinese BPE or tokenizers can differ significantly
    Korean 인공지능은 계속 진화하고 있습니다. May combine by syllable, subword, or morpheme units Being a phonemic script, its vocabulary coverage and corpus proportion affect splitting Requires judgment based on the specific tokenizer

    Korean is a phonemic script and should not be described simply as a “phonetic script.” The Korean example above is a valid sentence, but under different tokenizers its token count, token boundaries, and representation may differ. For developers, what really matters is not “a given language is always tokenized better or worse,” but “in the specific model and tokenizer version I use, will key entities be split apart, and will this introduce extra ambiguity?”

    5. How Tokenization Affects Factual Hallucination

    Tokenization is not the sole cause of hallucination, but it does affect factual performance. A more rigorous causal chain looks like this:

    1. The model generates text mainly based on probability distributions, rather than querying a fact database in real time;
    2. When the training corpus or application context lacks reliable facts, the model still tends to produce coherent expressions;
    3. Tokenization granularity can affect entity recognition, long-context reference, and inference cost;
    4. If key entities are split into fragments and the context does not provide sufficient evidence, the model is more likely to get confused or fill in the gaps;
    5. This ultimately shows up as wrong numbers, wrong citations, wrong entities, or wrong legal clauses.

    So we cannot simply attribute hallucination to “the more fragmented the tokens, the more hallucination there will be.” A more accurate statement is: tokenization granularity can affect long-context reference, entity recognition, and inference cost, but the main causes of factual hallucination are still the probabilistic generation mechanism, insufficient factual knowledge, a training objective that lacks fact-based constraints, and an application layer that does not provide authoritative context.

    Applicable conditions matter too. The following scenarios tend to amplify the risk:

    • Being asked to generate an obscure paper title, author, year, or DOI;
    • Being asked to explain a rare proper noun, internal system name, project name, or regulatory clause;
    • Being asked to restate exact numbers, units, percentages, or financial metrics;
    • The user provides no materials but demands a “definite answer”;
    • The context is very long, entities are scattered across positions, and the model must maintain consistent references across multiple paragraphs.

    In these scenarios, tokenization differences may become an additional risk source, but the core problem is still insufficient factual evidence.

    6. An Actionable Checklist for Developers

    To reduce factual hallucination, you cannot rely only on the model being “smarter.” Instead, you should use an engineering process to constrain probabilistic generation onto verifiable tracks. Here is an actionable checklist.

    1. Prioritize Retrieval-Augmented Generation

    Before answering a factual question, retrieve authoritative materials first, such as official documentation, databases, paper abstracts, product manuals, the text of regulations, or an internal knowledge base. Then have the model answer based only on the retrieved snippets.

    A recommended prompt constraint:

    Answer based on the following retrieved materials. If there is no clear basis in the materials, reply “insufficient information.” Do not fabricate sources, numbers, or clauses.

    This turns “open-domain generation” into “evidence-backed, constrained generation.”

    2. Require Verifiable Sources

    For high-risk content such as papers, policies, legal clauses, financial data, and statistical reports, the model should be required to output:

    • Source title;
    • Author or organization;
    • Publication date;
    • Link or document ID;
    • Original page, chapter, or snippet ID.

    If the model cannot provide a source, it should be marked as “pending verification” rather than entering formal output directly.

    3. Manually Review Low-Frequency Entities

    Obscure terms, proper nouns, personal names, organization names, product version numbers, and regulation names are high-frequency zones for hallucination. It is recommended to set up a dedicated review item in the workflow:

    • Is the person’s name spelled correctly?
    • Is the organization name complete?
    • Is the term the official translation?
    • Does the product name carry an incorrect suffix?
    • Does the regulation name match the currently effective version?

    Numerical errors often have a large impact. Key checks include:

    • Whether monetary units are consistent;
    • Whether a percentage is an absolute or a relative value;
    • Whether the year falls within a real time range;
    • Whether the legal clause number exists;
    • Whether the clause number matches its content;
    • Whether any number was swapped out within the context.

    It is advisable to extract numbers, dates, and clause numbers separately and compare them against a knowledge base or manual rules.

    5. Set Up Low-Confidence Trigger Rules

    When any of the following occurs, a fallback strategy should be triggered:

    • Retrieval results are empty;
    • Multiple sources conflict;
    • The model expresses uncertainty;
    • A key entity is missing;
    • The user demands precise citations that cannot be verified;
    • The output contains many proper nouns that cannot be traced back to a source.

    Low-confidence scenarios should not force a complete answer. Instead, you can switch to:

    The current materials are insufficient to confirm this question. Please provide the original document, version number, or query scope.

    Or you can route it to a manual review process.

    6. Build a Troubleshooting Table

    You can use the following troubleshooting table to quickly locate problems:

    Symptom Possible cause Recommended action
    Paper title does not exist Insufficient training knowledge, no retrieval Add paper-database retrieval and require DOI output
    Legal clause number is wrong Mixed regulation versions; the model completes by probability Connect a regulation database and retrieve by version number
    Numbers are inconsistent Weak long-context reference; tokenization affects entity tracking Extract numbers by segment and set up structured validation
    Proper noun is rewritten Low-frequency entity split into fragments Add an entity dictionary and manual review
    Fluent answer with no source Lack of source constraints Require source fields; refuse to answer if no source is provided

    7. Frequently Asked Questions

    What is tokenization?

    Tokenization is the process of converting text into units the model can process. A token can be a character, a subword, a word, or a special symbol, and different models and tokenizers may produce different results.

    Why does Chinese BPE matter?

    Chinese has no spaces as explicit boundaries, so tokenization depends more on word frequency, the character table, and corpus coverage. When using Chinese BPE, you should watch whether high-frequency words merge, whether low-frequency terms get split into fragments, and whether different versions affect key entity recognition.

    Why does retrieval-augmented generation reduce hallucination?

    Retrieval-augmented generation first supplies the model with external authoritative materials, changing generation from “completing from memory” to “organizing an answer based on evidence.” If the materials lack the answer, the model is also more likely to recognize that it “cannot answer.”

    How do I choose a tokenizer?

    Look at language coverage, vocabulary size, special-symbol handling, context length, inference cost, and version stability. For Chinese-language applications, focus on testing whether low-frequency proper names, numbers, dates, terms, and business entities are reasonably preserved.

    8. Summary

    Factual hallucination in large models is not a simple matter of “the model doesn’t know any better.” It is the combined result of the probabilistic generation mechanism, factual knowledge coverage, the training objective, and application context. Tokenization and Chinese BPE affect how the model perceives text and can increase the difficulty of entity recognition and long-context handling, but they do not replace an understanding of the main causes of hallucination.

    For developers, the more reliable approach is not to pursue a model that is always correct, but to build a verifiable engineering process: use retrieval-augmented generation to supplement facts, use source verification to limit fabrication, use low-frequency entity review to reduce risk, and use low-confidence rules to trigger a human safety net. Only this way can you enjoy the fluent expressive power of large models while genuinely improving your control over factual hallucination.