What Are Tokens in Large Language Models: Tokenization and Context Limits
This article explains what tokens are in large language models, how they differ from characters and words, how BPE-style subword tokenization works, why context windows cause errors, and how to estimate token usage before API calls.
What a Token Is, and How It Differs from Characters and Words
Read this sentence: “The weather is nice today.” Most people group it into phrases, not letter by letter. A large language model also splits text first, but it follows a fixed tokenizer rather than human word sense. The smallest unit after that split is a token.
Tokens are the basic unit of model input and output. Before text enters the model, a tokenizer turns it into a sequence of tokens. Each token maps to an ID in the vocabulary, and the model computes on the corresponding vectors. During generation, the model emits one token at a time and then concatenates them into text. How much the model can read, how long it can write, and how APIs bill usage all depend on token counts.
People often treat a token as “a word.” That intuition is sometimes close, but it is not accurate:
| Object | Common assumption | What actually happens |
|---|---|---|
| Character | One character is one unit | Common characters are often one token; rare ones may take several |
| Word | One complete word is one unit | cat is often one token; cats may split into cat and s |
| Punctuation and spaces | Separate from letters | Sometimes merged with neighbors, sometimes a token on their own |
Segmentation depends on the vocabulary, and the vocabulary depends on how often character combinations appeared in training data. The same text can have different token counts across models, so you cannot reuse one model’s counts for another.
How High-Frequency Pieces Get Packed
Most mainstream models use statistical subword tokenization. Byte Pair Encoding (BPE) is a common approach. Training roughly works as follows:
- Start from the smallest units (characters or bytes).
- Count how often adjacent unit pairs appear in the corpus.
- Merge the most frequent pair into a new unit and add it to the vocabulary.
- Repeat steps 2 and 3 until the vocabulary reaches the target size.
The code below shows a single merge. It does not count global frequencies or implement full BPE training. It only illustrates that matching adjacent symbols are merged. Do not treat it as a production implementation.
def merge_pair(corpus: dict, pair: tuple) -> dict:"""Merge every adjacent occurrence of pair into one new symbol."""a, b = pairnew_corpus = {}for word, freq in corpus.items():symbols = word.split()out, i = [], 0while i < len(symbols):if i < len(symbols) - 1 and symbols[i] == a and symbols[i + 1] == b:out.append(a + b)i += 2else:out.append(symbols[i])i += 1new_corpus[" ".join(out)] = freqreturn new_corpuscorpus = {"i n g": 10, "t i o n": 6, "p r e": 8}print(merge_pair(corpus, ("i", "n")))# {'in g': 10, 't i o n': 6, 'p r e': 8}
Real training repeats this merge tens of thousands of times or more. Frequent fragments such as ing are merged early and become vocabulary entries. Rare words get fewer merges and stay more fragmented, sometimes down to single bytes.
A direct result: common words are cheaper to process; rare words, proper names, and misspellings usually consume more tokens.
Why not tokenize by character? Two reasons. First, sequences become long. “I love machine learning” has 23 characters including spaces, but typical tokenization yields only a few tokens. Sequence length is capped, and longer sequences cost more compute. Second, single characters carry little meaning. The letters l, o, v, and e are weak on their own; love is meaningful. Tokenization is a compromise: shorter sequences while keeping frequent combinations.
Context Length Limits and Why Requests Fail
A model can process only a limited number of tokens at once. That limit is the context window. Input and output share it. System prompts, chat history, retrieved documents, the user question, and the generated answer all count toward the total.
Most APIs also expose an output-length control, often named max_tokens. It caps how many tokens the current answer may generate; it is not the input limit. If the input already uses most of the window, there may not be room for the requested output, so the request is rejected or the answer is truncated early. Parameter names and behavior differ by provider; use the official documentation.
Common overflow causes:
- The input already exceeds the model’s context limit.
- Input length plus
max_tokensexceeds the window. - Multi-turn history grows without truncation or summarization.
- Retrieved text, logs, or code are pasted into the prompt in full.
- You switch to a smaller-window model but keep old length assumptions.
Most APIs compare total length against the limit at validation time and return an error if it is exceeded. Automatic truncation is provider-specific; do not assume it.
Estimating Tokens and Planning Calls
Estimating length before a call prevents most overflows. A practical sequence:
- Identify the target model and use its tokenizer. Vocabularies differ.
- Count the real input. Prefer a provider counting API or the official tokenizer. Heuristics are only a first pass; calibrate on real samples before production.
- Reserve output space. Set
max_tokensfor the task and confirm input tokens plusmax_tokensstay inside the context window. - Keep a safety margin. Templates and user input vary; the margin should match business risk.
- Handle overflow: trim history, summarize, reduce retrieved chunks, split work, or use a larger-window model.
Rough estimates are only for early judgment. Chinese (and other languages) token counts vary by model. “One character equals one token” is a loose impression, not a rule. For cost or hard limits, measure with the target tokenizer.
When you hit a length error, check:
- Whether the system prompt is too long or large fixed blocks are concatenated on every request.
- Whether multi-turn history is truncated or summarized.
- Whether retrieved chunk count and size are bounded.
- Whether logs, code, or tables are dumped into the prompt.
- Whether
max_tokensexceeds remaining space after the input. - What the current model’s context limit is, and whether code matches it.
- Whether the
usagefield’s input token count matches your estimate.
Most APIs return a usage field with input, output, and total tokens. Log these in production and use them to tighten estimates over time.
