Token Counting in Large Language Models: Principles and Optimization Practices

1. From Characters to Tokens: The Basic Processing Unit of LLMs

People typically understand text in terms of characters and words. Traditional programs also often process text character by character. For large language models, however, neither granularity is ideal.

If text were processed strictly character by character, languages such as English would require very long sequences because each individual letter carries relatively little semantic information. This would significantly increase computational cost. Processing text strictly by word would create a different problem: vocabularies can grow without bound, and no fixed vocabulary can cover every new word, proper noun, or internet expression. The model would fail whenever it encountered an out-of-vocabulary term.

For these reasons, most modern large language models use a unit between characters and words: the subword. Text is divided into a sequence of tokens. A token is the basic unit processed by a language model. The model receives tokens as input and generates tokens as output. Whether the input is Chinese, English, source code, or mixed content, it is ultimately encoded as a token sequence. The model processes the sequence token by token and generates its response in the same way.

A token is not the same as a character or a word. It is a semantic fragment produced by a tokenizer according to the vocabulary and merge rules learned during training. The exact segmentation depends on the tokenization scheme used by the model.

2. How Tokens Map to Characters

Understanding the relationship between tokens and characters is essential for estimating costs and planning context usage. The general patterns are as follows.

2.1 English: Approximately 3–4 Characters per Token

English words are often split into subword units such as roots, prefixes, and suffixes. For example, unbelievable might be divided into un, believ, and able. Common high-frequency words such as the and and are often represented by a single token.

As a general industry estimate, one English token corresponds to roughly three to four characters on average. The actual ratio varies depending on the vocabulary and the specific text.

2.2 Chinese: Often About One Character per Token

Tokenizers designed for multilingual models commonly include frequently used Chinese characters and high-frequency Chinese words in their vocabularies. As a result, Chinese text often follows a pattern of approximately one Chinese character per token. Some common two-character words and four-character idioms may be combined into a single token.

However, Chinese and English have different information densities. A single Chinese character often conveys more information than an individual English letter, so equivalent content may require different numbers of tokens in the two languages.

2.3 Code and Symbols Have Higher Token Density

Indentation, line breaks, spaces, punctuation, and digit sequences are also split and counted during tokenization. In well-formatted source code, spaces and indentation consume tokens. Long sequences of digits are often divided into smaller pieces as well.

Consequently, code, JSON, and other structured content may consume more tokens than their visual length suggests.

2.4 The Same Text May Have Different Token Counts Across Models

Different models use different vocabulary sizes and tokenization rules. The same greeting, for example, may be divided into different numbers of tokens by two models.

Token counts therefore should not be compared directly across models. For accurate results, use the counting method associated with the model being used.

3. The Two Dimensions of Token Usage: Input and Output

Token usage in an LLM application has two components.

Input tokens are all the text sent to the model. This may include system instructions, predefined prompts, conversation history, the user’s current question, and context assembled for retrieval-augmented generation (RAG).

Output tokens are the tokens generated by the model in its response.

The basic formula is:

Total tokens = Input tokens + Output tokens

For example, if a user request and its conversation history consume 100 input tokens and the model generates a 200-token response, the total usage for that request is 300 tokens.

Two commonly overlooked facts follow from this calculation:

1. In a multi-turn conversation, each request typically sends the previous conversation history again as input. As the conversation becomes longer, input token usage accumulates.
2. Output length depends directly on the generation strategy. Without constraints, the model may produce unnecessarily long responses, causing output token usage to grow.

4. Estimating and Tracking Tokens in Production

In applications where cost or context capacity matters, visual estimation is unreliable. It is better to count tokens precisely before sending a request.

Recommended practices include:

1. Pre-encode the text using tokenization rules compatible with the target model and obtain the exact token count.
2. Compare the input token count with the model’s context window limit and reserve sufficient space for the output.
3. Set a maximum generation length to prevent excessively long responses.
4. Record and monitor input and output token distributions for every request to establish an observable baseline.

Illustrative code:

# Example: token usage accounting for a single request
input_tokens  = encode_and_count(system_prompt + history + user_query)  # Input side
output_tokens = encode_and_count(model_response)                        # Output side
total_tokens  = input_tokens + output_tokens                            # Total usage
assert input_tokens + max_output_budget <= context_window  # Validate the context window

Here, encode_and_count represents tokenization and counting based on the model’s vocabulary. context_window represents the context window specification of the selected model. Context window sizes vary, so always refer to the official documentation for the model in use.

5. How Token Count Affects Applications

Context Window Limits

Every model has a maximum context length, and the combined input and output must stay within that limit. Exceeding it may cause the earliest conversation history to be dropped or the request to fail altogether. Long-document analysis and extended multi-turn conversations require particular attention.

Resource Usage

Token count determines the sequence length the model must process and is a fundamental unit for estimating computational resource consumption and cost. More input and output tokens generally require more resources.

Response Latency

Model generation occurs token by token. The more output tokens the model generates, the longer the user may need to wait. Controlling response length is therefore an important way to improve interactive performance.

6. Practical Token Optimization

6.1 Simplify Prompts

Remove repetitive or vague wording and express requirements with concise instructions. For few-shot prompting, keep only the most representative examples.

6.2 Manage Conversation History

Summarize older turns and retain only the context relevant to the current question. You can also set a maximum number of historical turns or use a sliding window to prevent input growth from continuing indefinitely.

6.3 Explicitly Constrain Output

Specify the required length, number of items, or format in the prompt. For example:

Answer in three points, with no more than 50 Chinese characters per point.

Pair these instructions with a maximum generation length as a safeguard.

6.4 Select Retrieved Content Carefully

In RAG workflows, deduplicate, rank, and truncate retrieved passages. Inject only the content most relevant to the question rather than adding an entire document collection to the prompt.

6.5 Use Structured Output

For machine-consumed responses, request key-value pairs, tables, or other structured formats. This reduces explanatory and polite filler, saves tokens, and makes downstream parsing easier.

6.6 Use Caching and Streaming

For requests with repeated content or stable prefixes, caching can help avoid redundant computation. Streaming output allows results to be displayed as they are generated, reducing the user’s perceived waiting time.

6.7 Establish Continuous Observability

When deploying LLM applications, organizations can use model services and general-purpose logging and monitoring capabilities provided by cloud platforms such as Baidu AI Cloud. Continuously tracking token usage and its trends supports capacity planning and budget management.

7. Common Misconceptions

• Misconception 1: A token is the same as a character.
  Incorrect. A token is a subword unit. Individual English letters are much smaller than a token, while a Chinese character often corresponds to approximately one token.

• Misconception 2: Only Chinese characters and words are counted.
  Incorrect. Punctuation, spaces, line breaks, and numbers may also be split into tokens and counted.

• Misconception 3: Only the model’s response is counted.
  Incorrect. Input and output are counted separately and added together. The input also includes system instructions and the complete conversation history submitted with the request.

• Misconception 4: Different models produce the same token count.
  Incorrect. Different vocabularies and tokenization rules produce different results, so token counts cannot be compared directly across models.

• Misconception 5: Every turn in a multi-turn conversation consumes the same number of tokens.
  Incorrect. Conversation history accumulates over time, so input token usage usually increases in later turns unless the history is summarized or truncated.

8. Summary

Tokens are the unit of measurement for the large language model ecosystem. Models understand and generate text in tokens, while applications use token counts to plan context, estimate resource consumption, and optimize latency.

Understanding how tokens map to characters, how input and output usage is calculated, and how to measure and optimize tokens in production is a foundational skill for LLM application developers. Careful token management helps applications remain reliable, cost-efficient, and responsive without sacrificing quality.