Token Generation and Consumption in AI Systems: A Technical Deep Dive
Before you can reason about the cost, performance, or user experience of any AI application, you need to answer a more fundamental question: what is the model actually processing? The answer is tokens. A token is both the atomic unit through which a model reads the world and the unit of measurement for the resource cost of a single interaction. Estimating API spend, designing a context window, and debugging response latency all eventually come down to how tokens are generated and consumed. Understanding that chain end to end is the prerequisite for understanding how modern AI systems operate. This article walks through the token generation pipeline, the difference between input and output tokens, the token lifecycle across multi-turn agent loops, and why GPU utilization alone can no longer describe the real state of an AI system.
1. Understanding Tokens: The First Key to an AI System
A token sits at the center of nearly every practical decision about a model. It determines how long an input is, how much computation happens downstream, and how much a request costs. The sections below trace how a token is produced, how tokens accumulate across a conversation, and which metrics matter once you move from single-turn requests to production agent workloads.
2. The Token Generation Pipeline: Back and Forth Between Text and Vectors
Token generation is not a black box. It is a clear pipeline: text → numbers → text.
2.1 Tokenization: Cutting Text Into Discrete Atoms
A model does not read characters directly. Instead, a tokenizer first splits the input text into a set of subword units and maps each unit to an integer ID. In Chinese, a single word may be broken into one or more tokens. In English, common words usually map to a single token, while rare words and proper nouns are split into finer fragments.
# Conceptual illustration: tokenization turns text into a sequence of integer IDstext = "今天天气不错"token_ids = tokenizer.encode(text) # e.g. [1842, 5621, 8873]tokens = tokenizer.convert_ids_to_tokens(token_ids) # ['今天', '天气', '不错']
This step directly determines how long the input is. The token count is, in order of magnitude, equivalent to the scale of the computation that follows.
2.2 Vectorization: From Integer IDs to High-Dimensional Float Vectors
Once you have token IDs, the model replaces each ID with a high-dimensional floating-point vector through a lookup table. That vector is the embedding. These vectors typically have thousands or even tens of thousands of dimensions, and they are the representation that actually participates in computation inside the model. Different models may use different dimensions and compression schemes, and those choices are closely tied to model architecture and training objectives.
# Conceptual illustration: IDs → vectorsinput_embeddings = embedding_table[token_ids] # shape like [3, 4096]
The design considerations behind these dimensions and compression schemes relate mostly to the model’s own architecture and training process. As a regular user, you generally do not need to worry about the internal details. What matters is the fact that a token becomes a machine-computable vector at this stage.
2.3 Inference and Decoding: Back From Vectors to Tokens
The model runs multiple layers of computation over the input vectors and produces new vector representations. An output mapping then projects those vectors back to vocabulary size, normalization turns them into a probability distribution over the “next token,” and decoding finally yields a concrete token ID that is concatenated into text.
# Conceptual illustration: vectors → next-token probabilities → sampling → text fragmentlogits = lm_head(final_hidden_state) # project to vocabulary sizenext_id = sample(softmax(logits)) # sample the next token
A complete generation is therefore a loop: tokenize → vectorize → infer → decode → concatenate. Each iteration of the loop produces one token.
3. Input Tokens and Output Tokens: The Two-Way Flow of a Single Interaction
From a billing and performance perspective, tokens are usually split into input tokens and output tokens. The two are not symmetric in how they are produced or in how they are priced.
3.1 Three Components of Output Tokens
- Reasoning: the model’s “thinking” chain before it states a conclusion. It is often generated as tokens too, and it consumes output quota.
- Text output: the final body of text presented to the user.
- Tool call: the model’s decision during reasoning to make an external call. Its structured parameters are also output.
All three happen on the generation side. They are the parts the model actively produces.
3.2 The Dynamic Accumulation of Input Tokens
The input side has two parts. One is the user’s original input. The other is the content accumulated across previous turns, and in particular the results returned by tool calls, which are concatenated back in as the input for the next round. This means that with every additional loop iteration, input tokens stack up and grow rather than staying constant.
4. Loops and Caching: The Token Lifecycle in Multi-Turn Interactions
An agent-style application typically takes the form of a loop: model output (including tool calls) → execute the tool → write the result back into the input → run inference again. As the number of rounds increases, the input keeps accumulating and token consumption rises in steps.
To reduce repeated computation, caching has become a common practice in the industry. The vector representations of inputs that have already been computed, and sometimes parts of the output as well, are stored up front. When the same prefix shows up again, it is reused directly, so the same content does not have to be vectorized and run through inference a second time.
The practical implication is that a cached prefix costs significantly less to compute, while the newly added suffix still has to be recomputed. This also leads to a useful conclusion: treating “stable prefix, incremental suffix” as a prompt design principle is usually the more economical choice.
5. Observability Challenges: Why GPU Utilization Is No Longer Enough
Modern AI systems can no longer be described by GPU utilization alone. That single number misses what actually determines whether the system is working well.
In traditional cloud-native applications, we describe system health with four categories of metrics: CPU, memory, network, and disk. Large-model inference adds new dimensions:
- GPU memory usage and the KV cache: across multiple loop iterations the cache keeps accumulating, which makes memory a hard constraint.
- Token throughput (tokens/s): the core indicator of generation capacity per unit of time.
- Time to first token (TTFT): the key experience metric that shapes how long a user feels they have to wait.
- Batching and scheduling efficiency: how requests are aggregated, preempted, and queued directly affects overall throughput.
The key point is that high GPU utilization does not necessarily mean a good user experience. The GPU may simply be “filled” by a large backlog of queued requests rather than efficiently producing useful tokens. What is needed instead is a token-centric observability system to replace the single-utilization view.
6. Implications for Engineering Practice
Understanding how tokens are generated and consumed eventually has to translate into concrete engineering actions:
- Input determines output. The actual execution process is largely shaped by the user’s prompt instructions and the surrounding engineering environment. Well-designed instructions and clearly structured context are the prerequisites for high-quality output and smooth tool calls.
- Prefix equals cost. Put stable content such as system prompts and tool definitions at the front as much as possible, so that cache hits are maximized and repeated computation is reduced.
- Keep context restrained. Watch out for unbounded accumulation of tool results and conversation history. Summarize or trim when necessary to prevent input tokens from growing out of control.
- Make tokens the center of observability. Build monitoring and alerting around throughput, time to first token, and cache hit rate rather than watching GPU utilization alone.
If you need to elastically handle high-concurrency inference and caching, general cloud capabilities can serve as a stable service foundation: container platforms for elastic scaling, monitoring and alerting for metric collection, and message queues for smoothing request spikes. Whatever the infrastructure becomes, the thread to hold onto is how tokens are generated and how they are consumed. That is the foundation for understanding and designing modern AI systems.
