How Large Language Models Work: From Predicting the Next Token to Attention and Emergence

    Large language models can hold conversations, translate text, and write code, but at their core they do one thing: predict the next token. This article breaks down the mechanics behind that objective, from tokenization and embeddings to the attention mechanism, Q, K, and V, multi-head attention, and token-by-token generation. It also explains how emergent capabilities arise from scale and what this means for developers working with context, token costs, sampling randomness, and cloud infrastructure.

    Large language models (LLMs) can hold conversations, translate, and write code. Trace their workings back to the root, though, and they are all doing the same thing: given some preceding text, predicting the most likely next word — or, more precisely, the next token.

    That objective is so simple it resembles a word-chain game. Yet when a model is trained on massive amounts of text to the point of excellence, it is forced to learn grammar, facts, logic, and even patterns of reasoning along the way. To fill in text accurately, it has to implicitly master these rules.

    An Underrated Objective: Predicting the Next Word

    This brings us to what people mean by “emergence.” When model scale — parameter count, training data volume, and compute — crosses a certain threshold, a model suddenly displays combined capabilities that smaller models lack, such as multi-step reasoning, instruction following, and code generation.

    These capabilities are not features that were specifically programmed in. They are a byproduct of the model getting good enough at prediction. With that main thread in mind, the rest of this article breaks down the full pipeline inside an LLM, from text input to generated output.

    From Text to Numbers: Tokenization and Embeddings

    Machines only understand numbers. They cannot directly process Chinese characters, English letters, or symbols. So the first step is tokenization: splitting continuous text into a sequence of discrete tokens, where each token maps to an integer ID in the model’s vocabulary. A token can be a character, a word, a subword, or punctuation.

    Illustrative example (not a real vocabulary):

    1. Text: "I love smart search"
    2. Token IDs: [314, 52, 887, 1206]

    Next comes embedding: each integer ID is mapped to a high-dimensional, dense vector. Think of an embedding as a word’s “semantic coordinates.” Words with similar meanings sit closer together in this vector space. This is what allows a model to express and reason about relationships such as “king” and “queen” or “cat” and “dog” in geometric terms.

    The Core of the Transformer: The Attention Mechanism

    Once the model has vectors, the real question becomes: which earlier information should the current token refer to?

    When you read a pronoun in a sentence, you instinctively look back to see what it refers to. The attention mechanism turns that act of “looking back” into trainable matrix operations:

    • For the token at the current position, the model computes relevance scores against all preceding tokens. Higher scores mean a closer relationship.
    • The model then aggregates the earlier information weighted by those scores.
    • The result is a new representation of the current token that has absorbed its context.

    From Formula to Intuition: What Q, K, and V Actually Compute

    Attention is best understood as a kind of retrieval.

    • The current token issues a query.
    • Each preceding token provides two things: a key (a label for its content) and a value (the actual information it carries).
    • The query is matched against every key to compute a similarity. The higher the similarity, the more of the corresponding value is pulled in.

    The classic formula is:

    1. Attention(Q, K, V) = softmax( QKᵀ / √d_k ) V

    Dividing by √d_k controls the numerical scale, keeping the softmax from entering a saturated region where gradients become vanishingly small. The illustrative code below uses pseudocode to show how the computation works:

    1. import numpy as np
    2. def softmax(x):
    3. x = x - np.max(x, axis=-1, keepdims=True) # Numerically stable
    4. e = np.exp(x)
    5. return e / np.sum(e, axis=-1, keepdims=True)
    6. def attention(Q, K, V):
    7. d_k = Q.shape[-1]
    8. scores = Q @ K.T / np.sqrt(d_k) # Relevance of each query to each key
    9. weights = softmax(scores) # Normalize into weights; each row sums to 1
    10. return weights @ V # Weighted aggregation produces the context-aware representation

    In a nutshell, attention is a differentiable, trainable “soft retrieval” operation: relationships that matter get a closer look, while the rest is treated as background noise.

    Stacking It Up: Multi-Head Attention and Deep Networks

    A real Transformer does not run attention just once. Its architecture adds several more pieces:

    • Multi-head attention: Multiple attention heads run in parallel. Each head can attend to different patterns, such as syntax, coreference, or semantics. Their outputs are then concatenated and fused, giving the model richer expressive power.
    • Feed-forward network and residual connection: Each attention block is followed by a feed-forward network. Residual connections and normalization keep gradients stable as they flow backward through very deep networks.
    • Deep stacking: Layers like these are repeated dozens or even hundreds of times. Representations progress from shallow, word-level features toward deeper semantic understanding and reasoning.

    During generation, at every step the model outputs a probability distribution over its entire vocabulary. It then samples (or selects, according to some strategy) a token, appends that token to the input, and repeats the process. This loop produces a complete piece of text token by token.

    Where the Capabilities Come From: Emergence and Scale

    Any single next-token prediction is limited in what it can do. But as parameter count, training data, and compute grow together, the model is forced to encode vast amounts of world knowledge and patterns into its parameters.

    Once scale crosses the critical threshold, the model “incidentally” gains capabilities such as translation, summarization, reasoning, and code generation, none of which were explicitly trained for. That is emergence.

    It also explains why many capabilities of large models are hard to derive from a design document alone. They can only be progressively brought out through scaling up and the choice of training strategy.

    Practical Takeaways for Developers

    • Context is everything. Attention determines what the model “sees,” so how you organize the input — prompt design, context window management — directly affects output quality.
    • Tokenization affects cost and effectiveness. Token count determines the cost of each call and how much of the context window is used. Long-text scenarios need strategies such as truncation, chunking, and retrieval augmentation.
    • A probabilistic core means randomness. Output comes from sampling, so the same prompt may produce different answers. Configuration options such as temperature let you tune the trade-off between stability and variety.
    • Production requires supporting engineering. Embedding model capabilities into business workflows usually also calls for general-purpose cloud infrastructure, such as object storage, message queues, container platforms, logging services, and monitoring and alerting. These cover data ingestion, service deployment, and observability.

    Summary

    At its core, an LLM is a probabilistic system that has pushed “predicting the next word” to the extreme on massive amounts of text and, because of its scale, has developed emergent capabilities. Understanding the core pipeline of tokenization, embeddings, and attention gives you a grasp of how it works at a structural level. With that foundation, you can assess what it can and cannot do in real business scenarios more rationally.