How Images and Audio Become Tokens in Multimodal Models

    Multimodal models must convert continuous signals such as pixels and waveforms into position-aware token sequences that a Transformer can process. This article explains the main image and audio encoding strategies, how local features develop into semantics, how modalities align with text, and why token density, quantization, positional encoding, and separate caches directly affect latency, cost, fidelity, and response stability.

    Introduction

    Text is naturally a sequence of symbols. A tokenizer mainly decides where to place boundaries within an existing symbolic vocabulary. Images and audio have no equivalent vocabulary. Images are numerical values arranged on a two-dimensional grid, while audio is a sequence of samples along a time axis. Both signals are continuous, redundant, and locally correlated.

    For a multimodal model to use the same sequence modeling machinery as text, it must answer several questions: How can continuous signals become tokens that support attention? What determines the number of tokens? At which layer does semantic meaning emerge?

    The discussion below follows four stages: representation choices, the progression from local features to global meaning, cross-modal alignment, and engineering trade-offs. Each design choice is visible to users. Image resolution, recording duration, and whether caches are separated by modality can all change latency, cost, and answer consistency.

    1. What “Token” Means in a Multimodal Context

    In a multimodal system, a token is not simply a linguistic subword. It is a position in the sequence model. At minimum, that position carries three elements:

    • A vector, either a continuous embedding or an embedding looked up from a discrete ID
    • A position, such as a row and column in an image or a timestamp in audio
    • A modality indicator showing whether the content belongs to an image, audio, or text

    Later layers do not directly inspect raw pixels or waveforms. They perform weighted aggregation across these positions.

    The encoding stage therefore has to solve three problems at once:

    1. Compress the original samples into an acceptable sequence length.
    2. Map each position to a vector width compatible with text embeddings, or at least to a space that can be aligned through a projection.
    3. Preserve enough spatial or temporal structure for attention to recover information such as “the upper-left region” or “the second second of the recording.”

    Over-compression removes detail. Under-compression produces long sequences, and attention computation and memory usage generally grow approximately quadratically with sequence length.

    2. Two Main Paths for Image Encoding

    Path One: Regular Patches Followed by Linear Projection

    An image can be divided into non-overlapping square patches with a fixed side length. For an image with height (H), width (W), and (C) channels, using patches with side length (P) produces approximately:

    [
    (H/P) \times (W/P)
    ]

    patches.

    Each patch is flattened into a vector of length (P \times P \times C), then multiplied by a learned matrix to produce an embedding with a fixed dimension. In implementation terms, this is primarily a reshape followed by matrix multiplication. The encoder does not first run an explicit edge detector, corner detector, or handcrafted texture operator.

    The patches can receive learned positional embeddings or two-dimensional sinusoidal encodings so that the model knows their relative locations.

    The direct consequence is that higher resolution usually means more tokens, often growing approximately with image area. Doubling both image dimensions produces roughly four times as many patches. Users experience this as larger images taking longer to process, costing more, and consuming more of the context window.

    Common engineering trade-offs include:

    • Limiting the longest side while preserving the aspect ratio
    • Using larger patches to reduce token count, at the cost of making thin lines and small text less clear
    • Applying variable resolution or denser sampling only to regions of interest

    These approaches trade information retention for sequence length. They do not provide free increases in visual clarity.

    Path Two: A Learned Visual Vocabulary with Discrete IDs

    Another approach is to train an image codec with an encoder, a bottleneck, and a finite codebook. The encoder maps an image to a set of latent vectors. Each latent vector is replaced by the nearest codebook vector, producing a sequence of integer indices. A decoder can then reconstruct the image from those indices.

    The index sequence can be predicted autoregressively in the same general way as text tokens. Codebook size, the number of codes used at each spatial position, and the downsampling factor jointly determine the length and reconstruction quality of the resulting “visual sentence.”

    This approach is closer to the traditional tokenizer concept: the vocabulary is finite, representations can be reused, and the bottleneck encourages the model to discard pixel-level noise while retaining decodable structure.

    The costs are also clear. Quantization introduces reconstruction error. Fine textures, small characters, and unusual patterns can be mapped to similar codewords. If the codebook collapses and many images use only a small subset of codes, semantic discrimination is reduced.

    Generative tasks often benefit from discrete indices because the model’s next step is simply to predict the next ID. Understanding tasks often retain continuous embeddings to avoid quantization loss.

    The two paths can coexist. A model may use continuous patch embeddings for understanding and discrete codes for generation, or use continuous tokens for encoding and discrete tokens for decoding. The right choice depends on whether the task requires differentiable dense features or a finite set of sampleable symbols.

    3. Patch Extraction Is Not, by Itself, a Visual Algorithm

    A common misconception is that a patch encoder first recognizes edges and shapes before passing them to a language model. With regular patch encoding, a patch is initially just a small collection of pixel values. Half an ear, part of a table edge, or a highlight may look like nothing more than a combination of colors when viewed in isolation.

    The projection matrix learns how to map those numerical values into directions that are compatible with later layers. It does not explicitly output an edge map.

    Edges, textures, parts, and object-level semantics become increasingly separable across multiple layers of attention. The information radius expands layer by layer rather than becoming meaningful at one specific point.

    • In shallow layers, neighboring patches exchange information. Their responses may be sensitive to local contrast and directional gradients, resembling edges and simple textures.
    • In middle layers, larger neighborhoods are aggregated. Representations begin to correspond more reliably to repeated patterns such as hair, fabric, and metallic reflections, as well as parts such as ears, wheels, and window frames.
    • In deeper layers, positions across the image are aligned globally. The model can connect “a brown patch in the upper-left” with “the tip of an animal’s ear.”

    Therefore, patch design should not be evaluated only by whether a person can recognize an individual patch. The more important question is whether related pixels remain within a range that can exchange attention at the given patch size and network depth.

    If patches are too large, fine details are averaged away during projection, and shallow layers may not be able to recover them. If patches are too small, the sequence becomes longer. With the same number of layers, each layer covers a smaller physical area, so long-range relationships require deeper processing and become more expensive during training and inference.

    4. Audio Has No Natural “Visual Vocabulary”

    Raw audio is a one-dimensional sampling sequence, and its sampling rate is usually much higher than the rate at which semantic content changes. Assigning one token to every sample would overwhelm the text sequence. Practical encoders therefore aggregate information over time first.

    Continuous Audio Representations

    A common continuous pipeline divides the signal into frames, applies a window, and computes a short-time spectrum or uses a learned convolutional front end. The result is a sequence of frame-level vectors, which can then be projected to a dimension similar to that of text embeddings.

    The frame hop controls temporal resolution:

    • A smaller hop preserves phoneme boundaries more clearly but produces more tokens.
    • A larger hop shortens the sequence but may blur short events such as plosives and sibilants.

    A spectrogram can also be divided into two-dimensional patches, treating time and frequency like the two axes of an image. This makes it possible to reuse some image-style positional encoding. However, the frequency axis does not have the same translation invariance as a spatial axis. Low and high frequencies have different semantic importance, so positional encoding and normalization may need to distinguish the two axes.

    Discrete Audio Representations

    A discrete audio codec can use an encoder to produce a latent sequence, apply vector quantization—possibly through multiple stages of residual quantization—and reconstruct the waveform with a decoder.

    The resulting index rate is the audio token rate. It directly determines how much context a minute of speech consumes.

    A higher bitrate preserves more timbre and background detail, making it more suitable for high-fidelity reconstruction. A lower bitrate produces something closer to a semantic skeleton, which may work better for speech recognition or conversational content modeling but can lose speaker identity, emotion, and environmental sound.

    As with image codebooks, discrete audio tokens are well suited to autoregressive generation. For understanding tasks, continuous frame embeddings often avoid an additional quantization loss.

    Silence and low-energy segments are another major source of unnecessary sequence length. A simple energy threshold can remove long silent periods, but an aggressive threshold may cut off breaths and weak consonants and shift timestamps. In production systems, preserving temporal alignment and compressing silence should be treated as explicit policies rather than assuming that token length must follow waveform duration directly.

    5. Aligning Modalities with Text

    Before tokens from different modalities enter the same Transformer, they are typically projected to a shared width and given modality-type embeddings. This helps prevent image and audio vectors from occupying an indistinguishable region from subword embeddings.

    Contrastive or caption-based training can bring image-text or audio-text representations closer in a shared space. However, being close in that space does not mean that the modalities have become the same word. Alignment supports cross-modal retrieval and conditional generation, while fine-grained references still depend on attention over specific positions.

    Caching should also be designed separately for each modality.

    • Text key-value caches grow with the conversation.
    • Image tokens are often computed for the entire image at once. Repeated questions about the same image should reuse the visual cache instead of rerunning patch encoding.
    • Audio grows over time and is well suited to chunked encoding, with caches that append only newly received frames.

    If all modalities are placed into one cache with the same invalidation policy, two user-facing problems can occur: replacing an image may still hit stale visual state, or a long recording may push earlier text state out of the context window.

    Separating caches and allowing interaction only at defined cross-attention or sequence-concatenation boundaries makes visual recomputation and text continuation independent, producing more predictable latency.

    Positional encoding should not simply use one shared one-dimensional absolute position scheme:

    • Images need two-dimensional positions. After resizing, the same object may map to different patch indices, so relative or interpolated positional schemes are often more stable.
    • Audio needs monotonic time positions. If silence is inserted or segments are truncated, timestamps must be recalculated.
    • Concatenation order matters. Whether text comes before an image or text and image tokens are interleaved changes visibility under a causal mask.

    The sequence layout must match the product semantics of references such as “use this image.” Otherwise, the model may attend to the wrong image or associate a request with the wrong segment.

    6. Reading User-Visible Behavior from Design Choices

    Sequence length is the first major control variable. Image dimensions, patch size, audio frame hop, and codebook downsampling all affect latency and context consumption.

    The second is the choice between continuous and discrete representations. Continuous embeddings preserve detail for understanding. Discrete indices support generation and compact transmission, but reconstruction errors may appear as blurred small text, muffled sibilants, or artifacts in background noise.

    The third is the layer at which information becomes semantic. Shallow layers mainly preserve local statistics. Asking a model to read fine table lines while using very large patches means the evidence may already have been discarded during encoding. Later layers cannot recover information that was never preserved.

    The fourth is caching and alignment. Whether repeated questions about the same content trigger re-encoding, and whether multiple images or audio segments overwrite one another, affects stability as much as single-request accuracy.

    In practice, the architecture should be selected against an explicit task budget rather than treated as a search for one universally correct design.

    For understanding-focused systems:

    • Prefer continuous image patches or continuous acoustic frames.
    • Limit the longest image dimension and control audio frame rates.
    • Ensure that important details remain inside individual patches or frames.
    • Use invalidatable visual caches for static content.

    For generation-focused systems:

    • Use a codebook with sufficient capacity and controlled collapse.
    • Define the index rate explicitly.
    • Inspect reconstructed samples to verify that small objects and short phonetic events remain distinguishable.

    For long conversations:

    • Separate modality-specific caches.
    • Give text, image, and audio their own eviction policies.
    • Use fixed positional conventions and causal visibility rules when concatenating sequences.

    Evaluation should consider both task metrics and token usage. Optimizing accuracy while ignoring sequence expansion can hide the main source of cost and latency.

    Conclusion

    Turning images and audio into tokens is not the creation of a mysterious second tokenizer. It is the conversion of continuous samples into position-aware vector sequences, optionally followed by quantization into a finite set of IDs.

    Patch extraction and projection are intentionally simple. Semantics emerge through multiple layers of attention as the receptive information range expands from local neighborhoods to global structure. Discrete codebooks use a bottleneck to exchange detail for predictable symbols.

    The speed, fidelity, and relevance users experience can usually be traced to four factors: patch size, frame density, quantization strength, and whether caches are separated by modality. Making these constraints explicit in the system design is more useful for stabilizing multimodal behavior than simply adopting a new set of terminology.