How Multimodal Models Encode Images and Audio as Tokens
Multimodal models must convert continuous signals such as pixels and waveforms into manageable token sequences that can be processed by a shared sequence-modeling architecture. This article explains the main image and audio tokenization methods, including patch embeddings, discrete codebooks, temporal alignment, positional encoding, sequence compression, and the engineering trade-offs between fidelity, compression, and learnability.
In multimodal large language models, text is commonly represented as discrete tokens produced from a vocabulary and mapped to embeddings. Images and audio, by contrast, are continuous, high-dimensional signals. To process all modalities within the same sequence-modeling framework, the central challenge is to convert pixel matrices and waveforms into token sequences with controllable lengths and learnable semantics.
This article examines the most common engineering approaches for images and audio, along with key misconceptions and design trade-offs.
1. How Images Become Tokens
Image tokenization does not depend on handcrafted features from traditional computer vision pipelines. Most modern approaches fall into two broad categories: continuous patch embeddings and discrete visual codebooks. They serve different purposes. Patch embeddings prioritize end-to-end training with attention layers, while discrete codebooks emphasize compression and generative modeling with discrete symbols.
1.1 Patch Embeddings: Variable-Length Visual Sequences
The basic idea is to divide an image into patches on a fixed grid. Each patch is flattened and passed through a linear projection to produce a vector matching the model’s hidden dimension.
For example, a 224 × 224 image divided into 16 × 16 patches produces a 14 × 14 grid, or 196 patches in total. Each RGB patch contains 16 × 16 × 3 = 768 raw values. A learnable projection matrix transforms these values into one token embedding.
The sequence length grows approximately with the square of the image resolution. Larger images and smaller patches produce more tokens, increasing the computational cost of attention. In production systems, this is often addressed with adaptive resolution, dynamic patching, or hierarchical downsampling to prevent excessively long sequences from overwhelming the model.
It is important to distinguish patchification from visual understanding. Patchification is essentially a matrix rearrangement, and the projection is a matrix multiplication. There is no explicit edge detection, shape matching, or handcrafted visual algorithm in this step.
Capabilities such as recognizing edges, textures, parts, and semantics emerge gradually through subsequent self-attention layers:
- Early layers tend to focus on local contrast and orientation.
- Middle layers begin to organize textures and co-occurring parts.
- Deeper layers form semantic relationships across regions.
A single patch may contain only half an ear or a small section of an outline and may not be recognizable in isolation. Global attention allows patches to exchange information and combine into a representation of the complete object.
1.2 Discrete Codebooks: A Limited Visual Vocabulary
Another approach first trains an encoder-decoder model and introduces a finite codebook at its bottleneck. The codebook may contain thousands or tens of thousands of vectors. The encoder maps local image regions to their nearest codebook entries and outputs a sequence of integer indices. The decoder then reconstructs the image from those indices.
This process is closer to text tokenization:
- The codebook functions as a visual vocabulary.
- Each codebook index functions as a token ID.
The main advantages are strong compression and compatibility with autoregressive generation and cross-modal alignment. The costs include quantization error, codebook collapse, and reconstruction distortion, all of which may require dedicated regularization and training techniques.
A narrower bottleneck produces a more abstract representation. Fine details then depend more heavily on the generation or decoding head to fill them in.
1.3 Practical Combinations and Trade-Offs
Continuous patch embeddings are well suited to understanding tasks such as classification, retrieval, and visual question answering because gradients can flow directly through the pixel-level projection.
Discrete codebooks are more suitable for generation, compression, and scenarios that require a unified vocabulary of discrete symbols.
The two approaches can also be combined. For example, a system may patchify an image and then quantize the patch representations, or downsample the image before applying patchification.
Regardless of the approach, positional information is essential. Two-dimensional separable positional encodings, relative position representations, and resolution-aware position encodings can preserve the original grid structure after the image has been serialized into a sequence.
Training must also account for aspect ratios, padding, and variable resolutions. Otherwise, the model may treat padded regions as valid visual content.
2. How Audio Becomes Tokens
Audio is a one-dimensional temporal signal with a high sampling rate and substantial redundancy. Common approaches similarly include continuous frame embeddings and discrete acoustic codebooks.
2.1 Framing and Time-Frequency Representations
The waveform is first divided into overlapping short frames, often lasting several tens of milliseconds. These frames can be projected directly in the time domain, or they can be transformed into a time-frequency representation such as a spectrogram.
A spectrogram can then be divided into patches along its time and frequency dimensions. The remaining process resembles image patchification. Because a spectrogram turns sound into a visual-like representation, many techniques from image processing can be reused.
The number of audio tokens grows approximately linearly with the duration of the input. Long recordings therefore require downsampling, cross-frame aggregation, or hierarchical encoding. Without these measures, the sequence can become too long for the available context window.
2.2 Discrete Acoustic Units
Vector quantization can map local acoustic features to entries in a finite codebook, producing an index sequence similar to phonemes or acoustic words.
This type of representation is useful for aligning audio with text, speech-to-text systems, and speech generation. Codebook size, frame shift, and the use of multiple layers of residual quantization jointly affect intelligibility, timbre fidelity, and bitrate.
An overly coarse representation may lose phonetic details. An overly fine representation, however, reduces the value of compression.
2.3 Special Considerations for Text Alignment
Speech tokens usually have a finer temporal resolution than text tokens. Alignment may therefore require interpolation, explicit alignment procedures, or objectives similar to connectionist temporal classification so that acoustic and textual sequences correspond at the semantic level.
Noise, changes in speaking rate, and silent segments should be handled with energy gating or learnable masks. Otherwise, irrelevant frames may consume a large portion of the context window.
3. Architectural Considerations for Multimodal Unification
When image, audio, and text tokens are concatenated into a shared context, the architecture typically needs modality-type embeddings, spatial or temporal positional encodings, and, when necessary, a compressor that maps long visual or acoustic sequences to a fixed token budget.
Attention does not inherently understand images or audio. It only computes similarity-weighted interactions between vectors. Differences between modalities must be shaped through the embedding space, positional information, and training objectives.
From a computational perspective, visual sequences often contain far more tokens than text sequences, making them a major source of latency and memory consumption. Production systems therefore commonly limit image resolution, use windowed attention, or apply compression across layers.
4. Common Misconceptions and Engineering Recommendations
Misconception 1: Patchification is the Same as Visual Understanding
Patchification only converts a two-dimensional signal into a sequence. Visual understanding emerges through stacked attention layers and the interactions among the patch representations.
Misconception 2: A Larger Codebook Is Always Better
An excessively large codebook can suffer from codebook collapse, while an overly small one can introduce significant distortion. The codebook size should match the scale and characteristics of the training data.
Misconception 3: Concatenating Modalities Automatically Aligns Them
Simply placing tokens from different modalities in the same context does not guarantee alignment. Without objectives such as contrastive learning, generative training, or instruction fine-tuning, the modality-specific representation spaces may remain separate.
In practice:
- Prefer continuous embeddings for understanding tasks that benefit from preserving direct gradient paths.
- Prefer discrete codebooks for generation and compression.
- Treat sequence length as a first-class engineering metric.
- Control latency through image resolution, audio frame shift, and hierarchical encoding.
- Evaluate not only downstream task performance, but also reconstruction quality, codebook utilization, and the proportion of ineffective tokens.
Ultimately, turning images and audio into tokens is not a mysterious “perception module.” It is a combination of patching or framing, linear or quantized projection, positional encoding, and representations that emerge through attention. The core engineering problem is to transform continuous signals into computable sequences while balancing sequence length, fidelity, and learnability.
