Self-Hosted Multimodal Models: Building the Missing Video Input Pipeline

    Most self-hosted multimodal LLMs accept text and images but fail on video, even though video understanding usually reuses the same visual encoder weights. The gap is in the serving path, not the model. This article walks through the four missing pieces — video decoding and frame sampling, temporal chunking, time information injection, and request/parameter adaptation — plus context budgeting, failure handling, observability, and acceptance testing needed to turn them into a production-ready service.

    1. Where the Gap Actually Is: The Ingress Layer, Not the Weights

    Most teams that self-host a multimodal LLM run into the same situation: text chat works, image understanding works, and the moment you switch to video input the request cannot be processed properly. The usual reason is that the adapted serving path only covers text and images. It has no video reading, decoding, or frame sampling on the preprocessing side, and no multi-frame visual encoding or duration-related positional handling either. As a result, whatever video understanding the model has never shows up in a self-hosted deployment.

    That kind of gap is easy to misread as a reason to swap models or modify weights. In reality, video understanding usually reuses the same visual encoding weights as image understanding. The difference sits almost entirely in how the data is organized into model input.

    This article uses a locally deployed video understanding capability as the worked example and splits it into four parts:

    • Ingress layer: reads, decodes, and samples the video, turning it into an ordered frame sequence;
    • Fusion layer: handles multi-frame visual encoding, feature projection, and temporal information propagation;
    • Serving layer: handles request adaptation, parameter constraints, and caching;
    • Deployment layer: packages these capabilities into a reproducible, operable inference service.

    Breaking it down this way turns a vague “missing capability” into engineering problems that can be located and verified.

    2. How Three Modalities End Up in One Sequence

    Before getting into implementation, one question needs answering: how does data from different modalities actually enter the same language model?

    2.1 The Unified Entry Point: A Token Sequence

    A large language model only ever receives a token sequence. Text is tokenized and mapped to word vectors. Images and videos are first encoded into visual features, then projected into the language model’s hidden dimension and inserted into the sequence as visual vectors.

    Those vectors correspond one-to-one with placeholders in the prompt — for example, the <video> marker in “What event does the following video describe?” The placeholder is expanded into the visual vectors for that video, so the model ends up seeing a single sequence. (Feeding image features into a text model, as text-to-image pipelines do, is a different approach; this article is about visual input in native multimodal models.)

    2.2 Three Branches

    • Text: tokenize, then look up embeddings;
    • Image: read and decode → resize and pad, normalize → split into patches → visual encoding → projection;
    • Video: decode into a frame sequence → temporal sampling and chunking → then share the same preprocessing, encoding, and projection as images.

    What the video branch adds is the organization step that happens before the frames reach the visual encoder. The sampling strategy decides what the model sees; chunking and time markers decide whether the model can tell what happened first and what happened after.

    3. Four Gaps and Their Implementations

    In a typical locally adapted version, the text and image branches already have image preprocessing, visual encoding, feature projection, and answer generation. What’s missing for video ingress is four pieces.

    3.1 Gap One: Video Reading and Decoding

    The service needs to support multiple container formats and codecs, and the input should at minimum cover remote URLs, files in object storage, and local files. In the implementation, keep IO, demuxing, and frame decoding decoupled: the IO layer handles fetching and retries, the decoding layer only cares about producing frame data. That makes it easy to scale out and to swap backends.

    Points worth attention in practice:

    • Rotation and cropping: some videos carry display orientation metadata such as rotation. Without normalizing it, the picture comes out sideways or upside down;
    • Pixel format: convert everything to RGB before preprocessing to avoid channel-order mismatches;
    • High-framerate long videos: to keep memory and CPU costs under control, cap the maximum resolution, per-frame decode size, maximum file size, and maximum duration, and validate all of them explicitly at the request layer.

    Decoding is CPU-intensive, so run it asynchronously with a timeout rather than letting it stall the inference threads.

    3.2 Gap Two: Frame Sampling and Temporal Chunking

    Frame sampling is sampling along the time axis. There are three common strategies:

    • Uniform sampling: take frames at fixed intervals. Simple to implement, with even time coverage;
    • Sampling to a target frame count: derive the sampling interval from total duration and frame rate. The output frame count is stable, which makes visual token cost predictable;
    • Keyframe-first: prioritize frames where the scene changes noticeably. It cuts redundancy but costs more compute.

    Once the strategy is chosen, split the sampled frames in time order into chunks, where each chunk is a short, temporally contiguous segment. This helps in two ways: chunks can be encoded in parallel, and they give you boundaries to attach time positional information to later.

    If you use multiple chunks, keep the timestamp for each segment as well, and state in the prompt that there are N segments ordered from earliest to latest, so the model knows the ordering relationship between frames.

    3.3 Gap Three: Multi-Frame Visual Encoding

    Single-frame encoding typically cuts one image into patches, runs them through the visual encoder and feature projection, and yields a set of visual vectors. The multi-frame case repeats that process across the frame dimension, and two things matter:

    • Frame independence: each frame is encoded separately, so temporal order is preserved naturally and the implementation stays simple and easy to extend;
    • Time information injection: encode each frame’s (or each chunk’s) timestamp as time positional information and add or concatenate it with the visual features, so the model can tell which frame it is and when it occurred.

    Without explicit time information, the model just sees an unordered pile of visual vectors and will struggle with questions like “which action happened first” or “how long did it last.”

    After encoding, the per-frame visual vectors and the text vectors are concatenated into one unified sequence according to placeholder position. Where the video placeholder sits in the prompt determines the visual information’s position in the context.

    Illustrative implementation (pseudocode, flow only):

    1. # 1) Decode and sample: inputs are the video source and sampling parameters
    2. frames, timestamps = sample_frames(video_source, max_frames=32, strategy="uniform")
    3. # frames: List[Image], in original temporal order
    4. # timestamps: List[float], in seconds
    5. # 2) Chunk: split segments in time order for batched encoding and time injection
    6. chunks = split_by_time(frames, timestamps, chunk_size=8)
    7. # 3) Encode chunk by chunk: reuse the single-frame preprocessing, visual encoder and projector
    8. chunk_features = []
    9. for frames_in_chunk, ts_in_chunk in zip(chunks.frames, chunks.timestamps):
    10. feats = [projector(visual_encoder(preprocess(f))) for f in frames_in_chunk]
    11. # Inject time information so the model perceives relative and absolute frame timing
    12. feats = [f + time_encoding(ts) for f, ts in zip(feats, ts_in_chunk)]
    13. chunk_features.extend(feats)
    14. # 4) Concatenate with text vectors by placeholder into one unified sequence
    15. messages = build_prompt(system="You are a video understanding assistant", video=chunk_features, question=user_input)

    3.4 Gap Four: Inference Request and Placeholder Adaptation

    The inference service needs the request schema extended. Alongside the existing text messages and image input, it should accept a video field carrying the frame-sampling parameters.

    Those parameters need strict validation: target frame count, sampling strategy, whether time information is enabled, frames per chunk, and so on. They directly determine the number of visual tokens.

    The number of tokens a video occupies in context is roughly proportional to “frame count × visual tokens per frame,” and has no direct relation to resolution or the encoded input format. A rough calculation:

    1. visual tokens per frame ≈ (frame height / patch height) × (frame width / patch width)
    2. total visual tokens ≈ frame count × visual tokens per frame

    For example, with a patch size of 14×14 and a maximum supported side length of 448 in the visual encoder, a 448×448 frame after resize-and-pad is about 32×32 patches, so roughly 1024 visual tokens per frame. Thirty-two frames is in the 32K range of visual tokens; add text and output tokens and you can easily exceed the model’s context limit.

    So budget the tokens at the request layer:

    1. Read the video duration and resolution first;
    2. Combine the target frame count with the per-frame token count to estimate total visual tokens;
    3. If the estimate is over budget, lower the target frame count or scale down the resolution until it fits the context constraint.

    4. Engineering Notes: Turning the Pipeline into a Stable Service

    4.1 Performance and Concurrency

    Video preprocessing is a clear new cost: network download, demuxing, decoding, resizing, and encoding. Running it directly on the inference thread steals time from the GPU. Recommended:

    • Do video processing in a separate thread pool or worker process and send only the processed frames into inference;
    • Cache hot videos (popular assets, samples that get requested repeatedly) to avoid decoding them over and over;
    • Apply admission control based on visual tokens per request so long-video requests can’t exhaust GPU memory and context.

    4.2 Context Budget and Truncation Strategy

    You can’t sample long videos without limit, so define an explicit budget at the request layer: cap the visual tokens, and when exceeded, prefer dropping frames over failing outright. Return the frame count and time span actually used in the response so callers can judge coverage.

    4.3 Failure Handling and Observability

    The video path has more failure points than the text path: download failures, unsupported formats, decode exceptions, excessive duration. Recommended:

    • Return structured error codes and human-readable messages at each stage instead of making callers parse stack traces;
    • Log download time, decoded frame count, encoding time, and visual token count in logs and monitoring metrics so bottlenecks are easy to locate.

    5. Verification and Acceptance: Three Classes of Tests

    After the changes, three kinds of tests confirm whether video support actually works end to end:

    1. Temporal questions: ask what happened first and what happened after, to check that time information is really injected;
    2. Content questions: ask about objects, people, and actions in the frames, to check that visual encoding and projection are correctly reused;
    3. Long-video questions: ask about information spanning multiple chunks of a longer video, to check that chunking and budget control take effect.

    Also run regression comparisons against text-only and image-only requests to confirm video ingress hasn’t disturbed the existing paths.

    6. Deployment: From Code to a Production Service

    Organize the pipeline above into a reproducible deployment form. Typically this includes:

    • Configuration switches: enable video input via a parameter, with explicit control over the default frame count and maximum frame count;
    • Resource constraints: define a per-request video processing timeout and memory limit;
    • Containerized delivery: package preprocessing, encoding, and the service process together, with object storage for assets and logging and monitoring services for pipeline visibility, so the deployment can scale horizontally in production.

    One thing to stress: video ingress adds new load on the CPU and IO side, so when scaling out, evaluate the resource ratio between video processing and GPU inference together. Otherwise one side becomes the bottleneck.

    7. Summary

    Self-hosting video understanding comes down to applying the unified abstraction of multimodal input to real engineering: video is first decoded into an ordered frame sequence, then routed through the existing image preprocessing, encoding, and projection, and finally fed into the same sequence via time information and budget control.

    The four gaps map to four tasks: decode and sample frames, temporal chunking, time information injection, and request and parameter adaptation. Once they’re filled in, the video understanding of a local service can match the official API, rather than sitting in a state where “the model supports video but the service doesn’t.”