Native-Resolution Encoding and Parallel Scheduling in Multimodal Systems: Implementation Methods

    This article explains how to choose between native-resolution visual encoding and fixed-size preprocessing in multimodal systems, and when to use serial versus parallel task execution. It covers the cost of fixed resizing, patch packing with positional encoding, cu_seqlens offsets, and block-diagonal attention masks, common batch-mask errors, and video tubelet tokenization. A selection table maps image types and task dependencies to recommended approaches and risks. Readers gain practical criteria for building a reproducible evaluation set first, then choosing encoding and scheduling strategies based on image characteristics and dependency structure.

    Multimodal agents commonly encounter two classes of problems. The first is perception. Input images vary widely in aspect ratio, resolution, and text density. Fixed-size preprocessing struggles to preserve readability for small-font screenshots, dense tables, and extreme long images at the same time. Resizing can blur details, and cropping can remove critical regions.

    The second class is execution. When a task requires retrieving information, reading tables, writing code, and validating results, executing every step sequentially makes total latency equal to the sum of step latencies. The more steps there are, the longer users wait.

    This article discusses two approaches. The first is native-resolution visual encoding, which aims to preserve as much of the original visual input as possible and supports the organization of video spatiotemporal blocks. The second is a task dependency graph with parallel scheduling, which allows subtasks without dependencies to execute at the same time. The two approaches can be used separately or together.

    The following table can be used as a selection reference.

    Scenario Recommended Approach Applicable Conditions Risks to Evaluate
    Small-font screenshots, dense tables, images with extreme aspect ratios Native-resolution encoding GPU memory can accommodate the target maximum token count Increased GPU memory use and latency; upper limits need to be set
    Ordinary natural photos Fixed-size preprocessing Low detail requirements; throughput takes priority Small objects or fine text may be lost
    Strong dependencies between subtasks Serial execution A later step must use the output of an earlier step Parallelism provides no benefit and adds coordination cost
    Independent subtasks such as retrieval, table reading, and data lookup Parallel scheduling Resources are sufficient, and concurrency does not exceed the limit Rate limits, timeouts, and partial failures need to be handled
    No reproducible evaluation set Build an evaluation set before optimization None Without a benchmark, conclusions about speed and accuracy cannot be verified

    The decision order is: build an evaluation set first, then inspect image types, and finally examine dependencies between tasks. This avoids choosing an approach by intuition.

    2. Native-Resolution Visual Encoding

    2.1 The Cost of Fixed Resizing

    Many vision models require fixed input sizes, such as 224×224 or 336×336. Details lost during preprocessing cannot be recovered by downstream models. Consider an image that is 800 pixels high and 100 pixels wide. When compressed to 336×336, roughly 2.4 original pixel rows in the vertical direction are merged into one row, and text strokes become visibly distorted. If the input is enlarged to 672×672 to preserve details, the number of tokens is about four times the original, and the attention computation is about sixteen times.

    2.2 Patch Packing and Batch Masks

    Cut an image into fixed-size patches, flatten them into a one-dimensional sequence, and pass the sequence to a Transformer. For an image of height H and width W with patch size p, the number of patches is:

    N = ceil(H/p) × ceil(W/p)

    N can differ across images, so patches from multiple images can be concatenated into one long sequence to reduce padding waste.

    To distinguish each image, three types of information are needed.

    The first is positional encoding. Each patch records its two-dimensional coordinates in its own original image. The coordinates are computed from the original image size, not from the index in the concatenated sequence.

    The second is image boundaries. Use a cumulative offset array such as cu_seqlens to record the start and end positions of each image in the long sequence.

    The third is an attention mask. Patches within the same image can attend to each other, while patches from different images cannot. This mask is called a block-diagonal mask. If the batch also contains padding positions, those padding positions must also be masked out.

    Common errors include the following. Offset misalignment can cause one image to read positional information from the previous image. If padding positions are not masked, padded positions participate in attention computation and shift the output values. Cross-image attention leakage can make the result for a single image depend on other images in the same batch. The batch consistency checks in Section 4 can detect these issues.

    2.3 Video Spatiotemporal Blocks

    Video can be viewed as a combination of temporal and spatial dimensions. One approach is to merge consecutive t frames into a spatiotemporal block (tubelet), and each block is split into p×p patches in space. The number of video tokens is:

    N_video = ceil(T/t) × ceil(H/p) × ceil(W/p)

    where T is the number of frames, t is the temporal block length, and H and W are