Multimodal Connectors in Vision-Language Models: Roles and Training Strategies

    This article explains where the connector sits between a vision encoder and a language model, and the six functions it performs. It shows why conflicting results on skipping the connector, scaling its training data, and tuning resolution can coexist. It also covers token budget design, data coverage, a statistics-based construction method, and an ablation plan for comparing connector designs.

    Multimodal Connectors in Vision-Language Models: Roles and Training Strategies

    In multimodal large language models, the vision encoder turns an image into a set of continuous vectors, while the language model only consumes token embeddings. Between the two, there is usually a small module. It may be a single linear layer, a multilayer perceptron, a pooling layer, or a query-based resampler. This module is often called a projector, adapter, or connector. It looks minor, yet it is frequently debated, questioned, and sometimes dismissed as optional.

    Debates usually center on three questions. First, can the connector be skipped? Second, does adding connector training data keep helping? Third, do resolution and data mixture matter more than connector structure? Viewed alone, the three claims seem to contradict each other. Viewed within the full training pipeline, they often describe the same system from different angles.

    1. Locating the Connector in the Pipeline

    A typical multimodal text generation pipeline can be abstracted as follows:

    1. Image
    2. -> Vision encoder
    3. -> Visual token sequence V ∈ R^(N×d_v)
    4. -> Connector / adapter layer
    5. -> Visual token sequence Z ∈ R^(K×d_l) consumable by the language model
    6. -> Concatenated with text tokens
    7. -> Language model
    8. -> Text, structured answers, or action instructions

    Here, `