Full-Layer Dynamic Routing: Connecting ViT and LLM Visual Pathways

    Multimodal LLMs often fail not because they cannot reason, but because the visual evidence reaching the language model is poorly matched to the question. This article analyzes the structural bottlenecks of a single visual entry point, presents full-layer dynamic routing as a way for each language layer to query all visual layers with context-dependent gating, and covers implementation logic, experimental signals, deployment considerations, and comparisons with common visual access designs.

    1. From Menu Pricing to Multimodal Pipelines: Visual Evidence Mismatch Is More Hidden Than Reasoning Errors

    A common menu-pricing question appears to require only simple addition: the image lists the unit prices of two dishes, and the question asks for the total. A multimodal model may nevertheless misread one price, or even add a dish that was never ordered, producing a total that is clearly wrong. After improving the visual access method, the same model can read both correct prices and complete the calculation.

    The notable part of this case is not whether the model can do addition. It is whether the model has obtained visual evidence that matches the question and is reliable enough before it starts calculating. A typical multimodal LLM pipeline works as follows: an image first passes through a visual encoder to produce a set of visual features; then a projection or adapter module converts those features into a representation the language model can accept; finally, the language model performs understanding, reasoning, and answering. If the visual entry point keeps only one layer of features and applies only one projection, then no matter how many layers the downstream language model has, it starts from the same fixed visual representation.

    The problem is that the same image requires different visual granularity for different questions. Reading menu prices depends on local texture, character detail, and spatial position; judging whether a dish has been ordered requires object counting and alignment with a list; analyzing the historical significance of a painting relies more on high-level semantics, style, and global composition. If all tasks share the same visual entry point, the model can suffer a structural mismatch: it cannot get detail when detail is needed, and it may be disturbed by low-level noise when semantics are needed.

    Therefore, the key to a multimodal system is not only “reasoning more deeply” but also “retrieving evidence more accurately.” The layer selection and connection method for visual evidence are preconditions for reasoning quality.

    2. Four Structural Bottlenecks of a Single-Layer Visual Entry Point

    Common multimodal architectures often take only the last or penultimate layer of a Vision Transformer, apply one projection, and send the result to the LLM. This approach is simple and effective, but it exposes several bottlenecks in complex tasks.

    First, the information bottleneck. Different layers of a visual encoder encode different levels of granularity. Shallower layers usually retain more information about edges, colors, textures, and local positions; deeper layers tend to focus more on object categories, semantic relations, and global structure. Taking only the last layer may weaken details such as small text, thin lines, and local color differences; taking only a shallower layer may lack high-level semantics. The penultimate layer is a compromise, but it is still a fixed compromise.

    Second, inter-layer semantic mismatch. The language model is also layered internally: earlier layers focus more on lexical information, local collocations, and input alignment, while deeper layers focus more on task intent, context integration, and complex reasoning. If visual features are injected only once at the input stage, later language layers can only “recall” visual information indirectly through the language context. Even if the architecture allows layer-by-layer access to visual features, always accessing the same fixed representation still prevents the evidence granularity from adjusting to the language layer’s state.

    Third, objective conflict caused by task heterogeneity. A single projection module must serve very different tasks at the same time: text recognition, color judgment, action understanding, object counting, style analysis, and more. Different tasks may prefer opposite visual layers. Forcing one representation to handle everything easily produces an “averaged” representation that works a little for every task but is precise for none.

    Fourth, static connections lack sample adaptivity. Past improvements typically include selecting an intermediate layer, fusing multiple layers, or adding adapters. These methods can alleviate information loss, but most connection relationships are already determined before entering the language model, or are fixed only at the global level. They do not fully exploit two conditions: what the current question is, and what semantic stage the current language layer is in.

    Common remedies fall into roughly three categories: choosing a better single layer, fusing multiple layers in advance, and inserting adapters into the language model. Each has value, but they share a common problem: the connection relationship between visual layers and language layers remains preset, static, or only coarsely adjusted. This leaves design space for full-layer dynamic routing.

    3. Core Idea: From “Which Layer to Select” to “Which Layer to Connect To”

    The core of full-layer dynamic routing is not simply averaging or stacking more visual layers. It reframes the problem as: which layer of visual information should connect to which layer of the language model, and with what weight?

    This brings two changes in perspective.

    First, visual representations are hierarchical, and language representations are also hierarchical. Past discussion focused mostly on “whether to use the last layer, the penultimate layer, or a fusion of multiple intermediate layers,” with attention on “which visual layer to select.” A more complete view is that the visual side has multiple layers of features and the language side also has multiple layers of states. What really needs design is the connection relationship between the two hierarchies. In other words, the problem moves from “layer selection” to “wiring.”

    Second, the connection relationship should not be fully predetermined. It should be learnable and change dynamically with the language layer, task, and sample. Given an image and a question, every language layer can access all visual layers; a gating module assigns fusion weights to different visual layers based on the current image-question context and that language layer’s semantic state. In this way, when the same model handles a text-dense question, it can assign higher weight to visual layers that preserve detail; when handling a semantic analysis question, it can assign higher weight to high-level semantic features.

    In one sentence: do not simply average more visual layers; turn the routing from visual layers to language layers into a dynamic, learnable, context-dependent selection problem.

    4. Method Breakdown: How Full-Layer Access and Dynamic Gating Work Together

    Conceptually, full-layer dynamic routing includes three key steps: retaining multi-level visual features, generating a query at each language layer, and aggregating visual evidence by weight. The concrete implementation varies with the base architecture, but the overall logic can be abstracted as follows.

    1. Vision side: retain multi-layer features instead of a single exit

    The visual encoder itself does not need to be modified. After the image is input, visual features from multiple depths are retained to form a set of visual layers. Each layer’s features can be mapped to a dimension that the language model can fuse through a lightweight projection or normalization. The key point is that different visual layers are no longer compressed into a single entry point in advance; they are kept as candidate evidence.

    2. Language side: each language layer issues a query based on its own state

    For language layer j, take its current hidden state and combine it with information such as the question context and a global image summary to compute that layer’s relevance scores for each visual layer. It can be understood as: whatever semantics the current language layer is processing, it queries the visual side for evidence at the corresponding granularity.

    3. Gated aggregation: weighted fusion, not full concatenation

    Let the visual layer features be Fi, the language layer state be h_j, and the context be c. A gating module can compute relevance scores, normalize them across the visual-layer dimension to obtain weights α{j,i}, and finally aggregate the visual representation by weight:

    1. for each language layer j:
    2. scores = gate(h_j, question_context, visual_summary)
    3. weights = normalize(scores) # optionally sparsify
    4. visual_j = sum(weights[i] * project_i(visual_layer_i))
    5. h_j = fuse(h_j, visual_j)

    The pseudocode above is illustrative only and does not represent a specific interface or parameter setting. In engineering implementations, gating can be based on attention, feed-forward networks, or lightweight scorers; normalization can use soft normalization or sparse selection to control computation. What matters is that the weights are not globally fixed but vary with the language layer and the sample.

    4. Training objective: optimize routing jointly with the main task

    The routing module, projection module, and adaptation parts in the language model can be jointly optimized with the main task loss. If the base model is large, some parameters can be frozen and only lightweight gating and projection modules trained. Engineering work should pay attention to routing stability: if the gate keeps selecting only one layer for a long time, dynamic routing degenerates into a fixed entry point; if the weights are too uniform, it becomes simple averaging. Therefore, it is usually necessary to combine sparsity constraints, load balancing, entropy monitoring, or layer-wise regularization to observe whether the routing distribution is healthy. These are general engineering practices, not a fixed configuration for any specific implementation.

    5. What It Changes Is the Wiring, Not the Visual Encoder

    Full-layer dynamic routing is easily misunderstood as “adding another, larger visual module.” In fact, its focus is not redesigning the visual encoder or simply increasing the number of visual tokens. It changes how visual information enters the language model.

    Specifically, it preserves the feature-extraction capability of the original visual encoder and replaces the former one-time projection and fixed entry point with multi-layer accessible, per-layer queryable gated fusion. This has several direct implications.

    First, visual evidence can be called on demand. When a text-recognition task needs local detail, the language layer can obtain finer visual features; when a semantic-understanding task needs a global judgment, high-level features can receive greater weight.

    Second, language layers no longer passively receive the same representation. Each language layer can adjust visual evidence according to its own semantic state, alleviating the mismatch between “shallow layers needing alignment” and “deep layers needing reasoning.”

    Third, routing relationships have room for interpretability. For a given question, it is possible to observe which visual layers different language layers select. If a certain layer consistently attends to detail layers when reading prices and shifts to semantic layers when judging the scene, this routing map can help developers determine whether the visual evidence is reasonable.

    Fourth, the approach is architecture-compatible. It does not require overturning the existing visual encoder and can be used as a redesign idea for the vision-language connection layer, combining with common mechanisms such as projection, adapters, and cross-attention. Of course, compatibility does not mean zero cost; training and inference still need to consider memory, latency, and stability.

    6. What Experimental Signals Show—and What They Do Not

    Publicly reported results suggest that this type of full-layer dynamic routing achieves consistent gains across multimodal bases of multiple scales. On a 7B-scale base, the report states that all 11 metrics exceed the single-visual-entry baseline, with text-dense visual question answering improving by about 4.2 points, comprehensive multimodal evaluation by about 3.3 points, and both real-world and open-ended question answering metrics by about 3.1 points. Consistent gains were also observed on a 2B-scale base and other 7B-scale bases.

    The significance of these signals is that the gains do not appear only in a single task or a single model, suggesting that “dynamically routing visual evidence by layer and by sample” is a direction with general value. In particular, when improvements concentrate in text-dense, real-world, and comprehensive question-answering tasks, it is reasonable to infer that part of the benefit comes from more reliable acquisition of visual evidence rather than from purely stronger text reasoning.

    However, these results should not be overinterpreted. First, evaluation scores are affected by base-model capability, training data, resolution, visual encoder quality, and evaluation distribution; there is no guarantee that another model will obtain the same magnitude of improvement. Second, routing cannot recover information that does not exist in the image. If input resolution is too low, text is severely compressed, or the visual encoder itself does not retain the relevant features, gating can only reallocate existing evidence; it cannot create new evidence. Third, dynamic routing increases training and inference complexity and must be evaluated together with latency, memory, and throughput. Fourth, more accurate visual evidence does not automatically eliminate hallucination; grounding verification, answer consistency checks, and necessary external validation are still required.

    7. Engineering Deployment Checklist: From “It Runs” to “Stable and Usable”

    To practice full-layer dynamic routing in a real system, consider the following dimensions.

    Dimension Key question Recommendation
    Compute and memory Does full-layer access make feature caching and projection computation too large? Cache multi-layer visual features; use low-rank projection, grouped projection, or sparse gating; fuse only highly relevant layers
    Representation alignment Different visual layers have different dimensions and distributions Use layer normalization, independent projection, or grouped projection; avoid sharing one uncalibrated projection across all layers
    Routing stability Do weights collapse to a single layer, or degenerate into uniform averaging? Monitor routing entropy and layer distribution; add load balancing or sparsity constraints; ablate fixed layers versus dynamic layers
    Training strategy End-to-end training is costly and may disrupt existing alignment Train projection and gating first, then gradually unfreeze some language layers; use mixed data covering both detail and semantics
    Inference efficiency Does per-layer gating affect batch throughput? Precompute visual features; aggregate in batches; use low-precision or lightweight gating scores; measure actual latency
    Evaluation system Looking only at aggregate scores can hide evidence-selection problems Evaluate separately for text recognition, local attributes, object counting, and global semantics; add routing visualization and error analysis
    Safety and reliability Visual evidence errors may be amplified by the language model Add OCR/object verification, answer consistency checks, and uncertainty warnings; do not treat routing as a standalone cure for hallucination

    In addition, evaluation should specifically compare four types of baselines: fixed last layer, fixed penultimate layer, multi-layer early fusion, and per-layer adaptation with a fixed visual entry point. Only by separating “more parameters,” “more visual tokens,” “more complex fusion,” and “dynamic routing itself” can we determine where the gains come from.

    8. Comparison with Common Visual Access Approaches

    Approach How vision enters the LLM Adjustability at language layers Main advantages Main limitations
    Single-layer one-time projection Take the last or penultimate layer and project once Basically none Simple and low cost Difficult to balance detail and semantics; inter-layer mismatch
    Multi-layer early fusion Concatenate or weight before entering the LLM Static Retains more information High redundancy; cannot adjust per language layer
    Fixed layer selection Choose one visual layer for a task or model Globally fixed Straightforward to implement Weak sample adaptivity; difficult to switch tasks
    Per-layer adapters Insert an adapter at each layer, but the visual representation is fixed Some adjustment Good parameter efficiency Visual granularity remains fixed
    Full-layer dynamic routing Each language layer accesses all visual layers and dynamically weights them Varies by layer and sample Adjustable evidence granularity; interpretable routing Higher training and inference complexity

    The core conclusion from this table is that full-layer dynamic routing is not “more layers are always better.” It turns the vision-language connection from a fixed configuration into a learnable one. Its value comes from matching relationships, not from parameter stacking.

    9. Conclusion: The Next Design Space for Multimodal Architectures Is the Connection

    The capability bottleneck of multimodal LLMs is often not only in language-side reasoning. Whether the model has obtained visual evidence that matches the question before it calculates also determines final answer quality. A single-layer visual entry point compresses the needs of different granularities, different tasks, and different language layers into the same representation, which can easily cause detail loss, semantic mismatch, and rigid routing.

    Full-layer dynamic routing offers a more complete design idea: keep the visual encoder unchanged, let each language layer access all visual layers as needed, and dynamically allocate fusion weights based on the image-question context and the language layer’s semantic state. It moves the problem from “which visual layer to select” to “which language layer should a visual layer connect to,” expanding a predetermined connection into a learnable routing problem that varies with the sample.

    For developers, what matters is not a single score but this architectural view: a multimodal system should not only strengthen the encoder; it should also design the connection relationship between the two hierarchies. When visual evidence can reach the place where it is needed, layer by layer, task by task, and sample by sample, the language model’s reasoning has a more reliable foundation. Next, how to control the compute budget in dynamic routing, improve routing interpretability, extend to multi-image and video scenarios, and coordinate with other vision-language alignment mechanisms will be engineering questions worth continued exploration.