Multimodal Large Model Architecture and Implementation Practices

    This article presents an engineering guide to designing and deploying multimodal large models. It explains how to define capability boundaries, unify multimodal inputs with text-centric outputs, apply parameter-efficient adaptation, support bilingual instruction following, build evaluation and safety practices, manage inference and monitoring, and follow a practical rollout order.

    Multimodal large models place non-textual signals such as images, speech, video, and 3D point clouds into the same reasoning framework as natural-language instructions. The goal is to understand what is seen or heard, then return a controllable textual or structured answer. Engineering discussions usually focus on three questions: how to connect vision capabilities to an existing language model with a small incremental design, how to serve both Chinese and English instructions, and how to maintain stable performance across object recognition, scene description, and open-domain question answering. The following sections present a practical, implementation-oriented order without binding to a specific implementation.

    1. Define capability boundaries before selecting a model

    Before model selection, teams should define the task clearly instead of chasing parameter count. Capabilities can be divided into four layers.

    • Perception: Identify object categories, attributes, spatial relationships, and general scene context. For example, determine whether a device is present, its color, and where it is placed. If this layer fails, fluent generation adds no value.
    • Semantics: Align perception results with the instruction. If a user asks for the purpose of an object, the model should return a functional explanation rather than only repeating the visual appearance.
    • Knowledge: Supplement common sense when recognition is correct, but separate general knowledge from image evidence. Information that cannot be confirmed from the image should be stated as a general explanation rather than as an observation.
    • Generation: Produce output under constraints of language, style, and length. Writing an explanation, listing key points, and giving operational suggestions require different decoding preferences and should not be forced through the same prompt template.

    For evaluation, it is advisable to fix three types of samples: pure recognition, recognition plus purpose explanation, and instructions with negation or ambiguity. Judging only whether a poetic description reads smoothly can overstate real-world usability.

    2. Unified representation: multimodal inputs, textual outputs

    Most general-purpose approaches use a three-part structure: modality encoder, alignment module, and language model.

    A vision encoder divides an image into patches or regions and outputs a set of vectors. Speech and video can first be converted into frame sequences or segment sequences, then encoded by their respective encoders into similar vector representations. The alignment module maps these vectors into an embedding space accepted by the language model. Common methods include linear projection, small cross-attention layers, or a small number of learnable query vectors that compress visual tokens. The language model handles instruction understanding and answer generation.

    Token budget must be considered during design. If a high-resolution image is expanded into too many visual tokens, it competes with the textual context and increases latency and memory usage. In practice, teams usually downsample or compress first with queries, then decide whether to refine key regions based on the task. Video should not be fed in as full per-frame sequences; instead, sample frames over time and remove highly similar frames. Point clouds are better handled by voxelization or sampling a fixed number of local features to avoid uncontrolled sequence lengths.

    The input side can be multimodal, but the output side should remain text-centric. When image editing or speech synthesis is needed, generation results should be delegated to independent downstream modules rather than making the same decoder responsible for pixel-level generation of every modality. This keeps boundaries clear and makes it easier to replace components independently.

    3. Parameter-efficient adaptation rather than full retraining

    Adding vision capability to an existing language model through full-parameter fine-tuning is costly and can damage original text capabilities. A more common approach is to train only a small set of new parameters.

    Learnable prompts: Insert a set of trainable vectors before the input sequence so the model can absorb a new task format without modifying the main weights. Prompts should preferably be injected at multiple layers rather than only at the lowest layer; otherwise, deeper layers may fail to make effective use of visual information.

    Zero-initialized gating on attention side: A newly added cross-modal attention branch initially contributes close to zero, so it barely disturbs the original language model during early training. As the loss decreases, the gate gradually opens, allowing new knowledge to enter the residual stream. This strategy of first preserving old capabilities and then slowly writing in new capabilities is more stable than mixing everything from the beginning with a large learning rate.

    Linear-layer bias or low-rank increments: Only biases of selected linear layers or low-rank matrices are adjusted. The parameter count is much smaller than full weights, making instruction tuning feasible on multi-GPU single machines. Bias adjustment is often more sensitive than prompt-only training for out-of-distribution instructions and is suitable for a second stage.

    A practical two-stage workflow is as follows. In stage one, use image-text paired data to train the alignment module and new branches while freezing the main language model. The goal is to align visual tokens with the word space. In stage two, unfreeze a small number of adaptation parameters, use instruction data to learn “answer while looking at the image,” and mix in a certain proportion of pure text instructions to prevent the model from only chatting about images and losing general question-answering ability.

    The following training configuration is illustrative and only shows stage relationships; it does not represent a platform-specific interface:

    Stage one: Freeze the language backbone, train the projection layer and cross-modal branch, use a higher learning rate than stage two.

    Stage two: Unfreeze adaptation parameters, mix image-text instructions with pure text instructions, reduce the learning rate, and shorten warm-up.

    Validation: Separately track recognition accuracy, instruction-following rate, and the regression margin on pure-text benchmarks.

    If pure-text benchmarks drop noticeably, teams should reduce the proportion of visual data, strengthen gating constraints, or roll back to updating only the alignment layer.

    4. Bilingual capability and instruction following

    In Chinese-language scenarios, users often ask short, colloquial questions such as “What is this, and what is it used for?” The model must handle both language recognition and task recognition. Engineering design does not need to force every language to be perfectly covered by the same vocabulary, but it should ensure that the instruction language and answer language are configurable, that the model follows the user’s language by default, and that when training data is mainly English image-text pairs, Chinese instruction paraphrases are explicitly included to avoid the mismatch where the model understands the image but answers only in English.

    Instruction data should cover different granularity levels. Coarse granularity includes scene summaries; fine granularity includes attributes, counts, relative positions, and cases that cannot be determined. Negative samples are equally important: if an image is irrelevant to the question, the model should refuse to speculate rather than fabricate objects. For open-ended generation such as “write a poem,” evaluation criteria should be separated from factual question answering; poem fluency should not replace recognition accuracy.

    5. Data, evaluation, and safety

    On the data side, teams should prepare at least three non-overlapping sets: an alignment set, an instruction set, and a human sampling set. The alignment set emphasizes coverage of common objects and scenes. The instruction set emphasizes diverse question styles. The human sampling set should be constructed by business staff based on real tickets and should include blurry images, occlusion, multiple objects, and mixed Chinese-English instructions.

    Metrics should cover several dimensions: whether key objects are described correctly, whether nonexistent entities appear, whether answers miss the question, whether Chinese output is fluent, and whether latency and memory meet concurrency requirements. Automatic metrics can only serve as a filter; a fixed proportion of human review is still required before launch.

    For safety, multimodal models may read text, identity documents, and faces present in images. Production systems should detect and redact sensitive regions before input reaches the model and filter personal information on the output side. Medical or legal judgments that cannot be confirmed should be refused or redirected to human handling rather than presented as definitive conclusions. Logs should retain only necessary request identifiers and scores, avoiding long-term storage of raw images.

    6. Inference and engineering deployment

    During deployment, scale the encoder and the language backbone separately. Image preprocessing and feature extraction can be batched on GPUs, while language decoding can be scheduled according to request length. For interactive products, returning a short conclusion first and generating detailed explanations on demand can significantly improve perceived latency.

    Caching must be handled carefully. Visual features of the same image can be reused, but a complete answer cannot be reused if the instruction changes. Version management should record the combination of encoder, alignment layer, and language adaptation parameters. It is prohibited to update only one segment without regression testing pure-text capability.

    Monitoring should include at least four types of alerts: rising complaints about recognition, abnormal refusal rate, mismatch between output language and instruction language, and per-request token count or latency exceeding thresholds. After an alert, the preferred action is to replay the human sampling set rather than directly increasing generation temperature.

    Step one: Use a small-scale image-text pair set to verify whether alignment works, testing only “what is the main object in the image?”

    Step two: Add purpose explanations and Chinese instructions, and observe whether the model fabricates information.

    Step three: Open more trainable parameters such as biases or low-rank increments, and monitor text-capability regression.

    Step four: Integrate redaction, filtering, and human sampling, and only then consider longer sequences such as video, speech, or point clouds.

    Reversing this order often leads to the most common result: demo examples look impressive, but the system fails on a new batch of real images.

    The core of multimodal capability is not that “more parameters mean universal knowledge,” but whether evidence, instructions, and generation constraints are modeled separately. Designing trainable increments, bilingual instruction distributions, and refusal strategies clearly is more decisive for whether a system can enter production than simply piling up demo cases.