Local Deployment of a 320B Multimodal Model on Four 32GB GPUs: Practices and Selection

    This article describes a practical local deployment of an open-source multimodal model with about 320 billion parameters on four 32 GB professional GPUs. It covers when local deployment makes sense, how to choose an inference branch and pin a commit, quantization and VRAM trade-offs, vision projector version matching, VRAM estimation formulas, tensor versus pipeline parallelism, deployment and smoke testing steps, a quantitative evaluation plan, common troubleshooting, and selection guidance for online APIs, single-machine deployment, and multi-GPU local deployment.

    1. Why Deploy a Large Multimodal Model Locally

    Most developers call models through online APIs. This keeps integration costs low and removes the need to manage the runtime environment. However, local deployment is necessary in several scenarios: business data cannot leave the internal network, network conditions are restricted or offline operation is required, the inference pipeline needs fine-grained control, and long-term high-frequency calls make unit cost a concern.

    Multimodal capability extends input from plain text to images and video. The language model backbone must work with visual encoding and projection stages. Therefore, the difficulty of local deployment is not limited to the language model itself; it also depends on whether the inference framework fully supports the vision branch.

    The case in this article uses an open-source multimodal model with about 320 billion parameters, running local inference on four 32 GB professional-grade GPUs. Both text and image inputs have been validated. Video input is not included in the test scope because framework support is not yet complete.

    2. Inference Branch and Version Selection

    Open-source inference frameworks often support new model architectures on a feature branch first, then merge support into the main branch after stabilization. If the main branch does not support the target model, you need to switch to a feature branch. Pay attention to the following points.

    First, confirm that the target branch contains the corresponding implementation for the model architecture, including model configuration parsing, weight loading, and multimodal input handling. Seeing the model name in documentation is not enough to prove that the functionality is usable.

    Second, pin a specific commit hash. Feature branches change frequently, and interfaces or weight formats may change within a short period. Recording the commit hash is the only way to reproduce the same environment later.

    Third, evaluate the following dimensions: completeness of model architecture support, whether multimodal input works end to end, multi-GPU parallelism capability, quantization format ecosystem, and rollback cost.

    Dimension Main stable branch Feature branch
    New model support May be missing Usually available earlier
    Stability Relatively high Requires practical validation
    Interface changes Less frequent May change frequently
    Suitable scenarios Production environments Validation and early trials

    3. Quantization: Trade-offs Between Accuracy and VRAM

    Weight quantization represents original high-precision floating-point values with fewer bits. Lower bit width reduces weight footprint, but usually increases accuracy loss.

    The basic idea of dynamic quantization is to assign different bit widths to different layers or weight blocks. Numerically sensitive parts keep higher precision, while more redundant parts are compressed further. The benefit of this approach comes from the allocation strategy, not merely from reducing the overall bit width.

    In the case, the weights of the 320B model are compressed to about 108.7 GB, averaging about 2.7 bits per weight. The estimation formula is:

    Weight size ≈ parameter count × average bits ÷ 8

    When evaluating a quantized version, follow these steps:

    • Use a test set relevant to the business and compare outputs from the original precision and the quantized version.
    • Focus on mathematical reasoning, code generation, and structured output, because these tasks are most sensitive to detail errors.
    • Do not look only at file size. A smaller version does not necessarily meet business quality requirements.

    4. Vision Projection Module

    A common multimodal model architecture has three parts. The vision encoder splits an image into patches and extracts features. The projection layer maps visual features into the language model’s embedding space. The language model then takes the concatenated sequence as input and generates text.

    Vision projection weights are usually stored separately from the main model weights. Both must come from the same release version. Otherwise, image features and the language model embedding space will be misaligned, which can appear as answers unrelated to the image content or obviously abnormal output.

    The projection module is small, about 1.13 GB in the case, and is usually stored in half precision. Its VRAM overhead is modest, and the benefit of keeping higher precision is greater than the space saved, so aggressive quantization is generally not recommended for this part.

    5. VRAM Capacity Estimation

    Total VRAM usage consists of the following parts:

    Total usage = weights + vision projector + KV cache + activation and communication buffers + system reserve

    The KV cache grows linearly with context length and concurrency. It can be calculated as:

    KV cache ≈ 2 × layers × KV heads × head dimension × context length × concurrency × bytes per element

    The following code shows an illustrative estimation function. Parameter values should follow the model configuration file in use:

    1. def estimate_weight_gb(params_b: float, avg_bits: float) -> float:
    2. """Estimate weight size in GB (decimal) from parameters in billions and average bits."""
    3. return params_b * avg_bits / 8
    4. def estimate_kv_cache_gb(layers: int, kv_heads: int, head_dim: int,
    5. ctx_len: int, batch: int, bytes_per_elem: int = 2) -> float:
    6. """Estimate KV cache usage in GB. The factor 2 accounts for K and V caches."""
    7. total_bytes = 2 * layers * kv_heads * head_dim * ctx_len * batch * bytes_per_elem
    8. return total_bytes / 1e9
    9. weights_gb = estimate_weight_gb(320, 2.72) # about 108.8
    10. projector_gb = 1.13
    11. print(f"Weights plus projector: {weights_gb + projector_gb:.1f} GB")

    In the case, the total VRAM of the four GPUs is about 119.4 GB. After subtracting weights and projector, about 9.6 GB remains. This margin must simultaneously hold the KV cache, intermediate computation results, and driver reservation, so it can support only relatively short contexts and low concurrency. When planning long-context scenarios, calculate the KV cache with the formula first, then decide the context length and concurrency configuration.

    Actual available VRAM is usually lower than the nominal value because drivers and runtimes occupy part of the space. Estimation should be based on the available VRAM measured in the actual runtime environment.

    6. Multi-GPU Parallelism Strategies

    Two split strategies are common in multi-GPU deployment.

    Tensor parallelism splits the computation of each layer across multiple GPUs. Each layer’s computation requires inter-GPU communication, so it is sensitive to interconnect bandwidth. When inter-GPU interconnect is good, it can reduce latency for a single request.

    Pipeline parallelism splits the model into several segments by layer, placing each segment on one GPU. Only activations are passed between segments. Communication volume is smaller and interconnect requirements are lower, but a single request must pass through each GPU in sequence, so concurrency scheduling is needed to improve overall utilization.

    The selection principle is: when inter-GPU interconnect is weak or high-speed interconnect is unavailable, prefer pipeline parallelism; when first-token latency is sensitive and interconnect conditions are good, evaluate tensor parallelism.

    The following is an illustrative configuration showing the categories of parameters that need to be determined. Specific parameter names should follow the documentation of the chosen inference framework:

    1. # Illustrative configuration; adjust parameter names according to the chosen framework's documentation.
    2. model:
    3. main_weights: ./weights/main-model.weights
    4. projector_weights: ./weights/vision-projector.weights
    5. runtime:
    6. parallel_strategy: pipeline
    7. devices: [0, 1, 2, 3]
    8. context_length: 8192
    9. max_concurrency: 2
    10. server:
    11. host: 127.0.0.1
    12. port: 8080

    7. Deployment Steps

    Step 1: Environment check. Confirm the driver version, runtime version, available VRAM per GPU, and inter-GPU topology. Use the inspection tools provided by the hardware vendor, and record the results for later troubleshooting.

    Step 2: Obtain and verify weights. The main model weights and vision projection weights must come from the same release version. After downloading, verify each file against the hash values provided on the release page to avoid file corruption or version mismatch.

    Step 3: Build the inference environment. Switch to the selected branch, pin the commit hash, compile according to the official instructions, and record the compilation options. It is recommended to write environment information into a version file and archive it together with the weight files.

    Step 4: Start the service. Start the service with the configuration and monitor VRAM changes on each GPU. The VRAM peak during model loading is usually higher than during stable operation, so the value after loading completes should not be used as the sole basis for capacity judgment.

    Step 5: API validation. First send a text-only request to confirm that the model responds normally. Then send an image-text request to confirm that the vision branch works correctly.

    The following is an example of text-only validation:

    1. import json
    2. import urllib.request
    3. def post_json(url: str, payload: dict, timeout: int = 120) -> dict:
    4. req = urllib.request.Request(
    5. url,
    6. data=json.dumps(payload, ensure_ascii=False).encode("utf-8"),
    7. headers={"Content-Type": "application/json"},
    8. method="POST",
    9. )
    10. with urllib.request.urlopen(req, timeout=timeout) as resp:
    11. return json.loads(resp.read().decode("utf-8"))
    12. def smoke_test_text(url: str) -> None:
    13. payload = {
    14. "messages": [{"role": "user", "content": "Introduce yourself in one sentence."}],
    15. "max_tokens": 64,
    16. }
    17. print(post_json(url, payload))
    18. smoke_test_text("http://127.0.0.1:8080/v1/chat/completions")

    For image input, the image must be encoded and placed in the content field of the request. The exact field structure depends on the API documentation of the service in use. This article does not provide an unconfirmed field format.

    8. Functional Test Plan

    Testing should cover three categories of capability, and a quantifiable evaluation set should be built for each category.

    Basic short-answer functionality. Design questions covering instruction following, factual question answering, format constraints, and simple reasoning. Provide a standard answer or judging rule for each question and calculate the pass rate instead of relying only on subjective impressions.

    Long-text retrieval capability. Insert a piece of key information at different positions in a long text, then ask a question about that information. Calculate the hit rate by position and text length to observe retrieval stability at different depths. This type of test is very sensitive to context length settings and should be cross-checked with VRAM estimation results.

    Image understanding capability. The test set can include text recognition, chart reading, object counting, and scene description. Prepare fixed images and standard answers for each task type, and ensure that the same batch of images is reused across versions for comparison.

    Video input is outside the scope of this article. If the business requires video understanding, a common alternative is to extract keyframes at fixed intervals and feed them as multiple images. The effectiveness of this approach must be validated separately and cannot be directly treated as equivalent to native video processing.

    The case author’s evaluation of multimodal results is positive, but the article does not provide quantitative metrics. Before adoption, readers should build an evaluation set based on their own business and make decisions with data rather than impressions.

    9. Common Troubleshooting

    Loading failure or startup error. In most cases, this is a version mismatch. Check whether the commit hash of the inference branch matches the model architecture requirements, and whether the weight files are complete.

    Image answers unrelated to the image. First check whether the vision projection weights and the main model come from the same version. Version mismatch is the most common cause of this symptom.

    Insufficient VRAM. Try the following measures in order: shorten the context length, reduce concurrency, and recalculate the KV cache with the formula. If VRAM is still insufficient, consider a lower-bit quantized version, but re-evaluate accuracy.

    One GPU has noticeably higher load. Check whether layer splitting is even. Different layers may have different VRAM usage, so splitting should be adjusted according to actual usage rather than simply dividing the number of layers evenly.

    High first-token latency. In pipeline parallelism, a single request must pass through each segment in sequence, so latency increases with the number of segments. You can reduce concurrency to lower queuing, or evaluate whether conditions allow switching to tensor parallelism.

    10. Selection Recommendations

    Different deployment approaches suit different scenarios. The following table can be used for an initial judgment:

    Dimension Online API Single-machine local deployment Multi-GPU local deployment
    Data privacy Data must be transmitted externally Data stays local Data stays local
    Operations complexity Low Medium High
    Hardware investment None Requires dedicated hardware Requires multi-GPU hardware and interconnect
    Functional completeness Depends on service provider Depends on framework support Depends on framework support
    Iteration speed Updated with the service Must track versions internally Must track versions internally

    If the team does not have multi-GPU hardware, or the model functionality is not yet fully supported in the target framework, prioritize online APIs or cloud GPU compute instances. If data privacy is a hard requirement and the team can bear the operations cost, then consider local deployment.

    The core work of local deployment comes down to three things: confirm the completeness of framework support, estimate VRAM with formulas and leave margin, and validate functionality with a quantifiable test set. Once these three points are implemented, local deployment of a large-parameter model on non-mainstream hardware can run stably.