Deploying Multimodal LLMs on Non-NVIDIA Accelerators: A Practical Guide
A hands-on walkthrough for running a ~300B-parameter native multimodal open-source model on non-NVIDIA hardware, using a 4× Radeon AI PRO R9700 setup (32GB per card, 128GB total). It covers how to choose an accelerator based on ecosystem first, how to pick the right inference framework branch, how to select a quantization build, the often-overlooked mmproj projection layer, a 4×32GB memory budget breakdown, layer-splitting versus tensor parallelism, an eight-step deployment checklist with acceptance criteria, and common failure modes with fixes. The goal is a reproducible engineering process rather than an unconditional "it just works" claim.
1. First, Clear Up the Concepts: Native Multimodality vs. Two-Stage Inference
One of the most important shifts in large models over the past two years is that inputs are moving from plain text toward native multimodality. “Native multimodal” more precisely means that the model handles text, images, and video inputs in a unified way during both architecture design and training, rather than simply stitching several single-modality models together.
At the engineering level, however, many open-source models still take a two-stage path during inference: a vision encoder, a projection layer, and a language model. The vision encoder first turns an image into feature vectors, and the projection layer maps those features into the language model’s token-embedding space, so the language model can treat visual tokens much like text tokens. This implementation does not necessarily contradict “native multimodality” — it is simply one way current inference frameworks put the concept into practice. Keeping this distinction in mind avoids turning a conceptual debate into a deployment obstacle.
The reference setup in this article runs a native multimodal open-source model at roughly the 300B-parameter level on non-NVIDIA hardware. The case hardware is 4 Radeon AI PRO R9700 cards, each with 32GB of memory, for 128GB total. Because the case does not use the NVIDIA ecosystem, this article uses the general term “non-NVIDIA accelerator,” and later adds notes on adapting to Ascend and domestic GPUs.
The usual motivations for deploying a multimodal model locally are three: data privacy and compliance, long-term cost control, and the ability to customize and self-host. But when it actually comes to deployment, the hardware and software ecosystem are often harder to deal with than the model itself. Many open-source inference frameworks claim support for multiple hardware platforms, yet new models, new operators, and new quantization formats are usually adapted first for a small number of backends. An open model does not mean it runs out of the box on the hardware you happen to have.
2. Choosing a Non-NVIDIA Accelerator: Check the Ecosystem Before the Specs
Non-NVIDIA hardware falls roughly into three categories:
- AMD Instinct / Radeon series: Usually built on the open-source ROCm compute stack. The ecosystem is relatively mature, which suits individual developers and workstation users who want to validate.
- Dedicated AI accelerators such as Huawei Ascend: Built on the CANN software stack and common in enterprise deployments, but support for model operators varies significantly across chip models and framework versions.
- Other domestic GPU inference cards: Iterating quickly in recent years, commonly with 32GB or 64GB per card, but software stacks, operator coverage, and multi-card communication libraries are still maturing.
When choosing a domestic or otherwise non-NVIDIA accelerator, it helps to check in this order:
- Drivers and base libraries: First confirm whether the vendor’s driver, firmware, and runtime libraries are recognized by the current operating system.
- Inference framework backend: Does the target framework provide a ROCm, CANN, vendor SDK, OpenCL, Vulkan, or other backend?
- Operator support: Multimodal models typically involve attention variants, normalization, RoPE, the vision encoder, and quantization/dequantization. A missing operator can force a fallback or cause an error.
- Quantization format compatibility: GGUF, AWQ, and GPTQ are supported to different degrees across backends, and low-bit quantization is especially prone to compatibility differences.
- Multi-card communication: Pipeline parallelism has lower communication demands, while tensor parallelism relies more heavily on collective communication libraries. Be cautious with heavy tensor parallelism over PCIe.
- Memory and bandwidth: Multimodal inputs significantly amplify peak memory usage during the Prefill phase, so you cannot judge capacity by static weight size alone.
If a title or business context insists on emphasizing “domestic accelerator card,” you should specify exactly which product — Ascend, Cambricon, Hygon, Moore Threads, or another — and give the corresponding driver, CANN/SDK, framework backend, and operator-adaptation steps. Otherwise, it is better to use the general term “non-NVIDIA accelerator” as this article does, so the title stays consistent with the case.
3. Inference Framework Selection: Picking the Right Branch Matters More Than the Framework
For a multi-card workstation with 32GB-class memory, a lightweight inference framework is usually the more realistic starting point. Taking llama.cpp — which supports GGUF, CPU/GPU hybrid loading, and multi-card inference — as an example, its validation cost on non-NVIDIA hardware is lower, and community material is relatively abundant.
When choosing a framework, focus on three things:
- Support for the target model’s architecture: Including the language model structure, the vision encoder structure, and the loading logic for the multimodal projection layer.
- Support for the target hardware backend: For example ROCm, CANN, a vendor’s private backend, or a general compute backend.
- The ability to pin a stable version: When a model has just been released, its support may not yet be merged into the main branch, so an adaptation branch or a fixed commit is needed.
A common reality: on the day a model is released, an official or community branch may already support it, but the framework’s main branch has not yet merged the changes. In that case, do not pull the latest main branch directly — prefer a branch or commit that has already been confirmed to support the target model.
Using llama.cpp as an example, a build might proceed like this:
git clone https://github.com/ggml-org/llama.cpp.gitcd llama.cppgit log --oneline --all | grep <model-name-or-keyword>git checkout <commit confirmed to support the target model>cmake -B build -DGGML_ROCM=ONcmake --build build --config Release -j
If the target hardware is not AMD but Ascend or another domestic GPU, you cannot simply reuse the ROCm parameters above. Instead, check whether the framework supports CANN, a vendor SDK, or a specified backend, and replace the build parameters according to the official documentation. If the target framework does not support that domestic card, forcing a compile will usually fail, or it will run only on the CPU and provide no benefit from the accelerator.
So the key to framework selection is not “which framework is most famous,” but “which branch can complete model loading, vision encoding, and text generation on your hardware.”
4. Choosing a Quantization Version: Low Bit-Width Is Not the Goal — Running Is
A complete FP16 set of weights for a 300B-parameter model typically needs hundreds of GB of memory, and no single-machine 4×32GB setup can load it directly. Quantization is therefore almost inevitable.
Quantization is not simply “represent every parameter in 2-bit.” Mainstream K-quant-style schemes generally use a mixed-precision approach: keeping higher precision for sensitive parts such as embeddings, output layers, and normalization layers, using lower bit-widths for most other weights, and reducing error through grouped scaling.
Dynamic low-bit quantization goes further by assigning precision according to each layer’s sensitivity. Compared with fixed-bit-width pure 2-bit schemes, dynamic quantization usually retains more critical information at the same overall size, at the cost of being slightly larger and having higher loading and compatibility requirements.
When choosing a quantization version, follow three principles:
- Prefer quantization builds released in sync with the framework branch.
- Look at benchmarks such as perplexity and multiple-choice accuracy from the official or community source, not just the file size.
- For precision-sensitive tasks such as code generation, structured extraction, and contract review, compare 4-bit versions; if memory allows, spending a few extra GB for better quality is usually worth it.
When a quantization version does not match the framework branch, common symptoms include: the model failing to load, an unknown layer type, garbled output, ignored visual inputs, or a mid-inference crash. In that case, check the quantization repository notes and the framework’s issues first, rather than repeatedly tweaking launch parameters.
5. The Key to Multimodal Deployment: Don’t Forget the mmproj Projection Layer
One of the most common pitfalls in multimodal model deployment is downloading only the language model weights and missing the multimodal projection layer file — the mmproj file.
The mmproj file projects the image features output by the vision encoder into the main model’s token-embedding space. The main model handles language understanding, reasoning, and text generation; mmproj is what allows image features to be consumed by the main model. Many frameworks require both the main model and mmproj when loading.
mmproj is usually saved in FP16 and is not large — for example, about 1.13GB — but it is the key component of multimodal capability. When mmproj is missing, the model may still generate text but cannot correctly understand images or video.
When downloading and verifying mmproj, check the following:
- File existence: Confirm whether the release package contains an
mmproj-*.ggufor similar file. - Version match: The main model’s quantization version and the mmproj version should preferably come from the same release page or the same model card.
- Successful load in the startup log: Normal logs usually show a message about loading the projection layer, vision encoder, or multimodal module.
- Whether the image input takes effect: Test a simple image question-answer first, then test complex images and video frames.
- Whether errors point to the projection layer: If you see errors such as
mmproj not found,failed to load projector,vision encoder unsupported, orunknown model architecture, check file matching and backend support first.
If the official mmproj is missing or mismatched, try these alternative paths:
- Re-download the matching mmproj version from the model’s official release page.
- Check the file hash to avoid a corrupted or incomplete download.
- Re-match with a lower-risk main model quantization version.
- Temporarily disable multimodal input and verify text inference only, to determine whether the problem comes from the projection layer or the vision encoder.
At the launch-command level, parameters differ across frameworks. Taking a framework that supports GGUF and llama-server as an example, a representative command looks like this:
llama-server \-m model-ud-q2-kl-xl.gguf \--mmproj mmproj-f16.gguf \-c 8192 \-ngl 99 \-fa on
For an actual deployment, always refer to the help documentation of the framework version you are using, and do not copy parameters from older versions verbatim.
6. Memory Budget: Is 4×32GB Enough? It Depends on Dynamic Usage
Whether memory is sufficient cannot be judged by weight size alone; you also need to account for the KV Cache, framework overhead, multimodal visual tokens, and runtime peaks.
The static usage of the reference setup is roughly as follows:
| Item | Approx. Size | Notes |
|---|---|---|
| Main model quantized weights | 108.7GB | Dynamic low-bit quantization |
| Multimodal projection layer | 1.13GB | FP16 |
| Static total | ~109.8GB | Excludes KV Cache |
| Available memory | 128GB | 4×32GB |
| Theoretical headroom | ~18GB | Tighter after subtracting actual overhead |
This 18GB of headroom is not free memory that can be used freely. It has to cover the KV Cache, compute buffers, vision-encoding peaks, framework context, and potential memory fragmentation.
The KV Cache can be roughly understood as:
KV Cache ≈ 2 × num_layers × num_kv_heads × head_dim × bytes_per_element × num_tokens × concurrency
Here, the 2 accounts for K and V; the number of KV heads is affected by grouped-query attention; and bytes per element depends on the precision used for the KV Cache. The longer the context, the higher the concurrency, and the more image tokens, the more noticeably the KV Cache grows.
Multimodal input consumes more memory than plain text, because a single high-resolution image may be split into several sub-images and produce many visual tokens after encoding. The Prefill phase can show a memory peak noticeably higher than the Decode phase. Video input, if it samples many frames, amplifies the pressure further.
For actual deployment, enable mmap or memory-mapped loading to avoid reading all weights into memory at once. If the framework supports CPU/GPU hybrid offloading, you can keep some layers on the CPU when memory is tight, trading speed for capacity.
7. Multi-Card Parallelism: Prefer Layer Splitting and Pipelines — Don’t Mistake It for Tensor Parallelism
When the model cannot fit on a single card, multi-card parallelism becomes unavoidable. There are two common approaches:
- Tensor parallelism: Splits the matrix operations of a single layer across multiple cards, requiring cross-card communication at every layer.
- Pipeline parallelism or layer splitting: Splits by layer, with each card handling a contiguous segment of layers, and requests flowing between cards by stage.
On a machine with primarily PCIe interconnect, tensor parallelism is easily limited by communication bandwidth. With every layer and every token communicating across cards, it can significantly increase time-to-first-token and reduce throughput. Pipeline parallelism’s communication mainly happens at stage boundaries, at a lower frequency, making it better suited to multi-card workstations.
A common misconception needs clarifying here: multi-card parameters in some frameworks may be misread as “tensor parallelism.” For example, some tools have separate parameters for tensor splitting, layer splitting, and device assignment — the names look similar but the semantics differ. When deploying, you cannot simply interpret a “4-card parameter” as tensor parallelism; you must consult the current version’s documentation to confirm whether it performs layer splitting, tensor splitting, or plain device assignment.
The reference setup is closer to “layer splitting + pipeline-style execution.” For the 4×32GB scenario, you can start with something like this:
# Representative command; parameter names may differ across frameworks, so defer to the current version's documentationllama-server \-m model-ud-q2-kl-xl.gguf \--mmproj mmproj-f16.gguf \-c 4096 \-ngl 999 \-fa on \--split-mode layer
If your framework has no --split-mode, look in the documentation for parameters related to layer, pipeline, split, tensor-split, or device, and use the logs to confirm which layers each card loaded. After starting, you can judge whether the parallelism strategy is working via the framework logs, device-monitoring tools, or throughput stress tests.
When stress testing, record four kinds of metrics:
- Time to first token: Measures interactive experience.
- Generation speed (tokens/s): Measures pure-text throughput.
- Multimodal time-to-first-image: Measures vision encoding and Prefill pressure.
- Memory peak: Indicates whether OOM might be triggered.
If tensor-parallelism communication is slow, try lowering the degree of parallelism, switching to layer splitting, or moving to a persistent service to reduce repeated loading overhead.
8. Deployment Checklist: Every Step Needs an Acceptance Criterion
When deploying this kind of model on non-NVIDIA hardware, follow this checklist:
- Confirm hardware and drivers
- Acceptance criterion: The system recognizes all 4 accelerators, and the device count appears in the framework logs.
- Confirm backend and operators
- Acceptance criterion: The framework enables the target GPU backend rather than falling back to the CPU.
- Pin the framework branch
- Acceptance criterion: Build with a fixed commit; the build succeeds and a sample model runs.
- Download the main model quantization weights
- Acceptance criterion: The files are complete, and the startup log shows the main model loaded successfully.
- Download and match mmproj
- Acceptance criterion: The log shows the projection layer loaded successfully, and image question-answer returns reasonable results.
- Configure context and parallelism
- Acceptance criterion: Short text generates, image input does not error, and the memory peak stays below available capacity.
- Stress test and tune
- Acceptance criterion: Record throughput and memory peaks across different contexts, concurrency levels, and image sizes.
- Run as a service
- Acceptance criterion: The model stays resident in memory to avoid reloading on every request, and the service is accessible via API or client.
9. Common Errors and How to Handle Them
- Only the main model is downloaded, mmproj is missing: Image input does not work; add the projection layer.
- Quantization version does not match the branch: Loading fails or output is abnormal; switch to a quantization version released in sync.
- Context set too large: KV Cache overflows; lower the context or concurrency.
- High-resolution images cause OOM: Compress images, cap the resolution, or reduce the number of visual tokens.
- Multi-card communication is slow: Switch to layer splitting and reduce reliance on tensor parallelism.
- First load takes a long time: Large models load slowly by nature; consider running as a persistent service.
- The framework does not recognize the domestic card: Check the driver, CANN/SDK, backend compile parameters, and operator support.
10. Conclusion: Feasible, but Only Under Preconditions
Provided that the memory, drivers, framework branch, quantization version, mmproj matching, and data-compliance prerequisites are met, deploying a 300B-parameter-class native multimodal model on 4×32GB non-NVIDIA accelerators is a practical engineering task. But it should not be described as unconditionally “fully feasible.” Differences among domestic cards, software stacks, and model versions are large, and each item must be validated one by one.
Looking at the trend, open-source inference frameworks continue to improve their support for multi-hardware backends, and the quantization toolchain is iterating quickly. For individual developers and small-to-medium teams, running multimodal large models locally is shifting from mere tinkering toward a reproducible engineering process. Even so, before deployment you should still confirm the model license, the source of the weights, file checksums, and how sensitive data is handled. For scenarios such as healthcare, legal, industrial quality inspection, and content moderation, keeping data inside the network, running offline, and customizing on demand are often more valuable than simply saving on API costs.
