Running Multimodal Models Locally: The Overlooked Projector Weight and a Four-Model Comparison
Local inference servers can chat fluently yet never actually receive images, and the cause is often a missing multimodal projector weight rather than the model itself. This article explains the three-part structure of a vision-language model, the two-weight loading logic, and the common pitfalls around projector pairing, quantization, and context budgeting. It then walks through a hands-on comparison of four anonymized model profiles across two tasks — document screenshot OCR and web screenshot-to-code — scoring them on recognition, generation, and engineering metrics. It closes with practical guidance on image preprocessing, VRAM and concurrency, post-processing validation, regression testing, and how to split work between local and cloud inference.
1. Where It Started: A Projector Weight That’s Easy to Overlook
I’ve been using a local inference framework to run text-only models for coding, and it worked well enough. But one question kept nagging at me: why can’t a locally hosted inference service read images?
After digging through the documentation, I found the problem wasn’t the model — it was the startup configuration. A single file was missing from the load path: the multimodal projector weight.
1.1 The three-part structure of a vision-language model
A working VLM service is typically assembled from three pieces:
Image → [Vision encoder] → [Projector/adapter] → [Language model] → Text(ViT-like) (MLP / fusion module) (LLM backbone)
- Vision encoder: Splits the image into patches and outputs a sequence of visual feature vectors.
- Projector layer: Maps those visual features into the language model’s embedding space so the LLM can “read” them. It must be trained as a pair with the backbone model and cannot be mixed across models.
- Language model: Image tokens are inserted into the prompt as placeholders, concatenated with ordinary text tokens, and generated autoregressively as one stream.
So loading is really a matter of two weights: the backbone weights plus the projector weights. Load only the backbone, and the service starts fine and chats fine — it just never receives an image. The startup parameters look roughly like this (parameter names differ across frameworks, so defer to your framework’s documentation):
<inference-server> \--model llm-weights.Q4_K_M.gguf \--mmproj vision-projector.F16.gguf \ # omit this = text-only mode--ctx-size 16384 \--host 0.0.0.0 --port 8080
1.2 Three pitfalls you’ll hit often
- Mismatched projector and backbone: Mixing across models or parameter scales produces fluent but meaningless output — smooth-looking gibberish.
- Poorly paired quantization: Running the backbone at 4-bit while keeping the projector at F16/BF16 is a common, workable combination, but the official released pairing should take precedence; mixing precisions yourself tends to lose accuracy.
- Context consumed by image tokens: A high-resolution image can expand into thousands of tokens once tiled. If
--ctx-sizeis still set for text-only use, it will truncate output outright or throw an error.
2. Test Design: Turning “It Feels Better” Into Comparable Metrics
Environment: A single 24GB-class consumer GPU; the same inference framework and context length throughout; decoding fixed at temperature=0 (greedy) for reproducibility; identical image preprocessing settings.
Models under test (anonymized to represent four capability profiles):
| Code | Capability orientation |
|---|---|
| Model A | Document-parsing specialist (tables/layout first) |
| Model B | General-purpose multimodal (strong semantics and instruction following) |
| Model C | Mid-size, efficiency-focused (speed/VRAM friendly) |
| Model D | Code-capability enhanced |
Two scenarios:
- Scenario 1: A benchmark-style comparison-table screenshot → structured Markdown (with emphasis on table reconstruction)
- Scenario 2: A self-built web app screenshot → runnable HTML/CSS (+ a little JS)
Scoring dimensions:
- Recognition side: character accuracy, table structure fidelity (row/column and merged-cell reconstruction, along the lines of TEDS-style metrics), small text and handwriting, stamp occlusion, multilingual mixed text, header/footer removal
- Generation side: runnability, DOM structure similarity, style fidelity (layout/spacing/color), reasonableness of completed interactions, code conciseness
- Engineering side: first-token latency, decode throughput, peak VRAM, latency jitter under concurrency
3. Scenario 1: Document Screenshot Recognition (OCR)
The takeaway up front: the parsing-specialist model clearly leads on structured reconstruction, the general model wins on semantic understanding, and the mid-size model wins on speed.
| Dimension | Model A | Model B | Model C | Model D |
|---|---|---|---|---|
| Plain text paragraphs | Good | Excellent | Good | Excellent |
| Multi-column table structure | Excellent | Fair | Fair | Good |
| Merged cells | Good | Weak | Weak | Fair |
| Small font (<10px) | Good | Fair | Weak | Fair |
| Handwriting | Fair | Good | Weak | Fair |
| Stamps/occlusion | Excellent | Fair | Fair | Fair |
| Multilingual mixed text | Excellent | Good | Fair | Good |
| Relative throughput | Fair | Fair | Excellent | Fair |
(The table shows illustrative ratings of relative capability orientation to explain the evaluation dimensions; it does not represent measured scores for any specific product.)
Typical failure modes (more useful for engineering than a total score):
- Merged cells flattened into ordinary cells, losing row/column semantics
- Misplaced numbers and units or superscripts/subscripts (e.g., “94.6” mismatched with its column header)
- Headers, page numbers, and footnotes bleeding into body paragraphs
- Handwriting silently replaced with a visually similar common character — it reads “smoothly” but is wrong, and is the hardest to catch automatically
- Stamped or signed-over regions silently skipped rather than flagged as missing
- Information loss when traditional and simplified Chinese are normalized into simplified output
Countermeasures: For table targets, force output as a standard Markdown table with a declared column count; catch row/column drift with an automatic “row/column count consistency check”; for uncertain regions, require the model to emit a [?] placeholder instead of guessing, making uncertainty explicit.
4. Scenario 2: Web Screenshot → Frontend Code
| Dimension | Model A | Model B | Model C | Model D |
|---|---|---|---|---|
| Runs on first try | Fair | Good | Fair | Excellent |
| Semantic structure | Fair | Good | Fair | Excellent |
| Style fidelity | Good | Fair | Weak | Excellent |
| Interaction logic completion | Weak | Good | Weak | Excellent |
| Code conciseness (lower is better) | Excellent | Fair | Good | Fair |
The pattern is clear: the document-specialist model is stronger at “reading,” the code-enhanced model is stronger at “writing,” and the general model strikes a balance between the two. If the pipeline allows only one model, the general-purpose type is the safer default; if routing is possible, splitting by task yields the biggest gains.
Prompts matter enormously. On the same image, two ways of asking produce visibly different results:
# Weak: the model improvises — long explanation + inline stylesWhat is this image? Help me write a page.# Strong: constrain the output contract and add a quality barOutput only one complete HTML file, no explanation.Requirements:1. Use semantic tags (header/nav/main/section/footer);2. Put styles in <style>, and manage the primary color and spacing with CSS variables;3. Use placeholders + alt text for images, and add aria attributes to interactive elements;4. Preserve the information hierarchy and whitespace ratio of the original image.
5. Putting It Into Production: VRAM, Context, and Quality Safeguards
5.1 Image preprocessing determines the token budget
Long-side scaling plus dynamic tiling is the core lever for controlling cost. A rule of thumb: for documents dense with small text, upscale, tile, recognize, then reassemble; for screenshot-style tasks, cap the number of tiles so image tokens don’t crowd out output space. Always declare the budget explicitly in your config:
vision:max_side: 1536tile_size: 768max_tiles: 6placeholder_tokens_per_tile: 256 # calibrate against the model in practicegeneration:ctx_size: 16384max_new_tokens: 4096
5.2 VRAM and concurrency
VRAM ≈ backbone weights + projector + KV cache + activation overhead. Image tokens inflate the KV cache in lockstep, so “high concurrency + large images” is the combination most prone to OOM. On a single card, serialize requests or cap concurrency at 1–2, and meet throughput needs by scaling out horizontally rather than raising per-card concurrency.
5.3 Post-processing validation (not optional)
- HTML: Validate tag closure, a single
h1, and no inline event handlers with a parser; then take a headless-rendered screenshot and compare it against the reference image. - Tables: Check consistency between row/column counts and cell counts; on mismatch, retry or degrade to plain text.
- OCR: Cross-check numbers/units with regex and dictionaries; double-sample key fields and flag them for human review when the two passes disagree.
5.4 Build a regression evaluation set
Lock down 50–100 real business screenshots (including edge cases: very dark, skewed, occluded, extra-long tables), and re-run them every time you change the model, quantization level, prompt, or preprocessing parameters, producing a comparison report. Without a regression set, any “it feels better now” is untrustworthy.
6. Dividing Work Between Local and Cloud
- Keep local: Sensitive data, offline requirements, narrow and high-frequency tasks (such as fixed-layout receipts/tables), and cost sensitivity per request.
- Put in the cloud: Complex layouts, extra-long documents, multi-round iterative generation tasks, and traffic with clear peaks.
- Architecture suggestion: Run a lightweight local model up front for classification and preprocessing, routing only difficult samples to a large cloud model — this can significantly lower average cost.
For supporting engineering capabilities, it’s reasonable to assemble a general set of building blocks: object storage to hold original images and generated artifacts, a container platform to host stateless inference services with elastic scaling, and logging plus monitoring/alerting to build dashboards for latency, failure rate, and quality spot-checks. Baidu AI Cloud, for example, offers general capabilities — model services, containers and object storage, logging and monitoring — that can support this layered design directly, so you don’t have to build the entire infrastructure stack from scratch.
7. Conclusion and Checklist
That overlooked projector weight is, at its core, the step that turns “multimodal” from an exclusive privilege of cloud APIs into a local option. Before going live, confirm each item:
- Whether the projector weight is an official pairing with the backbone model, and whether the quantization levels match
- Whether the context length has been recalculated to account for image tokens
- Whether the image preprocessing budget (size/tile count) is explicitly declared and has been stress-tested for peak VRAM
- Whether decoding parameters are fixed to guarantee reproducibility
- Whether the prompt constrains both the output format and the quality bar
- Whether parser-level post-processing validation and a failure-retry strategy are in place
- Whether a regression evaluation set and baseline report exist
- Whether the routing condition for “fall back to the cloud when local fails” has been defined
To sum up the selection logic in one sentence: for reading documents, look at structure fidelity; for writing code, look at runnability — keep the two pipelines separate when you can, and when you can’t, pick the general-purpose model.
