Local Multimodal Inference: Deployment and Evaluation Practice

    Local LLM serving often ships text chat first and leaves vision behind. This article explains how vision projection weights, preprocessing, and task-specific eval (OCR vs. screenshot-to-code) turn image understanding into a maintainable production capability.

    In local inference services, text chat is usually the first path that works. Vision is easier to miss. Many vision-language models still split into two parts after quantization: language-model weights for generation, and vision (or multimodal) projection weights that map image features into the language embedding space. If the service starts without loading the projection files, it is effectively “voice only, no camera”: the API accepts text but cannot parse images. Including the projection layer in startup parameters is the first step to enabling image understanding, not a trick unique to any one tool.

    Architecturally, a typical pipeline is: a vision encoder produces a feature map from the image; a projection layer maps those features to embeddings with the same dimension as text tokens; those embeddings are concatenated with the user prompt and fed to the language model for autoregressive generation. Projection precision, resolution policy, and image preprocessing (resize, crop, normalize) all affect quality. At deploy time, the projection must match the backbone version so dimensions and layer counts stay consistent; mismatches cause silent quality loss or hard errors. On memory, the projection and vision tower add several GB. A common pattern on 24 GB consumer GPUs is more aggressive quantization on the text side and higher precision on the vision side, trading throughput against fine detail.

    Evaluation should not stop at “can it see the image.” Split into two very different tasks. The first is document and table OCR: contrast tables, paper screenshots, receipts, or handwritten forms, with structured text or Markdown as the expected output. Watch cell alignment, numeric accuracy, mixed languages, stamps, and rotated text. The second is UI screenshots to front-end code: an application page as input, with layout, component hierarchy, and styles restored. Watch information architecture completeness, hallucinated buttons, and whether colors and spacing stay maintainable. The two tasks weight fine-grained recognition versus spatial layout differently; one prompt set rarely optimizes both.

    For OCR, freeze a single table screenshot and compare column alignment, header hierarchy, footnotes, and units. Common failures include concatenating numbers from adjacent columns, treating percent signs as noise, and dropping merged cells. Mitigations include raising the input-resolution cap, prompting for row/column output with placeholders for empty cells, and simple post-checks on column count. Stamps, handwriting, and low-resource languages need their own subsets so English print scores do not hide real gaps. For latency, pin hardware, batch size, and concurrency to 1; measure “image to Markdown” and “PDF page to Markdown” separately, and note jitter from adaptive resolution.

    Screenshot-to-page work stresses structure. Prefer in-house page captures to avoid copyright and data-contamination issues. Evaluate in three layers: element recall (titles, nav, forms present), layout fidelity (grid, spacing, responsive intent), and runnable code (opens in a browser without obvious errors). Models often treat decorative shadows as new components or guess icon fonts as wrong characters. Constrain the prompt to a single HTML file, no invented backend APIs, and semantic tags. If one-shot generation is unstable, first emit a structured wireframe, then generate code in a second pass.

    Engineering practices that are tool-agnostic: (1) startup must explicitly set the projection-weight path, and health checks should send a tiny image to verify the vision path so production does not sit in text-only mode; (2) the request protocol must accept image bytes or a safe local reference, with max edge length and file size caps to avoid OOM; (3) logs should split vision-encode time from language-decode time to locate bottlenecks; (4) projection layers are more numerically sensitive—keep projection and vision-tower precision first, then compress the language model; (5) vision forward passes are often hard to batch, so rate-limit image requests separately so they do not stall text sessions.

    Security and compliance matter. Uploaded screenshots may contain IDs, bills, or internal dashboards; default to no disk persistence or short-lived encrypted storage, and instruct the model not to echo sensitive fields. Scan screenshot-to-code output statically to avoid inline malicious scripts. Do not use unauthorized third-party product UIs in eval sets, to limit copyright and trademark risk.

    Local multimodality is not something you get by swapping in a larger text model. It is engineering of backbone + projection + preprocessing + task-specific evaluation. Load the projection, self-check the vision path, and regress on fixed OCR and UI-restore samples so “seeing images” becomes a maintainable production capability. If the cloud side needs elasticity, object storage can hold samples and weights, and logging/monitoring can track vision latency and failure rates—as long as local and cloud share the same task split and acceptance criteria.