Local LLM Deployment Selection and GPU Cluster Upgrade Practices

    This guide presents a practical framework for selecting and deploying local large language models on a three-node GPU cluster, with one 32 GB GPU per node. It covers resource assessment, model sizing, quantization, deployment topologies, shared-storage considerations, rolling upgrades, rollback planning, and post-upgrade validation. The focus is not on naming specific models, but on matching business requirements with GPU memory, model scale, concurrency, and operational constraints.

    1. A Three-Node GPU Cluster Upgrade Scenario

    In a typical on-premises LLM platform upgrade, a team may already be running an inference cluster on three virtual machines. Each node has one GPU with 32 GB of VRAM. One node acts as the control-plane or primary node, while the other two serve as worker nodes.

    Models, caches, and business data are mounted from shared storage under separate directories. The primary node uses one data directory, and each worker node uses its own directory. The existing platform is stable, but a newer release offers improvements in scheduling, GPU memory utilization, and compatibility with newer model architectures.

    This scenario is common in production. The team wants better model compatibility and resource efficiency, but three nodes with 32 GB GPUs provide limited capacity. A compatibility issue, failed metadata migration, or shared-storage mount problem during the upgrade could interrupt service.

    An upgrade should therefore not focus only on the features introduced by the new release. Model selection, VRAM planning, deployment topology, upgrade validation, and rollback procedures should be considered as one continuous process.

    This article does not list specific models. Instead, it provides a reusable framework for local LLM selection, cluster upgrades, and validation. Specific models can be evaluated against the matrix described here using a compliant model repository.

    2. Local LLM Recommendations Should Not Be a Fixed List

    The question “Which large language models can currently be deployed locally?” cannot be answered effectively with a few model names. Production usability depends on several factors:

    1. VRAM budget: Model weights, the KV cache, batching workspace, runtime overhead, and reserved system memory all determine whether a model can run reliably. A 32 GB GPU does not provide 32 GB entirely for model weights.
    2. Context length and concurrency: Long contexts and concurrent requests significantly increase KV-cache and scheduling overhead. A model that works with short prompts may become unstable under long-context, high-concurrency workloads.
    3. Task type: General conversation, knowledge-base question answering, coding assistance, structured extraction, embedding, reranking, speech, and vision require different model sizes and deployment patterns.
    4. Deployment topology: Single-GPU replicas, multiple replicas on one node, and distributed inference across nodes have different throughput, latency, and failure-isolation characteristics. A three-node cluster does not necessarily need one extremely large model; multiple medium or small replicas may be more practical.
    5. Licensing and compliance: Model-weight licenses, data governance, log retention, access control, and tenant isolation must be reviewed during model selection.
    6. Operational cost: Larger models take longer to load, are more difficult to upgrade, and are harder to recover after failures. Production deployments should prioritize observability, rollback, and elastic rate limiting.

    Model selection is therefore better understood as a matching problem:

    1. Business scenario → Resource budget → Model scale → Quantization strategy → Deployment topology

    Define the business metrics first, then use benchmark and load-test results to make the final selection.

    3. Establish a Baseline Before Upgrading

    The most important action before an upgrade is not installing the new version. It is documenting the current environment.

    The existing cluster can be represented as follows:

    Node Role GPU VRAM Data directory Description
    Node A Primary/control plane 32 GB Data directory A Handles management, scheduling, and some inference workloads
    Node B Worker 32 GB Data directory B Provides inference capacity
    Node C Worker 32 GB Data directory C Provides inference capacity

    The baseline should include at least:

    • Operating system distribution, kernel version, GPU driver, and compute runtime for every node;
    • GPU memory capacity, utilization, temperature, power consumption, and health status;
    • Current platform version, deployment parameters, model directories, model versions, and checksums;
    • Shared-storage mount points, permissions, user and group mappings, available capacity, and read/write behavior;
    • Network bandwidth, latency, and isolation policies between the primary and worker nodes;
    • Existing model replicas, routing rules, rate limits, logs, and monitoring metrics;
    • The acceptable maintenance window, rollback time objective, and acceptance criteria from the business side.

    Store this information as a pre-upgrade snapshot and compare it item by item with the compatibility requirements of the new release.

    Shared storage requires particular attention. When multiple nodes mount different directories from the same disk, verify that permissions are consistent, mount order is explicit, and write paths are isolated. Otherwise, different versions may write to the same metadata directory after the upgrade.

    4. Upgrade Principles: Validate, Canary, Then Roll Forward

    An upgrade may provide better resource scheduling, more efficient GPU memory usage, and support for newer model architectures. These benefits must be confirmed through a complete validation loop.

    Recommended principles include:

    1. Do not upgrade directly in production: Prepare a test node or temporary environment with the same drivers and storage mounts. Validate model loading, inference, scheduling, monitoring, and rollback.
    2. Back up before upgrading: Back up platform metadata, deployment configuration, model inventories, credentials, shared-storage snapshots, and critical business data. Perform a restoration test rather than merely confirming that a backup exists.
    3. Keep the old packages and rollback scripts: Metadata migrations may be irreversible. If the database schema changes, rollback should normally use a snapshot restore instead of a simple downgrade.
    4. Upgrade the control plane and workers in stages: A common sequence is to upgrade the control plane first and verify the management interface, followed by one worker at a time. Before upgrading a worker, stop new scheduling for that node, complete health checks, and then restore traffic.
    5. Validate shared storage as read-only first: Model-weight directories can be mounted read-only, while business-data directories remain isolated by node. During the upgrade, do not allow old and new versions to write to the same metadata area.
    6. Define acceptance gates: All nodes must be ready, GPUs must be visible, VRAM must be detected correctly, models must load, inference results must be correct, long-context and concurrency tests must pass, and failure handling must meet expectations.

    A representative upgrade workflow is:

    1. Upgrade plan:
    2. 1. Freeze changes and record the baseline of the old version
    3. 2. Back up metadata, configuration, model inventory, and shared-storage snapshots
    4. 3. Install the new version on a test node and restore the data
    5. 4. Validate node registration, model loading, inference, and monitoring
    6. 5. During the maintenance window, stop new jobs and drain workloads
    7. 6. Upgrade the control plane and check the management interface and metadata
    8. 7. Upgrade workers one by one:
    9. drain, stop the old service, install the new version,
    10. verify mounts, start the service, rejoin the cluster, and run regression tests
    11. 8. Run smoke tests and compare performance
    12. 9. Remove the change freeze after the system remains stable
    13. Rollback conditions:
    14. - Models cannot load or inference results are incorrect
    15. - Nodes cannot join the cluster or GPUs are not visible
    16. - Shared-storage permissions or data writes are incorrect
    17. - Latency, throughput, or VRAM usage becomes significantly worse

    5. VRAM Estimation and Model-Size Recommendations

    For a three-node cluster with one 32 GB GPU per node, model selection should not be based on parameter count alone. A conservative estimate is:

    1. Weight memory ≈ Parameter count × Bytes per parameter
    2. Total VRAM ≈ Weight memory
    3. + KV cache
    4. + Activation workspace
    5. + Runtime overhead
    6. + Safety margin

    FP16 generally requires about 2 bytes per parameter, INT8 about 1 byte, and INT4 about 0.5 bytes. Actual usage is higher because of runtime overhead and other allocations.

    On a 32 GB GPU, space must be reserved for the KV cache, batching, long-context requests, and runtime operations. The GPU should not be filled entirely with model weights.

    A practical selection matrix is:

    Resource condition Suitable workloads Suggested model scale Quantization and context guidance Deployment recommendation
    One 32 GB GPU Everyday conversation, knowledge Q&A, summarization, lightweight coding assistance 7B–14B Prefer 8-bit or 4-bit quantization; determine context length through testing Run one stable replica or multiple small-model replicas for higher throughput
    One 32 GB GPU Stronger general capability and complex instruction following Around 30B dense models Start with 4-bit quantization, low concurrency, and moderate context Use one replica with a VRAM safety margin; long-context testing is required
    Three nodes with 96 GB total Complex reasoning, batch jobs, and higher-quality generation 30B–70B Use 4-bit quantization and split across nodes or use pipeline parallelism Consider cross-node inference only with sufficient network bandwidth; otherwise prefer multiple medium or small replicas
    Small always-on services Embedding, reranking, classification, and lightweight extraction 0.3B–3B Use 8-bit or 16-bit precision These services can share nodes with conversational models, but concurrency and VRAM quotas must be controlled
    Multimodal or speech workloads Image-text understanding, speech recognition, or speech synthesis 3B–14B Use 8-bit or 4-bit quantization Use a separate queue and resource pool to prevent interference with the main conversational service

    A 70B-class model may still exceed the capacity of a single 32 GB GPU even with 4-bit quantization. It generally requires multi-GPU sharding. Cross-node inference also introduces communication overhead, and latency and stability depend heavily on the network.

    For a three-node cluster, a safer production strategy is often:

    1. Multiple 7B–14B replicas
    2. + one medium-sized, higher-quality replica
    3. + a small embedding model running continuously

    This is usually more practical than dedicating all resources to one very large model.

    Recommendations by Workload

    • General conversation and knowledge-base Q&A: Start with a quantized 7B–14B model and multiple replicas, combined with retrieval-augmented generation. If answer quality is insufficient, evaluate a quantized 30B-class model. In knowledge-base systems, persistent embedding and reranking services may provide more value than simply increasing the generation model size.
    • Coding assistance and structured extraction: Prioritize 7B–14B models with strong coding and instruction-following capabilities. Evaluate structured-output reliability, long-file handling, and tool-calling reliability. Production systems should also use schema validation, retries, and timeouts.
    • Complex reasoning and high-quality generation: A quantized 30B–70B model can be used at low concurrency, but higher latency and operational complexity must be accepted. For higher concurrency, place complex tasks in a queue and process them asynchronously rather than allowing every user request to compete for the large model.
    • High-frequency, low-latency services: Multiple small-model replicas are often better than one large model. Routing, caching, rate limiting, and batching can improve throughput and prevent excessive queueing.
    • Multimodal and speech workloads: Use a separate resource pool and monitor VRAM and request queues independently. These workloads often have substantial input-preprocessing overhead and cannot be estimated using text-model assumptions alone.

    6. Deployment Topologies for a Three-Node Cluster

    With three 32 GB GPUs, four deployment patterns are worth considering.

    Option 1: One Small or Medium Replica per Node

    Deploy a quantized 7B–14B model on each node and use a unified entry point for load balancing.

    The advantages are good failure isolation, high throughput, and relatively simple upgrades. The limitation is reduced support for complex, high-quality workloads. This is a suitable production baseline.

    Option 2: One Higher-Quality Medium Replica Plus Multiple Small Replicas

    One node can host a quantized 30B-class model, while the other two host smaller conversational models, embedding services, or rerankers.

    This approach balances quality and throughput, but it requires more precise routing. Simple requests should go to smaller models, while complex requests should be directed to the medium-sized model.

    Option 3: Shard a Large Model Across Nodes

    A 30B–70B model can be distributed across two or three nodes.

    The advantage is the ability to run a larger model. The disadvantages include communication overhead, a larger failure domain, and significantly greater tuning complexity. This pattern is recommended only when the nodes have fast, stable interconnects and the workload has low concurrency and can tolerate higher latency. A degradation strategy is also required.

    Option 4: Reserve the Primary Node for the Control Plane

    If the primary node handles substantial management and monitoring work, avoid placing critical inference replicas there, or run only low-priority workloads on it. This reduces the risk that inference load will destabilize the control plane.

    Regardless of the topology, shared storage should follow three principles:

    1. Model weights: read-only
    2. Cache: local whenever possible
    3. Business data: isolated by node

    Model-weight directories should be mounted read-only. Inference nodes should use local high-speed caches to reduce pressure on shared storage. Each worker should have an isolated business-data directory, and old and new versions should not write to the same metadata area during an upgrade.

    7. Upgrade Execution and Shared-Storage Considerations

    A recommended upgrade sequence is as follows:

    1. Review the change and confirm compatibility: Check the new version’s requirements for GPU drivers, compute runtime, operating-system kernel, and storage mounts. Review release notes for incompatible changes and migration requirements.
    2. Back up and rehearse restoration: Back up platform metadata, deployment configuration, model inventories, credentials, and shared-storage snapshots. Restore the backup at least once in a test environment.
    3. Pre-upgrade a test node: Use production-equivalent GPUs, drivers, and storage mounts. Validate node registration, model loading, inference requests, monitoring, and access controls.
    4. Open a maintenance window and switch traffic: Stop new jobs during a low-traffic period and drain active inference workloads. Notify business teams and switch to read-only or degraded operation if necessary.
    5. Upgrade the control plane: Install the new control-plane version, run the metadata migration, and verify the management interface, node list, model list, and storage configuration. If migration fails, restore from a snapshot before attempting further changes.
    6. Upgrade workers one by one: For each worker, stop new scheduling, confirm that no critical workload remains, stop the old service, install the new version, verify mounts, start the service, rejoin the cluster, run smoke tests, and restore scheduling. Keep at least one worker available throughout the process.
    7. Recheck shared storage: Verify mount points, permissions, remaining capacity, and read/write performance. Confirm that model directories are read-only, cache directories are writable, and business-data directories are isolated. Treat metadata directories with particular caution to prevent competing writes.
    8. Rehearse rollback and update documentation: Record versions, configurations, migration steps, and exception-handling procedures. If unacceptable problems appear during the observation period, restore the snapshot and the old version according to the rollback plan.

    Shared storage is a major upgrade risk. Network-storage latency, locking semantics, permission mapping, and mount stability can all affect platform startup.

    Separate model weights from frequently written directories. Avoid having every node scan the same directory concurrently at high intensity. In the startup sequence, verify that storage is available before starting inference services.

    8. Post-Upgrade Validation and Load Testing

    An upgrade is not complete when installation finishes. It is complete only after the validation loop has passed.

    Basic Smoke Tests

    Verify that:

    • All nodes are online and have the correct roles and labels;
    • Every GPU is visible and its VRAM capacity is detected correctly;
    • Model directories are readable, cache directories are writable, and permissions match expectations;
    • A small model can load and return correct results for simple inference requests;
    • Embedding and reranking services are available;
    • The primary management interface, logs, and monitoring are working;
    • After restarting one worker, the cluster can reschedule or recover workloads.

    Performance and Stability Tests

    Use a fixed prompt set and fixed generation parameters to compare the old and new versions. Measure:

    • Time to first token;
    • Output speed;
    • P95 latency;
    • Concurrent throughput;
    • Peak VRAM usage;
    • Long-context behavior;
    • Batch-request behavior;
    • VRAM headroom under high concurrency;
    • Shared-storage read latency;
    • CPU utilization;
    • Network bandwidth;
    • Queue length.

    Also simulate a single-node failure. Confirm that requests can be degraded, queued, or retried without data corruption. Verify that rate limiting, timeouts, retries, and circuit breaking still work as expected.

    Security and Operations Tests

    Confirm that:

    • Tenant isolation, access controls, and audit logs work correctly;
    • Sensitive data is not written to unnecessary logs;
    • Permissions for models and business data are clearly separated;
    • Monitoring and alerts cover GPU memory, temperature, service error rates, queue backlog, and available storage capacity;
    • Backup and restoration procedures have been tested in practice.

    A representative configuration relationship is:

    1. Deployment parameters:
    2. - Model source: read-only directory on shared storage
    3. - Replica count: two medium or small model replicas
    4. - Quantization: 4-bit or 8-bit, determined through load testing
    5. - Minimum VRAM per replica: reserve 4–8 GB as a safety margin
    6. - Maximum context: start at 8,192 tokens, then validate against business needs
    7. - Maximum batch size: start at 4 and increase gradually
    8. - Routing: send simple requests to the small model and complex requests to the medium model
    9. - Degradation: route to the small model or return a retryable response
    10. when the high-quality model exceeds its queue timeout
    11. - Metrics: time to first token, output speed, P95 latency, peak VRAM,
    12. error rate, and queue length

    These values illustrate configuration principles rather than fixed interfaces or universal settings. Actual deployments should use the configuration options supported by the selected platform and the results of load testing.

    9. Practical Recommendations for a Three-Node, 32 GB Cluster

    A cluster with three nodes and one 32 GB GPU per node is generally better suited to a multi-model, multi-replica platform with workload-aware routing than to a single extremely large model.

    A three-layer design is practical:

    1. Baseline layer: Deploy quantized 7B–14B conversational-model replicas across the nodes to handle most everyday requests. Add small embedding and reranking models for knowledge-base retrieval. This layer prioritizes stability, low latency, and easy recovery.
    2. Quality-enhancement layer: Deploy a quantized 30B-class model on one node to handle complex questions, long-document summarization, and higher-quality generation at low concurrency. Route complex requests to this model without allowing it to affect baseline services.
    3. Experiment and batch layer: Use 70B-class or cross-node-sharded models for offline batch processing, low-concurrency experiments, or quality comparisons. If inter-node bandwidth and stability are insufficient, do not use this configuration as the primary online service.

    The question of whether a new platform version supports more models should be evaluated separately from whether the business actually needs larger models. A version upgrade may improve scheduling and compatibility, but model selection must still be based on task requirements, VRAM budget, concurrency, and operating cost.

    The safest path is:

    1. Back up and test
    2. → Canary-upgrade the control plane and one worker
    3. → Validate the results
    4. → Roll through the remaining workers
    5. → Compare latency, throughput, peak VRAM, and answer quality
    6. using a fixed evaluation set

    Only this complete validation loop can produce a reusable balance between stability, quality, and cost for a three-node GPU cluster.