# How to Build and Upgrade a GPU Cluster for On-Premises LLM Deployment
This article walks through building and upgrading a GPU cluster for running large language models on your own hardware. It explains why teams move to local deployment, how to size VRAM and storage for a multi-node inference cluster, and why cross-major-version upgrades demand extra caution. It then covers the concrete steps for backing up data, fixing directory permissions, and planning storage before an upgrade, followed by migrating from an embedded database to a standalone relational database. Finally, it describes how to validate services and measure performance after the upgrade, and how to keep production stable through automated backups, monitoring, staged rollouts, and rollback plans.
1. Why more teams are deploying LLMs on their own hardware
As the open-source model ecosystem keeps maturing, running inference, fine-tuning, and serving LLMs on owned hardware has become a routine choice for many teams. Compared with calling a cloud API directly, local deployment offers several clear advantages.
Data compliance. Industries such as finance, healthcare, and government have audit requirements for sensitive data. On-premises deployment keeps data inside the internal network, which makes it easier to pass security and compliance reviews.
Predictable latency. The inference path never leaves the network, so internal calls are more stable. This suits business scenarios that need low-latency interaction.
Predictable long-term cost. The upfront hardware investment is higher, but once the model size and call volume stabilize, the per-inference cost is often more controllable than paying for a cloud API by usage over the long run.
High customizability. Teams can freely choose model size, quantization strategy, inference framework, batching behavior, and context length, which makes it easier to optimize deeply for a specific workload.
That said, local deployment is not a matter of “install it and it runs.” Hardware selection, GPU scheduling, platform version maintenance, and data backup and migration all matter. A single mishandled step can lead to service outages or data loss. This article uses a cross-major-version upgrade as a concrete example to explain the key points of GPU cluster setup, upgrade preparation, database migration, and production validation.
2. Planning VRAM and storage for a GPU cluster
A typical multi-node inference cluster consists of several servers, each with dedicated GPUs, managed together through a unified model management platform and exposed as an inference service. Take a three-node cluster as an example: three servers run the same version of a Linux OS, each equipped with a GPU with roughly 32GB of single-card VRAM, and all on matching GPU driver and compute runtime versions. After the three nodes are networked, one is designated as the master node (the control plane) and the other two as worker nodes (the data plane).
A few practices are worth attention when planning a cluster like this.
2.1 Keep hardware homogeneous
Keep the OS, GPU model, driver version, and compute runtime version consistent across nodes. Hardware homogeneity reduces scheduling failures, performance jitter, and image compatibility issues caused by environment differences. For production, include node images, driver versions, and OS patches in unified management to avoid version drift from manual upkeep.
2.2 How to choose VRAM
VRAM is one of the core constraints of local LLM deployment. It determines the model parameter count you can host, the quantization approach, the context length, the number of concurrent requests, and the batch size.
Start by defining the business goal: How large a model do you need to run? Is the focus inference or fine-tuning? Do you need long context? Is single-card VRAM enough to support the target batch size?
With 32GB of single-card VRAM, you can usually run medium-sized quantized models comfortably. For example, common 7B and 13B models have different options under INT4, INT8, or FP16 mixed strategies. To run larger full-precision models or support higher concurrency, you will need multi-card tensor parallelism, pipeline parallelism, or a more granular VRAM offloading strategy.
A common mistake is looking only at the model weight size while ignoring KV Cache, runtime VRAM, and batching overhead. The longer the context and the higher the concurrency, the greater the VRAM pressure. So when planning VRAM, reserve some headroom to avoid out-of-memory (OOM) errors under sudden concurrency spikes in production.
2.3 Unify storage
For a three-node cluster, use a shared storage set mounted into different directories through a network file system. Have the master and worker nodes write to separate subdirectories. This isolates each node’s data while keeping backup and recovery centralized.
Before an upgrade, especially, confirm that the storage mount relationships are clear. A cross-version upgrade may change how data is written. If mount paths are messy, the shared storage is unstable, or directory permissions differ across nodes, problems can be amplified during migration.
3. Why cross-major-version upgrades require caution
Jumping from one major version to the next, for example from 0.x to 2.x, usually means a substantive change in the underlying architecture rather than a simple feature stack. Such upgrades typically involve changes to data structures, storage components, runtime behavior, the permission model, and configuration options, so the risk is clearly higher than a minor-version update.
Take this upgrade as an example. The new version introduces two significant changes.
First, performance and model support improve substantially. It can run newer models and optimizes inference scheduling, VRAM management, and model loading.
Second, the underlying database switches from a lightweight in-process embedded database to a separately deployed relational database. This is essentially “replacing the foundation.” If you do not prepare thoroughly beforehand, you can easily run into historical data that cannot be read, a service that will not start, an interrupted migration, or insufficient permissions.
So before upgrading, read the official upgrade documentation thoroughly. Confirm the version compatibility path, data migration requirements, database version requirements, known issues, and the rollback plan. Never run the upgrade directly on production. Walk the entire process once in a validation environment first.
4. Preparing backup, permissions, and storage before an upgrade
4.1 Data backup comes first
No matter how mature the upgrade plan is, data backup is the baseline. Before upgrading, take a full backup of the entire data directory, preferably to another disk or a separate shared storage, so that a failure of the original disk does not take out the backup and the source data together.
You can use a file copy command with preservation options:
# Preserve permissions, timestamps, and symlinks; create a full snapshot-style backupcp -a "${your-data-dir}" "${your-data-dir}.backup.$(date +%Y%m%d)"
The -a archive mode preserves file permissions, ownership, timestamps, symlinks, and other metadata, avoiding attribute corruption on restore.
After the backup, do not delete the source directory right away, and do not start the upgrade immediately. First verify the directory’s integrity: confirm that the file count, directory structure, and file sizes match expectations. For critical business data, sample a few configuration files, database files, or log files and confirm they are readable.
4.2 How to sort out directory permissions
After the new version introduces a standalone database, the process often runs as a non-root user. If the historical data directory and its parent paths lack execute (x) permissions, the new process may be unable to access the data due to insufficient permissions.
To be safe, complete the path permissions in advance even if you currently run as root. The script below adds read/write/execute permissions to the current directory and adds execute permission level by level up the parent directories, so an unprivileged user can traverse the entire path.
dir="${your-data-dir}"# Add read and execute permissions to the current data directorychmod a+rx "$dir"# Starting from the parent directory, walk upward and add execute permission level by levelp=$(dirname "$dir")while [ "$p" != "/" ]; dochmod a+x "$p"p=$(dirname "$p")done
The a+x here mainly makes directories traversable, letting an unprivileged user reach the target directory along the path. It does not equate to granting write access to the root directory. If the workload also involves a log directory, check its permissions separately:
chmod a+rx "${dir}/log" 2>/dev/null || true
This step is often overlooked in containerized deployments, yet it directly affects whether the new version’s containers can read the old data. Add the permission-fixing script to the upgrade checklist and verify it both before and after the upgrade.
4.3 Plan storage mounts and isolation
Before upgrading, confirm that the storage mount relationships on each node are still clear. Are the master node’s data and each worker node’s data mounted in separate subdirectories? Is the network share stable? Is there enough disk capacity to support the new version’s writes?
A cross-version upgrade changes how data is written, so sorting out the mount structure in advance significantly reduces confusion during migration. If the shared storage has a single point of failure, confirm a backup path and a failover plan beforehand.
5. Migrating from an embedded database to a standalone relational database
This is the most technically demanding and error-prone step in the upgrade. The old version uses an in-process embedded lightweight database to store metadata and business data, while the new version switches to a separately deployed relational database. The two have incompatible data formats, so an explicit migration is required.
In a containerized deployment, the migration is usually triggered by passing a “data migration” flag when starting the new-version container. On first startup, the container reads the old embedded database files, converts their structure and business data, and writes them into the new standalone database.
To make the operation actionable, confirm a few prerequisites before upgrading.
First, confirm the target relational database is deployed and reachable from the new-version container. Common relational databases include PostgreSQL and MySQL, but the specific choice should follow the platform’s official documentation. Confirm the database version, character set, time zone, user permissions, and connection address in advance.
Second, confirm the image source and version. Use the official image or a fixed version from a trusted internal image repository, and record the image digest or version number to avoid pulling an unstable image on the fly.
Third, confirm driver and runtime dependencies. GPU inference platforms usually depend on specific GPU driver, compute runtime, and container toolchain versions. Before upgrading, confirm that the host environment matches the image requirements.
Fourth, confirm the database connection string is ready and passed in through environment variables or a secure configuration method. Never write plaintext passwords into scripts, images, or public repositories.
The following is an illustrative startup command that uses the master node as an example and enables the migration flag via an environment variable. It assumes the target database is PostgreSQL; replace it with the database type and connection string supported by your platform.
docker run -d --name inference-platform \--restart=unless-stopped \--network=host \--privileged \--env DATA_MIGRATION=true \--env DB_TYPE=postgresql \--env DB_DSN='postgres://user:password@inference-db:5432/inference' \--volume "${your-data-dir}:/var/lib/data" \--volume /var/run/docker.sock:/var/run/docker.sock \platform-image:2.x
A special note: this example uses --privileged and mounts /var/run/docker.sock. Both significantly raise the container’s privileges. --privileged opens broad host access, while mounting docker.sock is, in most cases, equivalent to handing the container root access to the host. Use these only in a trusted internal environment or during a migration window where they are truly necessary, and prefer the platform’s least-privilege approach when available. If the platform does not need access to the host’s Docker daemon, remove both of these settings.
Keep the following in mind during migration.
The migration is usually a one-time action. After it completes and the data is written to the new database, turn off the migration flag and restart, to avoid anomalies from re-running it.
Before migrating, confirm the standalone database is ready and can be reached normally. If the target database is unreachable, the account lacks permissions, or the database version is incompatible, the migration will fail.
During migration, watch the container logs closely. Confirm that table creation, row counts written, index building, and task status match expectations. Do not just check whether the container started; check whether the data actually finished migrating.
Minimize business writes during migration to avoid inconsistency from mixed old and new data. If the platform supports a maintenance window or read-only mode, enable it before migrating.
Keep the old embedded database files until the new version has run stably for a while. This leaves room to roll back if the migration goes wrong or the new version does not behave as expected.
6. Validating and observing performance after the upgrade
Finishing the upgrade does not mean the work is done. To make the new version truly usable, validate systematically.
Service health check. Confirm that all master and worker nodes are online, that the control plane and data plane communicate normally, and that the model management platform can manage the nodes. For a container deployment, check container status, log output, and restart counts.
Data integrity verification. Sample key business data in the new database and confirm it is complete. Compare record counts against the old database and check that timestamps, status fields, and relationships are correct. For high-value data, prepare a verification script in advance and run it once before and once after the upgrade.
Model loading test. Pull a medium-sized model and verify that the inference path works end to end. Send a short request and then a long-context request, and observe whether time-to-first-token, throughput, and VRAM usage meet expectations.
Resource usage observation. Use the monitoring dashboard to check VRAM, compute, memory, disk I/O, and network traffic, and confirm the new version’s scheduling policy takes effect. If a node shows abnormal resource usage, check its model loading logs, device visibility, and driver status.
When evaluating the upgrade’s benefits, give quantified metrics rather than a vague claim like “throughput improved significantly.” When comparing before and after, keep the same model, quantization method, input/output length, batch size, and hardware conditions, then compare tokens/s, time-to-first-token, peak VRAM, and error rate. This tells you whether the improvement comes from the new version’s scheduling optimizations or from changed test conditions.
7. Keeping production stable
Once validation passes and you want long-term stable service, complete a few more items.
Automated backups. Do not rely on manual cp. Set up scheduled jobs to take periodic snapshots of both the data directory and the standalone database, and retain multiple historical versions. Practice restore drills for the database backups regularly to confirm the backup files are actually usable.
Monitoring and alerting. Set threshold alerts for node liveness, VRAM levels, GPU utilization, inference latency, error rate, queue backlog, and database connection counts. The earlier you detect a problem, the lower the cost of handling it.
Staged upgrades. Run the new version on one worker node first. Once model loading, inference service, and data consistency check out, roll out the remaining nodes gradually. For a core production cluster, keep the previous-version container or node as a rollback path.
Documentation and rollback plan. Record each step, its parameters, expected results, and the owner. Write the rollback path in advance, including how to restore the original data directory, how to switch back to the old image, how to restore the database connection, and how to notify stakeholders. That way, if anything goes wrong at any stage, you can fall back to the last stable state.
Least privilege. The upgrade process may need high privileges, but day-to-day operation should follow the principle of least privilege. Do not let business containers run with excessive privileges long-term, and do not write database passwords or image registry credentials into command history or commit them to a code repository.
8. Common mistakes and pitfalls
The first common mistake is skipping the backup and upgrading directly. A cross-major-version upgrade can change data formats, and if the migration fails with no backup, you are in a very difficult position.
The second common mistake is checking only whether the container started, not whether the data migrated completely. A successfully started container does not mean a successful migration; you must verify table structures, data volumes, and key fields.
The third common mistake is ignoring directory permissions. When the old version runs as root and the new version runs as non-root, path permission problems surface immediately.
The fourth common mistake is not preparing the target database in advance. Once the migration flag is on, a failed database connection, insufficient permissions, or an incompatible version will force the upgrade to stop.
The fifth common mistake is using high-privilege options blindly. Configurations like --privileged, mounting docker.sock, and disabling security restrictions may be convenient during migration, but they expand the attack surface. Unless the platform explicitly requires them, prefer a more secure deployment approach.
Summary
Deploying LLMs on-premises is a systems engineering effort: hardware planning, cluster management, version upgrades, and data migration are tightly interlinked. A cross-major-version upgrade tests rigor especially hard: read the docs, then back up the data; fix permissions, then migrate; validate functionality, then assess performance; prepare the rollback, then move to production.
Put data safety and rollback capability first, and you can gain the new version’s performance and model support while completing a smooth evolution from the validation environment to production.
