Robust Deployment of Long-Running Services in Virtual Machines

    Long-running services need unattended availability, automatic recovery after reboots, persistent logs and data, and controlled upgrades. This guide explains how to isolate such workloads in a dedicated Linux virtual machine, covering resource allocation, remote access, runtime version management, non-root execution, service supervision, backups, upgrades, security, and observability.

    Long-running applications are not the same workload as desktops that are opened only occasionally. A persistent service must remain available without an active user session, recover automatically after system reboots, and retain its logs and data when a session ends. Desktop environments, in contrast, prioritize graphical interfaces, peripheral passthrough, and interactive usability.

    Placing a persistent service inside a container-based desktop workstation can introduce session cleanup, resource contention, and unpredictable upgrade windows. A more reliable approach is to use virtualization on the host device to create a dedicated Linux virtual machine. Keep the service, its dependencies, and its data inside that virtual machine.

    1. Define the Deployment Boundary

    Before deployment, clarify three points:

    • Must the service be available 24/7?
    • Does it require a graphical desktop?
    • What recovery time is acceptable after a failure?

    If the answers are “it must remain online,” “it does not require a graphical interface,” and “recovery should take minutes,” do not use a graphical desktop workstation as the service foundation. Desktop environments are useful for debugging and occasional maintenance. Production services should run on a minimal installation or a system containing only the required components, managed through a remote terminal.

    The host should provide CPU, memory, storage, and networking. Install the runtime, application, and service processes inside the virtual machine. This prevents host upgrades or desktop component changes from directly interrupting the service. The virtual machine can also have its own snapshots and rollback process.

    2. Allocate Virtual Machine Resources

    For services built around scripting runtimes and lightweight web or proxy processes, two virtual CPUs and 4 GB of memory are often sufficient. Actual concurrency and log volume should determine the final configuration.

    Reserve enough memory for the host operating system and its runtime overhead. Assigning nearly all host memory to the virtual machine can make the host unstable.

    Consider separating the system disk from the data disk:

    • Store the operating system and software packages on the system disk.
    • Store configuration files, logs, caches, and application data on the data disk.

    With this layout, the operating system can be reinstalled without moving application data.

    For networking, choose either bridged networking or host-side port forwarding:

    • Use bridged networking with a fixed address when other devices on the local network must access the service directly.
    • Use port forwarding when access should be limited to the host or to a small number of exposed ports.

    In either case, open only the required ports in the firewall. Use a server or desktop distribution that is still within its maintenance lifecycle. Set the language and time zone during installation so that later regional changes do not affect log timestamps.

    3. Initialize the System and Configure Remote Access

    After the virtual machine starts for the first time, update the software sources and install the basic tools needed for version management, downloads, editing, and remote login.

    Create a regular administrative account and perform daily operations under that account. Use temporary privilege escalation only when installing or changing system-level software.

    Remote access should preferably use key-based authentication. Configure the listening address and port according to the network environment, and restrict permitted login sources to the required network ranges.

    Avoid keeping direct root login and password-only authentication enabled for convenience. If the root password is exposed, an attacker gains control of the entire machine. A safer approach is to use key-based login for a regular user and sudo for privileged commands. Once key authentication has been verified, disable password authentication.

    After changing the configuration, restart the remote login service and test the new settings in a separate session. Confirm that the new session works before closing the existing one. This prevents an incorrect configuration from locking you out.

    4. Isolate Runtime Versions

    Many services depend on a specific major version of a scripting runtime. If the distribution’s global runtime is used directly, a system upgrade may silently change the version and cause the service to fail at startup.

    Install the runtime in the user’s home directory or in a separate prefix, and use a version management tool to pin the required major version. For example, standardize on a supported long-term version and make it the default.

    Before installing an application globally, confirm that the current shell has loaded the version manager’s environment variables. Otherwise, an installation may appear to succeed while a newly opened terminal cannot find the command.

    Document the runtime major version used by the team. This prevents one machine from using a newer version while another uses an older one. When localized interfaces or language resources are required, prefer a distribution channel that stays synchronized with upstream and adds language packages without diverging from the core release. Review the change notes before upgrading, and avoid tying language packages and core versions together in a state that cannot be rolled back.

    The following is an illustrative workflow. Replace paths and package names for the actual environment:

    1. # Install and enable the required major runtime version
    2. # The following steps are illustrative and are not tied to a specific tool
    3. load_runtime_manager
    4. install_runtime 22
    5. use_runtime 22
    6. set_default_runtime 22
    7. install_service_package

    After installation, use which or an equivalent command to verify that the executable comes from the expected prefix rather than the system-provided path.

    5. Run the Service as a Non-Privileged User

    Do not run service processes as root. Create a dedicated system user for each service, restrict its home directory permissions, and ensure that configuration files containing keys or tokens are readable only by that user.

    If the service must listen on a port below 1024, use firewall forwarding or grant only the minimal capability required to bind to low ports. Do not run the entire service as root.

    Keep the working directory, log directory, and temporary directory separate. Set reasonable disk quotas or limits to prevent logs from filling the system disk.

    6. Supervise the Process and Enable Automatic Startup

    A process started manually in a terminal will usually end when the session closes. It is therefore unsuitable for long-term operation.

    Register the application with the system service manager. Define the startup command, working directory, service user, environment file, and restart policy. The service should restart after an unexpected exit, but repeated restarts should be limited to prevent a crash loop from consuming CPU and other resources.

    Enable startup at boot, and configure the service to start only after the network is ready.

    Send standard output and standard error to the system logging facility instead of redirecting them to one unrotated file. After changing the unit configuration:

    1. Run a syntax check.
    2. Reload the service manager.
    3. Check the service status and recent logs.
    4. Confirm that the expected port is listening.
    5. Verify that the health check succeeds.

    7. Separate Configuration, Data, and Upgrades

    Keep application configuration separate from the program files. Programs can be reinstalled, while configuration, databases, and user data should remain on the data disk or in an independent directory. Include these locations in regular backups.

    Before an upgrade, create a virtual machine snapshot or at least back up the configuration directory. Upgrade within the same major version first, observe the service for a period of time, and only then consider moving to another major version.

    When rolling back, restore the program version without overwriting the data directory that is still in use.

    If both an upstream release and a localized release are present, document which package is used by routine commands. Two global commands may otherwise compete for the same port or data directory. Keep only the version actually in use, or assign different ports and data paths to each one.

    8. Security and Observability

    Minimize the exposed surface:

    • Allow only the required management and service ports.
    • Avoid exposing management ports directly to the public internet.
    • When remote access is necessary, use an encrypted tunnel.

    Apply operating system security updates regularly, but schedule application upgrades separately from system upgrades.

    Enable basic monitoring for:

    • Whether the process is alive
    • Whether the expected port is listening
    • Disk and memory utilization
    • Sudden increases in error logs

    Prepare simple checklists for service restarts, failed upgrades, and low-disk-space conditions. A short operational checklist is usually more effective than searching through large logs after an incident.

    9. Common Mistakes

    Typical mistakes include:

    • Treating a desktop container as a production foundation, causing the service to disappear after sleep or session termination
    • Enabling root login with passwords for convenience, increasing the impact of a compromise
    • Installing the runtime in the wrong user environment, so the service unit cannot find the command
    • Storing logs and application data on the system disk without rotation, eventually filling the disk and preventing system login
    • Upgrading without a snapshot, leaving no easy rollback path when localized and upstream packages behave differently

    Follow this order of implementation: virtual machine isolation, regular user access, a pinned runtime version, service-manager supervision, separation of data from program files, and minimal network exposure. This turns a long-running service on an edge device from something that merely starts successfully into something that remains manageable after reboots and upgrades.