Deploying Ollama Locally: Installation, API Calls, and Security Configuration

    This guide walks through the local deployment of Ollama, covering hardware requirements, installation across Windows, Linux, and Docker, model execution, API integration, storage path migration, and essential security settings.

    1. Verify Hardware Requirements

    Local deployment bottlenecks are typically VRAM and disk space. The table below estimates requirements based on Ollama’s default Q4_K_M quantized version and should be used as a starting point. Longer contexts and more concurrent requests will increase VRAM and RAM usage. Always rely on the actual output of ollama ps for precise resource consumption.

    Model Size Example Models Model File Size (Approx.) Recommended VRAM Recommended RAM
    3B llama3.2:3b, qwen2.5:3b 2 GB 4 GB minimum, 6 GB recommended 16 GB
    8B llama3.1:8b 4.9 GB 8 GB 16 GB+

    If VRAM is insufficient, Ollama will offload some model layers to RAM for CPU computation, significantly reducing generation speed. For NVIDIA GPUs, ensure the official drivers are installed and nvidia-smi runs correctly in the terminal. Since model loading speed depends on disk read speed, an SSD is recommended, with at least 20 GB of free space reserved for multiple models.

    2. Install Ollama

    Windows: Download the OllamaSetup.exe installer from the official Ollama website and follow the setup wizard. After installation, open PowerShell and run:

    1. ollama --version

    If the version number is displayed, the installation is successful. If you encounter a prompt indicating the command is not recognized, close and reopen the terminal to refresh the system PATH.

    Linux: It is recommended to download the installation script, review its contents, and then execute it. Avoid piping network scripts directly to the shell to prevent executing unverified content.

    1. curl -fsSL https://ollama.com/install.sh -o install.sh
    2. less install.sh
    3. sh install.sh

    After installation, verify the version using ollama --version, and check the background service status with systemctl status ollama.

    Docker: To isolate the runtime environment, use the official Docker image. The following command binds the port only to the local loopback address, preventing direct access from external machines:

    1. docker run -d --name ollama -p 127.0.0.1:11434:11434 -v ollama:/root/.ollama ollama/ollama

    3. Pull Models and Verify Execution

    Only pull models from the official Ollama model library, and review the corresponding model licenses to ensure they permit your use case. Using a 3B model as an example:

    1. ollama pull llama3.2:3b
    2. ollama list

    The first pull downloads the complete model file. A 3B model is about 2 GB, taking roughly 3 minutes on a 100 Mbps network; an 8B model is about 4.9 GB, taking around 7 minutes. If a pull is interrupted, re-running the same command will resume from where it left off.

    ollama list displays the model name, ID, size, and modification date. Once the target model appears in the list, start an interactive chat with the --verbose flag:

    1. ollama run llama3.2:3b --verbose

    Enter a prompt; after the response is generated, the terminal will print the eval rate metric, indicating tokens generated per second. This is actual benchmark data from your machine and is more reliable than the table values. Type /bye to exit interactive mode.

    Models are only loaded into VRAM after their first invocation. Therefore, after a chat or API request, open a new terminal and run:

    1. ollama ps

    If the PROCESSOR column shows “100% GPU”, the model is running entirely in VRAM. If it shows a CPU/GPU split, some computation is handled by the CPU; consider using a smaller model or closing other VRAM-consuming applications. An empty list usually means the model is not loaded yet, not necessarily an error.

    4. Call Models via API

    Ollama provides an HTTP API by default at 127.0.0.1:11434. On Linux or macOS, test the generate endpoint using:

    1. curl http://127.0.0.1:11434/api/generate -d '{"model": "llama3.2:3b", "prompt": "Introduce Ollama in one sentence", "stream": false}'

    Here, model specifies the model name (must match the name in ollama list), prompt is the input text, and setting stream to false makes the server wait until the entire response is generated before returning it.

    In Windows PowerShell, curl defaults to an alias for another command. Use curl.exe and escape the quotes:

    1. curl.exe http://127.0.0.1:11434/api/generate -H "Content-Type: application/json" -d "{\"model\": \"llama3.2:3b\", \"prompt\": \"Introduce Ollama in one sentence\", \"stream\": false}"

    The response is a JSON object with key fields:

    • response: The generated text.
    • done: true indicates generation is complete.
    • eval_count: The number of generated tokens.
    • eval_duration: Generation time in nanoseconds.

    Divide eval_count by the seconds in eval_duration to calculate the generation speed.

    For multi-turn conversations, call the chat endpoint /api/chat and pass a messages array:

    1. curl http://127.0.0.1:11434/api/chat -d '{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Hello"}], "stream": false}'

    In Python, use the requests library to call the generate endpoint. This code is suitable for local testing:

    1. import requests
    2. resp = requests.post(
    3. "http://127.0.0.1:11434/api/generate",
    4. json={"model": "llama3.2:3b", "prompt": "Introduce Ollama in one sentence", "stream": False},
    5. timeout=120,
    6. )
    7. resp.raise_for_status()
    8. print(resp.json()["response"])

    If the API returns an error, troubleshoot as follows:

    • Connection refused: The service is not running. Run sudo systemctl start ollama on Linux; ensure Ollama is running in the background on Windows.
    • 404 or model not found: The model name is misspelled or not pulled. Verify the name with ollama list and run ollama pull.
    • First request is slow: The model is loading into VRAM, which is normal. Wait and retry.
    • Empty response or timeout: Check if VRAM is insufficient. Use ollama ps to view the running device and switch to a smaller model if necessary.

    5. Modify the Model Storage Path

    Model files typically occupy several GB to tens of GB. If your system disk is running out of space, migrate the model directory to another disk. Ensure the target disk has sufficient space and correct directory permissions before migrating.

    Windows: The default model directory is %USERPROFILE%\.ollama\models. Steps:

    1. Quit Ollama from the system tray.
    2. Create a target directory, e.g., D:\ollama\models.
    3. Add a new system environment variable OLLAMA_MODELS with the value D:\ollama\models.
    4. Restart Ollama.
    5. Copy the contents from the old directory to the new one. Confirm ollama list shows your models before deleting the old directory.

    Linux: When installed via systemd, models are saved in the ollama user’s home directory, typically /usr/share/ollama/.ollama/models. Steps:

    1. sudo mkdir -p /data/ollama/models
    2. sudo rsync -av /usr/share/ollama/.ollama/models/ /data/ollama/models/
    3. sudo chown -R ollama:ollama /data/ollama
    4. sudo systemctl edit ollama.service

    In the opened editor, add the following and save:

    1. [Service]
    2. Environment="OLLAMA_MODELS=/data/ollama/models"

    Then reload the configuration and restart the service:

    1. sudo systemctl daemon-reload
    2. sudo systemctl restart ollama
    3. ollama list

    If the service fails to start, run journalctl -u ollama to check logs. The most common causes are incorrect directory permissions or active SELinux/AppArmor policies.

    Docker: The container saves models in /root/.ollama/models using a volume named ollama. To use a host directory instead, stop and remove the old container, copy the volume contents to the target directory, and restart with a bind mount:

    1. docker stop ollama
    2. docker rm ollama
    3. mkdir -p /data/ollama
    4. docker run --rm -v ollama:/from -v /data/ollama:/to alpine cp -a /from/. /to/
    5. docker run -d --name ollama -p 127.0.0.1:11434:11434 -v /data/ollama:/root/.ollama ollama/ollama

    After copying, verify the models are still visible using docker exec ollama ollama list.

    6. Security Configuration

    Ollama’s API does not have built-in authentication. Anyone who can access the port can call the model and consume computing resources. Therefore, the core of security configuration is restricting access scope.

    • Keep default listening: By default, it listens only on 127.0.0.1, accessible only locally. Do not set OLLAMA_HOST to 0.0.0.0 unless LAN access is explicitly required.
    • Do not expose port 11434 to the public internet: Avoid port forwarding on routers or opening the port on cloud servers. Be careful with Docker port mappings; omitting the 127.0.0.1: prefix exposes the port to all network interfaces.
    • Add protection for remote access: If access from other machines is necessary, place an Nginx or Caddy reverse proxy in front, enable TLS encryption, and configure access tokens or basic authentication. Use a firewall to restrict source IPs.
    • Be cautious with cross-origin access: OLLAMA_ORIGINS controls which web origins can access the API. Keep the default settings unless explicitly needed; avoid setting it to a wildcard.
    • Review installation scripts: Always read the script contents before installation on Linux to ensure the download URLs and execution commands are trustworthy.

    7. Summary

    By following these steps, you can select a model based on your VRAM and disk space, complete the installation, pulling, and execution verification, and finally call the model via the API. The model directory can be migrated to another disk, and the service is accessible locally by default. When remote access is required, configure a reverse proxy, TLS, and authentication before opening access.