Deploying Ollama Locally: Installation, API Calls, and Security Configuration
This guide walks through the local deployment of Ollama, covering hardware requirements, installation across Windows, Linux, and Docker, model execution, API integration, storage path migration, and essential security settings.
1. Verify Hardware Requirements
Local deployment bottlenecks are typically VRAM and disk space. The table below estimates requirements based on Ollama’s default Q4_K_M quantized version and should be used as a starting point. Longer contexts and more concurrent requests will increase VRAM and RAM usage. Always rely on the actual output of ollama ps for precise resource consumption.
| Model Size | Example Models | Model File Size (Approx.) | Recommended VRAM | Recommended RAM |
|---|---|---|---|---|
| 3B | llama3.2:3b, qwen2.5:3b | 2 GB | 4 GB minimum, 6 GB recommended | 16 GB |
| 8B | llama3.1:8b | 4.9 GB | 8 GB | 16 GB+ |
If VRAM is insufficient, Ollama will offload some model layers to RAM for CPU computation, significantly reducing generation speed. For NVIDIA GPUs, ensure the official drivers are installed and nvidia-smi runs correctly in the terminal. Since model loading speed depends on disk read speed, an SSD is recommended, with at least 20 GB of free space reserved for multiple models.
2. Install Ollama
Windows: Download the OllamaSetup.exe installer from the official Ollama website and follow the setup wizard. After installation, open PowerShell and run:
ollama --version
If the version number is displayed, the installation is successful. If you encounter a prompt indicating the command is not recognized, close and reopen the terminal to refresh the system PATH.
Linux: It is recommended to download the installation script, review its contents, and then execute it. Avoid piping network scripts directly to the shell to prevent executing unverified content.
curl -fsSL https://ollama.com/install.sh -o install.shless install.shsh install.sh
After installation, verify the version using ollama --version, and check the background service status with systemctl status ollama.
Docker: To isolate the runtime environment, use the official Docker image. The following command binds the port only to the local loopback address, preventing direct access from external machines:
docker run -d --name ollama -p 127.0.0.1:11434:11434 -v ollama:/root/.ollama ollama/ollama
3. Pull Models and Verify Execution
Only pull models from the official Ollama model library, and review the corresponding model licenses to ensure they permit your use case. Using a 3B model as an example:
ollama pull llama3.2:3bollama list
The first pull downloads the complete model file. A 3B model is about 2 GB, taking roughly 3 minutes on a 100 Mbps network; an 8B model is about 4.9 GB, taking around 7 minutes. If a pull is interrupted, re-running the same command will resume from where it left off.
ollama list displays the model name, ID, size, and modification date. Once the target model appears in the list, start an interactive chat with the --verbose flag:
ollama run llama3.2:3b --verbose
Enter a prompt; after the response is generated, the terminal will print the eval rate metric, indicating tokens generated per second. This is actual benchmark data from your machine and is more reliable than the table values. Type /bye to exit interactive mode.
Models are only loaded into VRAM after their first invocation. Therefore, after a chat or API request, open a new terminal and run:
ollama ps
If the PROCESSOR column shows “100% GPU”, the model is running entirely in VRAM. If it shows a CPU/GPU split, some computation is handled by the CPU; consider using a smaller model or closing other VRAM-consuming applications. An empty list usually means the model is not loaded yet, not necessarily an error.
4. Call Models via API
Ollama provides an HTTP API by default at 127.0.0.1:11434. On Linux or macOS, test the generate endpoint using:
curl http://127.0.0.1:11434/api/generate -d '{"model": "llama3.2:3b", "prompt": "Introduce Ollama in one sentence", "stream": false}'
Here, model specifies the model name (must match the name in ollama list), prompt is the input text, and setting stream to false makes the server wait until the entire response is generated before returning it.
In Windows PowerShell, curl defaults to an alias for another command. Use curl.exe and escape the quotes:
curl.exe http://127.0.0.1:11434/api/generate -H "Content-Type: application/json" -d "{\"model\": \"llama3.2:3b\", \"prompt\": \"Introduce Ollama in one sentence\", \"stream\": false}"
The response is a JSON object with key fields:
response: The generated text.done:trueindicates generation is complete.eval_count: The number of generated tokens.eval_duration: Generation time in nanoseconds.
Divide eval_count by the seconds in eval_duration to calculate the generation speed.
For multi-turn conversations, call the chat endpoint /api/chat and pass a messages array:
curl http://127.0.0.1:11434/api/chat -d '{"model": "llama3.2:3b", "messages": [{"role": "user", "content": "Hello"}], "stream": false}'
In Python, use the requests library to call the generate endpoint. This code is suitable for local testing:
import requestsresp = requests.post("http://127.0.0.1:11434/api/generate",json={"model": "llama3.2:3b", "prompt": "Introduce Ollama in one sentence", "stream": False},timeout=120,)resp.raise_for_status()print(resp.json()["response"])
If the API returns an error, troubleshoot as follows:
- Connection refused: The service is not running. Run
sudo systemctl start ollamaon Linux; ensure Ollama is running in the background on Windows. - 404 or model not found: The model name is misspelled or not pulled. Verify the name with
ollama listand runollama pull. - First request is slow: The model is loading into VRAM, which is normal. Wait and retry.
- Empty response or timeout: Check if VRAM is insufficient. Use
ollama psto view the running device and switch to a smaller model if necessary.
5. Modify the Model Storage Path
Model files typically occupy several GB to tens of GB. If your system disk is running out of space, migrate the model directory to another disk. Ensure the target disk has sufficient space and correct directory permissions before migrating.
Windows: The default model directory is %USERPROFILE%\.ollama\models. Steps:
- Quit Ollama from the system tray.
- Create a target directory, e.g.,
D:\ollama\models. - Add a new system environment variable
OLLAMA_MODELSwith the valueD:\ollama\models. - Restart Ollama.
- Copy the contents from the old directory to the new one. Confirm
ollama listshows your models before deleting the old directory.
Linux: When installed via systemd, models are saved in the ollama user’s home directory, typically /usr/share/ollama/.ollama/models. Steps:
sudo mkdir -p /data/ollama/modelssudo rsync -av /usr/share/ollama/.ollama/models/ /data/ollama/models/sudo chown -R ollama:ollama /data/ollamasudo systemctl edit ollama.service
In the opened editor, add the following and save:
[Service]Environment="OLLAMA_MODELS=/data/ollama/models"
Then reload the configuration and restart the service:
sudo systemctl daemon-reloadsudo systemctl restart ollamaollama list
If the service fails to start, run journalctl -u ollama to check logs. The most common causes are incorrect directory permissions or active SELinux/AppArmor policies.
Docker: The container saves models in /root/.ollama/models using a volume named ollama. To use a host directory instead, stop and remove the old container, copy the volume contents to the target directory, and restart with a bind mount:
docker stop ollamadocker rm ollamamkdir -p /data/ollamadocker run --rm -v ollama:/from -v /data/ollama:/to alpine cp -a /from/. /to/docker run -d --name ollama -p 127.0.0.1:11434:11434 -v /data/ollama:/root/.ollama ollama/ollama
After copying, verify the models are still visible using docker exec ollama ollama list.
6. Security Configuration
Ollama’s API does not have built-in authentication. Anyone who can access the port can call the model and consume computing resources. Therefore, the core of security configuration is restricting access scope.
- Keep default listening: By default, it listens only on
127.0.0.1, accessible only locally. Do not setOLLAMA_HOSTto0.0.0.0unless LAN access is explicitly required. - Do not expose port 11434 to the public internet: Avoid port forwarding on routers or opening the port on cloud servers. Be careful with Docker port mappings; omitting the
127.0.0.1:prefix exposes the port to all network interfaces. - Add protection for remote access: If access from other machines is necessary, place an Nginx or Caddy reverse proxy in front, enable TLS encryption, and configure access tokens or basic authentication. Use a firewall to restrict source IPs.
- Be cautious with cross-origin access:
OLLAMA_ORIGINScontrols which web origins can access the API. Keep the default settings unless explicitly needed; avoid setting it to a wildcard. - Review installation scripts: Always read the script contents before installation on Linux to ensure the download URLs and execution commands are trustworthy.
7. Summary
By following these steps, you can select a model based on your VRAM and disk space, complete the installation, pulling, and execution verification, and finally call the model via the API. The model directory can be migrated to another disk, and the service is accessible locally by default. When remote access is required, configure a reverse proxy, TLS, and authentication before opening access.
