Deploying AI Coding Agents in Online and Local Modes
This guide explains how to deploy AI coding agents using either cloud-hosted model services or local model inference runtimes. It covers environment preparation, configuration for both modes, switching between modes, tuning, security considerations, troubleshooting, and an acceptance checklist.
Deploying AI Coding Agents in Online and Local Modes
1. Deployment overview: two paths and three patterns
An AI coding agent is an intelligent assistant built on top of a large language model. It integrates tools for file reading and writing, command execution, code search, and other development tasks. Users describe tasks in natural language, and the agent autonomously plans steps, invokes tools, inspects results, and iterates until the work is complete. Typical tasks include code generation, bug fixing, refactoring, and test writing.
Whether an agent runs reliably depends mainly on three factors:
- The runtime environment is complete and usable.
- The model inference service is available.
- Credentials and network settings are correctly configured.
Based on where model inference happens, deployment can be divided into two modes:
- Online mode: The agent calls a cloud-hosted model through an API. This mode offers higher capability ceilings, does not require local compute power, and is billed according to usage, but code must be sent to an external service.
- Local mode: A local inference runtime loads an open-source model, and the agent connects to the local interface. Data stays inside the local environment, marginal cost is close to zero, but performance is constrained by local hardware.
In practice, these two modes can be combined into three patterns:
| Pattern | Model location | Typical use cases |
|---|---|---|
| Online only | Cloud API | General development, complex refactoring, highest reasoning capability |
| Local only | Local inference service | Sensitive code, isolated intranets, high-frequency low-cost completion |
| Hybrid | Cloud for complex reasoning; local for completion and simple Q&A | Routing by data sensitivity and task complexity while balancing capability and compliance |
Regardless of the pattern chosen, the deployment process can be broken into four stages:
Environment preparation → Online mode configuration → Local mode configuration → Tuning and acceptance.
2. Environment requirements before deployment
2.1 Software dependencies
AI coding agents are usually distributed as command-line programs, and many are built on JavaScript runtimes. Before deployment, confirm each dependency item:
| Dependency | Version requirement | Check command | Description |
|---|---|---|---|
| Operating system | Mainstream desktop systems, such as Windows, Linux, or macOS | Check system version information | Unix-like environments must have a usable shell terminal |
| JavaScript runtime | 18 or later; 20 LTS recommended | node --version |
Best balance between compatibility and long-term support |
| Package manager | Installed with the runtime; 10 or later recommended | npm --version |
Used to install the agent CLI globally |
| Version control tool | Any recent stable version | git --version |
Allows the agent to detect changes, review modifications, and perform version operations |
| Memory | 16 GB minimum; 32 GB or more recommended for local mode | Check system information | Insufficient memory can cause frequent paging during local inference |
| Disk | At least 20 GB of available space | Check remaining disk space | A single coding model is typically around 2–10 GB, allowing multiple versions to coexist |
Version control tools are often overlooked on Windows. On Windows, Git also provides a Unix-like command-line environment, which gives the agent the basic capabilities needed to run shell-like operations. It should therefore be installed as well.
2.2 Step 1: Install the runtime
- Download the LTS installer from the runtime’s official website, run the installer, and complete the installation with default options. The installer will configure environment variables automatically.
- On Windows, you can also use the built-in package manager. On Unix-like systems, you can use the official install script or the distribution’s package manager.
- Close and reopen the terminal to ensure the new environment variables take effect.
2.3 Step 2: Verify the base environment
Run the following commands in the terminal:
node --version # Expected output: v20.x.xnpm --version # Expected output: 10.x.xgit --version # Expected output: git version 2.x.x
If all three commands return version numbers normally, the base environment is ready.
2.4 Step 3: Install and start the command-line coding agent
Install the agent globally through the package manager:
npm install -g @anthropic-ai/claude-code
Then run the following commands to confirm the installation:
claude --version # A version number means installation succeededclaude doctor # Checks installation status, version, and auto-update configuration
On the first run of claude, the tool will guide you through authentication. You can configure an API key or subscription authorization for a cloud model service, or connect to your own inference service or a compatible third-party interface. The exact process depends on the selected mode and is explained in Sections 3 and 4.
3. Online mode: connecting to a cloud model service
3.1 Prepare an API key
Go to the console of the target service provider, create an API key, and keep it secure. The key is fully displayed only once when created. It is equivalent to account-level invocation permission, so follow these principles:
- Least privilege: Create a separate key for the agent and set a spending cap.
- One key per purpose: Use different keys for different teams and projects to make cost allocation and quick revocation easier.
- Do not commit or expose it: Never write the key into code or commit it to a repository. Before sharing terminal screenshots, make sure the key is not visible.
3.2 Configure the API key
Set the key temporarily in the terminal:
# Windows PowerShell$env:ANTHROPIC_API_KEY = "sk-ant-xxxxxxxx"# Linux / macOSexport ANTHROPIC_API_KEY="sk-ant-xxxxxxxx"
If you do not want the key to remain in shell history, create a .env file in the project root that is readable only by the current user:
ANTHROPIC_API_KEY=sk-ant-xxxxxxxxANTHROPIC_BASE_URL=https://<model-service-address>
ANTHROPIC_BASE_URL overrides the default service address. It is needed when using an enterprise gateway, a private deployment of a compatible interface, or a third-party service. After configuring it, add .env to the ignore list to prevent accidental commits:
# .gitignore.env.env.*!.env.example
.env.example should contain only variable names and comments, not real values. It can be shared as a configuration template for the team.
3.3 Configure the agent and network
Enterprise intranets usually need to access the internet through a forward proxy:
# Windows PowerShell: current session only$env:HTTPS_PROXY = "http://proxy.corp.local:8080"# Linux / macOS: current session onlyexport HTTPS_PROXY="http://proxy.corp.local:8080"# Linux / macOS: permanent# Append the following line to ~/.bashrc or ~/.zshrc:# export HTTPS_PROXY="http://proxy.corp.local:8080"# Bypass the proxy for internal addresses; separate multiple addresses with commas$env:NO_PROXY = "localhost,127.0.0.1,.corp.local"
You can run the following command through the proxy to verify connectivity and certificate-chain completeness:
curl -I https://<model-service-address>
- If the response indicates proxy authentication errors, add account information to the proxy address.
- If certificate validation fails, the issue is usually that the enterprise self-signed root certificate has not been imported into the system trust chain, rather than a network failure.
3.4 Start and verify
Start the agent in the project repository root:
cd <your-project-directory>claude
After entering the interactive interface, run /status to confirm the account, model, and authentication method. Then give a simple task, such as “list all files in the current directory,” to verify that tool calls work. If authentication fails, run /logout and log in again, or check whether the key is complete.
3.5 Notes for online mode
- Capability differences: Different providers and models vary significantly in tool-call stability. After switching, use
/modelto change the model and re-test key capabilities such as file editing and command execution. - Context and cost: The agent continuously carries conversation history, and longer tasks consume more resources. For long-term use, combine session compression strategies with regular session cleanup.
- Version updates: Agent CLI tools evolve quickly. You can upgrade with
claude update, or disable automatic updates and upgrade after the team standardizes on a version. - Cloud-hosted models: Some cloud services support hosting models in dedicated instances and exposing compatible endpoints. In that case, the agent does not need to change; you only need to point
ANTHROPIC_BASE_URLto the instance address.
4. Local mode: deploy a local model inference service
Local mode requires two components to work together: the inference runtime loads the model, exposes the interface, and uses GPU acceleration when available; the command-line agent points its model interface to that runtime.
4.1 Install the local inference runtime
On Windows, download the installer from the inference runtime’s official website and complete the wizard. After installation, the runtime will start as a background service when the system starts and listen on the local address by default. If you prefer a graphical management interface, you can also choose a desktop application that supports model downloads, model execution, and parameter adjustment. The usage is similar.
4.2 Choose a model based on hardware
Model size directly affects resource usage and response speed. Choose a model based on the following table:
| Hardware condition | Recommended model size | Recommended download command | Characteristics |
|---|---|---|---|
| Dedicated GPU with 8 GB or more VRAM | 7B or larger coding model | ollama pull <model-name>:<parameter-size> |
Better quality; supports larger context |
| CPU only or low-spec (around 16 GB RAM) | 3B coding model | ollama pull <model-name>:3b |
Balanced speed and quality |
| Very low-spec or functionality validation only | 1.5B coding model | ollama pull <model-name>:1.5b |
Small footprint and fast startup |
Replace <model-name>:<parameter-size> with the actual name and tag available in the runtime’s model library. Search for coding models in the official model library, verify the parameter count and quantization format, and then download.
If you want to download multiple versions for comparison, use short tags that include the model name, such as <model-name>:7b-q4, to avoid overwriting each other.
4.3 Set key environment variables
If the agent and the inference runtime are not on the same machine, or if the inference service is deployed in a container or cloud host, the inference service must listen on an external address:
# Takes effect after restarting the inference service$env:OLLAMA_HOST = "0.0.0.0:11434"[Environment]::SetEnvironmentVariable("OLLAMA_HOST", "0.0.0.0:11434", "User")# Linux / macOS: current session onlyexport OLLAMA_HOST="0.0.0.0:11434"# Persist to user-level environment variablesecho 'export OLLAMA_HOST="0.0.0.0:11434"' >> ~/.bashrc
OLLAMA_HOST is the listening address and port of the inference service. Its value works as follows:
- By default, it binds to the loopback address and is accessible only from the local machine. If set to
0.0.0.0, other devices on the same network can connect. - If set to
0.0.0.0, restrict source IPs through firewall or security-group rules. The interface has no authentication by default, so exposing it to the public internet is equivalent to opening model invocation to everyone. - For containerized deployments, set the host address to
0.0.0.0.
To connect the agent to a local model, choose either of the following methods:
- Switch through environment variables: Set
ANTHROPIC_BASE_URL=http://<inference-service-address>:11434, then setANTHROPIC_AUTH_TOKENto any non-empty placeholder value. Local services usually do not validate authentication, but some clients refuse requests when this variable is empty. You can store key variables in a.envfile for unified management. - Switch through a slash command: In the agent session, run
/model, select a custom model provider, and enter the endpoint address and model name. You can also use the agent’s built-in third-party inference configuration feature, where the wizard completes the connection automatically.
4.4 Verify the local inference service
# List downloaded modelsollama list# View the service versionollama --version# View model name, parameter size, and quantization formatollama show <model-name>:<parameter-size># Verify that the model can generate code without going through the agentollama run <model-name>:<parameter-size> "Write a quicksort function in Python"
If all commands return normally, the inference service is ready. In the agent session, run /model to switch to the local model and run a small coding task for end-to-end validation.
4.5 Make the inference service more stable
- Keep-alive policy: By default, the inference service keeps the model loaded for a period after the last request before unloading it. If tasks are frequent, extend the keep-alive time to reduce repeated loading delays.
- Context length: Agent tasks usually involve long single-request contexts. Increase the context length as appropriate, but this also increases VRAM or memory usage. Adjust it based on actual resource limits.
- Model unloading: Before switching models, manually unload the current model to avoid multiple models occupying VRAM at the same time.
5. Switching between modes: configuration system and switching methods
5.1 Configuration precedence
Agent configuration sources usually include managed enterprise settings, command-line arguments, project-level configuration (hidden folders beginning with a dot in the project root), user-level configuration, environment variables, and authentication information. The general principle is “nearest override wins”: configuration closer to a specific task has higher priority, while managed enterprise settings usually cannot be overridden. For example, on the same machine, project-level configuration can specify a different model for a specific repository.
5.2 Five switching methods
| Method | Operation | Typical scenario |
|---|---|---|
| In-session switching | Run /model to select a model or custom provider |
Temporary comparison or single-task switching |
| Launch parameter | claude --model <model-name> |
Scripted or automated invocation |
| Project-level configuration | Settings file in the project root | Different projects using different models or inference services |
| User-level configuration | Settings file in the user directory | Personal default preferences |
| Environment variables | ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN |
Overall online/local switching, containers, CI environments |
5.3 One-step switching scripts
The following scripts are examples. Create the script file in your user directory, enable the appropriate line as needed, and execute it with dot space in the terminal so that environment variables take effect in the current session:
# Agent mode switching script; call with ". ./switch-agent-mode.ps1 online"param([Parameter(Mandatory = $true)][ValidateSet("online", "local")][string]$Mode)if ($Mode -eq "online") {# Online mode: make sure a valid ANTHROPIC_API_KEY is configured in .env before switching.Remove-Item Env:ANTHROPIC_BASE_URL -ErrorAction SilentlyContinueRemove-Item Env:ANTHROPIC_AUTH_TOKEN -ErrorAction SilentlyContinueWrite-Host "Switched to online mode (cloud model service)."}else {# Local mode: change the address to the actual inference service host.$env:ANTHROPIC_BASE_URL = "http://127.0.0.1:11434"$env:ANTHROPIC_AUTH_TOKEN = "local"Write-Host "Switched to local mode (local inference service)."}
# Linux / macOS versionmode=$1case "$mode" inonline)# Online mode: make sure a valid ANTHROPIC_API_KEY is configured in .env before switching.unset ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKENecho "Switched to online mode (cloud model service).";;local)# Local mode: change the address to the actual inference service host.export ANTHROPIC_BASE_URL="http://127.0.0.1:11434"export ANTHROPIC_AUTH_TOKEN="local"echo "Switched to local mode (local inference service).";;*)echo "Usage: . ./switch-agent-mode.sh {online|local}";;esac
Note that export only affects the current terminal session. New windows, integrated terminals, and some processes launched by graphical applications may not read the variable. In such cases, use user-level configuration or permanent environment variables.
6. Tuning: quantization, context, and agent loops
The real-world experience of local mode depends largely on the balance of three parameters: quantization precision, context length, and agent loop rounds.
- Quantization precision: Quantization reduces model size by lowering the numeric precision of weights. A 4-bit quantization can typically compress a 7B model to around 4 GB with limited precision loss, making it suitable for many coding tasks. Higher-precision quantization is more stable but requires significantly more VRAM.
- Context length: A typical coding agent first searches relevant files, then sends code, errors, and conversation history to the model. Insufficient context can truncate retrieved code, causing the agent to repeatedly “miss” important files. In local deployments, prioritize sufficient context within the available VRAM.
- Agent loop rounds: The agent sends a model request each time it performs a tool call, and tool output enters the next round of context. Token consumption therefore grows rapidly as task complexity increases. Set maximum rounds and timeout limits per task to prevent the agent from entering loops and continuously consuming compute resources.
Two engineering practices are useful:
- Small commits: Have the agent complete one verifiable small goal at a time and record it immediately in version control. If the direction goes wrong, rollback is fast.
- Tighten permissions: During read-only analysis, restrict file-writing and command-execution permissions. After confirming the task direction, enable editing and command execution.
7. Security and compliance considerations
- Route by data sensitivity: Code involving core business logic should prefer local mode. General scripts and documentation generation can use online mode. In hybrid mode, project-level configuration can assign different models to different repositories.
- Control inference-service exposure: Local inference services do not authenticate by default. Before binding to a non-loopback address, make sure the network boundary is restricted. For remote access, prefer a jump host or internal network channel.
- Credentials and logs: Agent logs, terminal history, and session records may contain keys and code snippets. Redact them before sharing, and rotate API keys regularly.
- Seat and usage management: Administrators can use the official deployment toolkit to install the agent at scale and use managed enterprise policies to standardize models, permissions, and update schedules. For high usage, use logging and monitoring alerts to track invocation volume and cost by project.
- Cloud-hosted inference: If you want to deploy a local model on a cloud host or container platform for elasticity and stability, you can store model files in object storage, run the inference service on a container platform, and expose it through load balancing. On the agent side, you only need to point the endpoint address to the service. The deployment pattern is otherwise the same as local deployment.
8. Troubleshooting common issues
| Symptom | Common cause | Solution |
|---|---|---|
| Command not recognized | The terminal was not restarted after installation; the global package directory is not in PATH |
Reopen the terminal, and configure PATH manually if necessary |
| Local model connection refused | The inference service is not running, or it is bound to loopback but accessed remotely | Start the service and set the address to 0.0.0.0:11434 |
| Authentication failure or 401 | Missing, expired, or malformed key | Check the key, whether it has been revoked, and log in again |
| 429 or quota errors | Rate or usage limits exceeded | Reduce concurrency, spread usage over time, or request a quota increase |
| 502, 504, or timeouts | Proxy or gateway interception; slow model cold start | Check HTTPS_PROXY and NO_PROXY; increase the timeout in local mode if needed |
| Context overflow | Context length is insufficient | Increase context length, split the task, or clear the session |
| Output quality drops noticeably | Model is too small, quantization is too aggressive, or inference is not using GPU | Use a larger model, higher-precision quantization, or verify GPU driver availability |
| Configuration does not take effect | Variables only apply to the current session; a higher-priority configuration exists | Use user-level configuration or permanent environment variables, and check project-level configuration |
9. Acceptance checklist
- Base environment: The runtime, package manager, and version control tool versions meet requirements, and the terminal can recognize the agent command normally.
- Online mode: Authentication succeeds, tool calls can read and write files normally, and usage and cost match expectations.
- Local mode: The inference service remains running with the system, models load correctly, and GPU acceleration is enabled when available.
- Dual-mode switching: Both
/modeland the switching scripts can switch between the two modes, and configuration remains valid after opening a new terminal. - Security and compliance: Keys are not committed to repositories, inference-service exposure is controlled, and logs and terminal history have been redacted.
- Engineering constraints: Per-task rounds and timeouts have limits, code changes are committed in small steps, and rollback is fast.
Conclusion
Online mode and local mode are not mutually exclusive. They should be combined flexibly based on data sensitivity, task complexity, and hardware conditions. Use online mode for higher capability ceilings, use local mode to protect data boundaries, and add fine-grained tuning for quantization, context, and agent rounds. This approach helps build a stable, compliant, and continuously evolving AI coding workflow.
