Deploying AI Coding Agents in Online and Local Modes

    This guide explains how to deploy AI coding agents using either cloud-hosted model services or local model inference runtimes. It covers environment preparation, configuration for both modes, switching between modes, tuning, security considerations, troubleshooting, and an acceptance checklist.

    Deploying AI Coding Agents in Online and Local Modes

    1. Deployment overview: two paths and three patterns

    An AI coding agent is an intelligent assistant built on top of a large language model. It integrates tools for file reading and writing, command execution, code search, and other development tasks. Users describe tasks in natural language, and the agent autonomously plans steps, invokes tools, inspects results, and iterates until the work is complete. Typical tasks include code generation, bug fixing, refactoring, and test writing.

    Whether an agent runs reliably depends mainly on three factors:

    • The runtime environment is complete and usable.
    • The model inference service is available.
    • Credentials and network settings are correctly configured.

    Based on where model inference happens, deployment can be divided into two modes:

    • Online mode: The agent calls a cloud-hosted model through an API. This mode offers higher capability ceilings, does not require local compute power, and is billed according to usage, but code must be sent to an external service.
    • Local mode: A local inference runtime loads an open-source model, and the agent connects to the local interface. Data stays inside the local environment, marginal cost is close to zero, but performance is constrained by local hardware.

    In practice, these two modes can be combined into three patterns:

    Pattern Model location Typical use cases
    Online only Cloud API General development, complex refactoring, highest reasoning capability
    Local only Local inference service Sensitive code, isolated intranets, high-frequency low-cost completion
    Hybrid Cloud for complex reasoning; local for completion and simple Q&A Routing by data sensitivity and task complexity while balancing capability and compliance

    Regardless of the pattern chosen, the deployment process can be broken into four stages:

    Environment preparation → Online mode configuration → Local mode configuration → Tuning and acceptance.

    2. Environment requirements before deployment

    2.1 Software dependencies

    AI coding agents are usually distributed as command-line programs, and many are built on JavaScript runtimes. Before deployment, confirm each dependency item:

    Dependency Version requirement Check command Description
    Operating system Mainstream desktop systems, such as Windows, Linux, or macOS Check system version information Unix-like environments must have a usable shell terminal
    JavaScript runtime 18 or later; 20 LTS recommended node --version Best balance between compatibility and long-term support
    Package manager Installed with the runtime; 10 or later recommended npm --version Used to install the agent CLI globally
    Version control tool Any recent stable version git --version Allows the agent to detect changes, review modifications, and perform version operations
    Memory 16 GB minimum; 32 GB or more recommended for local mode Check system information Insufficient memory can cause frequent paging during local inference
    Disk At least 20 GB of available space Check remaining disk space A single coding model is typically around 2–10 GB, allowing multiple versions to coexist

    Version control tools are often overlooked on Windows. On Windows, Git also provides a Unix-like command-line environment, which gives the agent the basic capabilities needed to run shell-like operations. It should therefore be installed as well.

    2.2 Step 1: Install the runtime

    1. Download the LTS installer from the runtime’s official website, run the installer, and complete the installation with default options. The installer will configure environment variables automatically.
    2. On Windows, you can also use the built-in package manager. On Unix-like systems, you can use the official install script or the distribution’s package manager.
    3. Close and reopen the terminal to ensure the new environment variables take effect.

    2.3 Step 2: Verify the base environment

    Run the following commands in the terminal:

    1. node --version # Expected output: v20.x.x
    2. npm --version # Expected output: 10.x.x
    3. git --version # Expected output: git version 2.x.x

    If all three commands return version numbers normally, the base environment is ready.

    2.4 Step 3: Install and start the command-line coding agent

    Install the agent globally through the package manager:

    1. npm install -g @anthropic-ai/claude-code

    Then run the following commands to confirm the installation:

    1. claude --version # A version number means installation succeeded
    2. claude doctor # Checks installation status, version, and auto-update configuration

    On the first run of claude, the tool will guide you through authentication. You can configure an API key or subscription authorization for a cloud model service, or connect to your own inference service or a compatible third-party interface. The exact process depends on the selected mode and is explained in Sections 3 and 4.

    3. Online mode: connecting to a cloud model service

    3.1 Prepare an API key

    Go to the console of the target service provider, create an API key, and keep it secure. The key is fully displayed only once when created. It is equivalent to account-level invocation permission, so follow these principles:

    • Least privilege: Create a separate key for the agent and set a spending cap.
    • One key per purpose: Use different keys for different teams and projects to make cost allocation and quick revocation easier.
    • Do not commit or expose it: Never write the key into code or commit it to a repository. Before sharing terminal screenshots, make sure the key is not visible.

    3.2 Configure the API key

    Set the key temporarily in the terminal:

    1. # Windows PowerShell
    2. $env:ANTHROPIC_API_KEY = "sk-ant-xxxxxxxx"
    3. # Linux / macOS
    4. export ANTHROPIC_API_KEY="sk-ant-xxxxxxxx"

    If you do not want the key to remain in shell history, create a .env file in the project root that is readable only by the current user:

    1. ANTHROPIC_API_KEY=sk-ant-xxxxxxxx
    2. ANTHROPIC_BASE_URL=https://<model-service-address>

    ANTHROPIC_BASE_URL overrides the default service address. It is needed when using an enterprise gateway, a private deployment of a compatible interface, or a third-party service. After configuring it, add .env to the ignore list to prevent accidental commits:

    1. # .gitignore
    2. .env
    3. .env.*
    4. !.env.example

    .env.example should contain only variable names and comments, not real values. It can be shared as a configuration template for the team.

    3.3 Configure the agent and network

    Enterprise intranets usually need to access the internet through a forward proxy:

    1. # Windows PowerShell: current session only
    2. $env:HTTPS_PROXY = "http://proxy.corp.local:8080"
    3. # Linux / macOS: current session only
    4. export HTTPS_PROXY="http://proxy.corp.local:8080"
    5. # Linux / macOS: permanent
    6. # Append the following line to ~/.bashrc or ~/.zshrc:
    7. # export HTTPS_PROXY="http://proxy.corp.local:8080"
    8. # Bypass the proxy for internal addresses; separate multiple addresses with commas
    9. $env:NO_PROXY = "localhost,127.0.0.1,.corp.local"

    You can run the following command through the proxy to verify connectivity and certificate-chain completeness:

    1. curl -I https://<model-service-address>
    • If the response indicates proxy authentication errors, add account information to the proxy address.
    • If certificate validation fails, the issue is usually that the enterprise self-signed root certificate has not been imported into the system trust chain, rather than a network failure.

    3.4 Start and verify

    Start the agent in the project repository root:

    1. cd <your-project-directory>
    2. claude

    After entering the interactive interface, run /status to confirm the account, model, and authentication method. Then give a simple task, such as “list all files in the current directory,” to verify that tool calls work. If authentication fails, run /logout and log in again, or check whether the key is complete.

    3.5 Notes for online mode

    • Capability differences: Different providers and models vary significantly in tool-call stability. After switching, use /model to change the model and re-test key capabilities such as file editing and command execution.
    • Context and cost: The agent continuously carries conversation history, and longer tasks consume more resources. For long-term use, combine session compression strategies with regular session cleanup.
    • Version updates: Agent CLI tools evolve quickly. You can upgrade with claude update, or disable automatic updates and upgrade after the team standardizes on a version.
    • Cloud-hosted models: Some cloud services support hosting models in dedicated instances and exposing compatible endpoints. In that case, the agent does not need to change; you only need to point ANTHROPIC_BASE_URL to the instance address.

    4. Local mode: deploy a local model inference service

    Local mode requires two components to work together: the inference runtime loads the model, exposes the interface, and uses GPU acceleration when available; the command-line agent points its model interface to that runtime.

    4.1 Install the local inference runtime

    On Windows, download the installer from the inference runtime’s official website and complete the wizard. After installation, the runtime will start as a background service when the system starts and listen on the local address by default. If you prefer a graphical management interface, you can also choose a desktop application that supports model downloads, model execution, and parameter adjustment. The usage is similar.

    4.2 Choose a model based on hardware

    Model size directly affects resource usage and response speed. Choose a model based on the following table:

    Hardware condition Recommended model size Recommended download command Characteristics
    Dedicated GPU with 8 GB or more VRAM 7B or larger coding model ollama pull <model-name>:<parameter-size> Better quality; supports larger context
    CPU only or low-spec (around 16 GB RAM) 3B coding model ollama pull <model-name>:3b Balanced speed and quality
    Very low-spec or functionality validation only 1.5B coding model ollama pull <model-name>:1.5b Small footprint and fast startup

    Replace <model-name>:<parameter-size> with the actual name and tag available in the runtime’s model library. Search for coding models in the official model library, verify the parameter count and quantization format, and then download.

    If you want to download multiple versions for comparison, use short tags that include the model name, such as <model-name>:7b-q4, to avoid overwriting each other.

    4.3 Set key environment variables

    If the agent and the inference runtime are not on the same machine, or if the inference service is deployed in a container or cloud host, the inference service must listen on an external address:

    1. # Takes effect after restarting the inference service
    2. $env:OLLAMA_HOST = "0.0.0.0:11434"
    3. [Environment]::SetEnvironmentVariable("OLLAMA_HOST", "0.0.0.0:11434", "User")
    4. # Linux / macOS: current session only
    5. export OLLAMA_HOST="0.0.0.0:11434"
    6. # Persist to user-level environment variables
    7. echo 'export OLLAMA_HOST="0.0.0.0:11434"' >> ~/.bashrc

    OLLAMA_HOST is the listening address and port of the inference service. Its value works as follows:

    • By default, it binds to the loopback address and is accessible only from the local machine. If set to 0.0.0.0, other devices on the same network can connect.
    • If set to 0.0.0.0, restrict source IPs through firewall or security-group rules. The interface has no authentication by default, so exposing it to the public internet is equivalent to opening model invocation to everyone.
    • For containerized deployments, set the host address to 0.0.0.0.

    To connect the agent to a local model, choose either of the following methods:

    1. Switch through environment variables: Set ANTHROPIC_BASE_URL=http://<inference-service-address>:11434, then set ANTHROPIC_AUTH_TOKEN to any non-empty placeholder value. Local services usually do not validate authentication, but some clients refuse requests when this variable is empty. You can store key variables in a .env file for unified management.
    2. Switch through a slash command: In the agent session, run /model, select a custom model provider, and enter the endpoint address and model name. You can also use the agent’s built-in third-party inference configuration feature, where the wizard completes the connection automatically.

    4.4 Verify the local inference service

    1. # List downloaded models
    2. ollama list
    3. # View the service version
    4. ollama --version
    5. # View model name, parameter size, and quantization format
    6. ollama show <model-name>:<parameter-size>
    7. # Verify that the model can generate code without going through the agent
    8. ollama run <model-name>:<parameter-size> "Write a quicksort function in Python"

    If all commands return normally, the inference service is ready. In the agent session, run /model to switch to the local model and run a small coding task for end-to-end validation.

    4.5 Make the inference service more stable

    • Keep-alive policy: By default, the inference service keeps the model loaded for a period after the last request before unloading it. If tasks are frequent, extend the keep-alive time to reduce repeated loading delays.
    • Context length: Agent tasks usually involve long single-request contexts. Increase the context length as appropriate, but this also increases VRAM or memory usage. Adjust it based on actual resource limits.
    • Model unloading: Before switching models, manually unload the current model to avoid multiple models occupying VRAM at the same time.

    5. Switching between modes: configuration system and switching methods

    5.1 Configuration precedence

    Agent configuration sources usually include managed enterprise settings, command-line arguments, project-level configuration (hidden folders beginning with a dot in the project root), user-level configuration, environment variables, and authentication information. The general principle is “nearest override wins”: configuration closer to a specific task has higher priority, while managed enterprise settings usually cannot be overridden. For example, on the same machine, project-level configuration can specify a different model for a specific repository.

    5.2 Five switching methods

    Method Operation Typical scenario
    In-session switching Run /model to select a model or custom provider Temporary comparison or single-task switching
    Launch parameter claude --model <model-name> Scripted or automated invocation
    Project-level configuration Settings file in the project root Different projects using different models or inference services
    User-level configuration Settings file in the user directory Personal default preferences
    Environment variables ANTHROPIC_BASE_URL, ANTHROPIC_AUTH_TOKEN Overall online/local switching, containers, CI environments

    5.3 One-step switching scripts

    The following scripts are examples. Create the script file in your user directory, enable the appropriate line as needed, and execute it with dot space in the terminal so that environment variables take effect in the current session:

    1. # Agent mode switching script; call with ". ./switch-agent-mode.ps1 online"
    2. param(
    3. [Parameter(Mandatory = $true)]
    4. [ValidateSet("online", "local")]
    5. [string]$Mode
    6. )
    7. if ($Mode -eq "online") {
    8. # Online mode: make sure a valid ANTHROPIC_API_KEY is configured in .env before switching.
    9. Remove-Item Env:ANTHROPIC_BASE_URL -ErrorAction SilentlyContinue
    10. Remove-Item Env:ANTHROPIC_AUTH_TOKEN -ErrorAction SilentlyContinue
    11. Write-Host "Switched to online mode (cloud model service)."
    12. }
    13. else {
    14. # Local mode: change the address to the actual inference service host.
    15. $env:ANTHROPIC_BASE_URL = "http://127.0.0.1:11434"
    16. $env:ANTHROPIC_AUTH_TOKEN = "local"
    17. Write-Host "Switched to local mode (local inference service)."
    18. }
    1. # Linux / macOS version
    2. mode=$1
    3. case "$mode" in
    4. online)
    5. # Online mode: make sure a valid ANTHROPIC_API_KEY is configured in .env before switching.
    6. unset ANTHROPIC_BASE_URL ANTHROPIC_AUTH_TOKEN
    7. echo "Switched to online mode (cloud model service)."
    8. ;;
    9. local)
    10. # Local mode: change the address to the actual inference service host.
    11. export ANTHROPIC_BASE_URL="http://127.0.0.1:11434"
    12. export ANTHROPIC_AUTH_TOKEN="local"
    13. echo "Switched to local mode (local inference service)."
    14. ;;
    15. *)
    16. echo "Usage: . ./switch-agent-mode.sh {online|local}"
    17. ;;
    18. esac

    Note that export only affects the current terminal session. New windows, integrated terminals, and some processes launched by graphical applications may not read the variable. In such cases, use user-level configuration or permanent environment variables.

    6. Tuning: quantization, context, and agent loops

    The real-world experience of local mode depends largely on the balance of three parameters: quantization precision, context length, and agent loop rounds.

    • Quantization precision: Quantization reduces model size by lowering the numeric precision of weights. A 4-bit quantization can typically compress a 7B model to around 4 GB with limited precision loss, making it suitable for many coding tasks. Higher-precision quantization is more stable but requires significantly more VRAM.
    • Context length: A typical coding agent first searches relevant files, then sends code, errors, and conversation history to the model. Insufficient context can truncate retrieved code, causing the agent to repeatedly “miss” important files. In local deployments, prioritize sufficient context within the available VRAM.
    • Agent loop rounds: The agent sends a model request each time it performs a tool call, and tool output enters the next round of context. Token consumption therefore grows rapidly as task complexity increases. Set maximum rounds and timeout limits per task to prevent the agent from entering loops and continuously consuming compute resources.

    Two engineering practices are useful:

    • Small commits: Have the agent complete one verifiable small goal at a time and record it immediately in version control. If the direction goes wrong, rollback is fast.
    • Tighten permissions: During read-only analysis, restrict file-writing and command-execution permissions. After confirming the task direction, enable editing and command execution.

    7. Security and compliance considerations

    • Route by data sensitivity: Code involving core business logic should prefer local mode. General scripts and documentation generation can use online mode. In hybrid mode, project-level configuration can assign different models to different repositories.
    • Control inference-service exposure: Local inference services do not authenticate by default. Before binding to a non-loopback address, make sure the network boundary is restricted. For remote access, prefer a jump host or internal network channel.
    • Credentials and logs: Agent logs, terminal history, and session records may contain keys and code snippets. Redact them before sharing, and rotate API keys regularly.
    • Seat and usage management: Administrators can use the official deployment toolkit to install the agent at scale and use managed enterprise policies to standardize models, permissions, and update schedules. For high usage, use logging and monitoring alerts to track invocation volume and cost by project.
    • Cloud-hosted inference: If you want to deploy a local model on a cloud host or container platform for elasticity and stability, you can store model files in object storage, run the inference service on a container platform, and expose it through load balancing. On the agent side, you only need to point the endpoint address to the service. The deployment pattern is otherwise the same as local deployment.

    8. Troubleshooting common issues

    Symptom Common cause Solution
    Command not recognized The terminal was not restarted after installation; the global package directory is not in PATH Reopen the terminal, and configure PATH manually if necessary
    Local model connection refused The inference service is not running, or it is bound to loopback but accessed remotely Start the service and set the address to 0.0.0.0:11434
    Authentication failure or 401 Missing, expired, or malformed key Check the key, whether it has been revoked, and log in again
    429 or quota errors Rate or usage limits exceeded Reduce concurrency, spread usage over time, or request a quota increase
    502, 504, or timeouts Proxy or gateway interception; slow model cold start Check HTTPS_PROXY and NO_PROXY; increase the timeout in local mode if needed
    Context overflow Context length is insufficient Increase context length, split the task, or clear the session
    Output quality drops noticeably Model is too small, quantization is too aggressive, or inference is not using GPU Use a larger model, higher-precision quantization, or verify GPU driver availability
    Configuration does not take effect Variables only apply to the current session; a higher-priority configuration exists Use user-level configuration or permanent environment variables, and check project-level configuration

    9. Acceptance checklist

    1. Base environment: The runtime, package manager, and version control tool versions meet requirements, and the terminal can recognize the agent command normally.
    2. Online mode: Authentication succeeds, tool calls can read and write files normally, and usage and cost match expectations.
    3. Local mode: The inference service remains running with the system, models load correctly, and GPU acceleration is enabled when available.
    4. Dual-mode switching: Both /model and the switching scripts can switch between the two modes, and configuration remains valid after opening a new terminal.
    5. Security and compliance: Keys are not committed to repositories, inference-service exposure is controlled, and logs and terminal history have been redacted.
    6. Engineering constraints: Per-task rounds and timeouts have limits, code changes are committed in small steps, and rollback is fast.

    Conclusion

    Online mode and local mode are not mutually exclusive. They should be combined flexibly based on data sensitivity, task complexity, and hardware conditions. Use online mode for higher capability ceilings, use local mode to protect data boundaries, and add fine-grained tuning for quantization, context, and agent rounds. This approach helps build a stable, compliant, and continuously evolving AI coding workflow.