Offline-Capable AI Coding Assistant: A Practical Free Setup Guide
Learn how to set up an AI coding assistant from scratch, initialize project context, connect cloud or local model services, use free quotas responsibly, manage context and costs, protect API keys, and integrate the assistant into a safe day-to-day development workflow.
Introduction: Connecting a Model Service Is the Final Step in an AI Coding Assistant Setup
Over the past two years, terminal-based AI coding assistants have become part of many developers’ daily workflows. Unlike early tools that only provided single-line completions, these assistants can read files in a project directory, search code, and repeatedly perform an “understand–modify–verify” loop around a task. Developers describe what they need in natural language, and the assistant proposes changes or even writes the code.
Most of these tools primarily provide an interaction layer:
- Collecting and trimming context;
- Calling a model service;
- Parsing the response;
- Applying changes to the file system.
The actual reasoning and code-generation capabilities come from the underlying model. That makes connecting the assistant to a usable model service the most important step during initial setup. A failed connection, exhausted quota, or unsuitable model can significantly reduce the quality of the entire experience.
This guide focuses on that setup process and follows the sequence “environment preparation → project context → model connection → free quota usage → daily collaboration.” It provides a practical configuration approach while explaining the underlying trade-offs.
1. The Overall Architecture: Three Layers and Three Decisions
1.1 Understand the Three Layers
Before using an AI coding assistant, it helps to understand how a typical interaction is structured:
- Project layer: Source code, configuration, documentation, and convention files that describe the project structure. These tell the assistant where and how changes should be made.
- Context layer: The content sent to the model for each request, including system prompts, conversation history, retrieved code snippets, and user instructions. It determines the input size and directly affects token usage.
- Model layer: The large language model that performs the actual reasoning. It may be hosted by a model provider or deployed by your team.
Many usability problems can be traced to one of these layers. Incorrect changes may result from missing project conventions. Slow responses and unexpectedly high costs are often caused by poor context management. Inconsistent results may be related to model selection.
1.2 Three Key Decisions
During initial setup, answer three questions:
- Environment: Is the runtime ready, and should the assistant run inside the project directory?
- Context: How will the assistant understand the project structure and follow coding conventions?
- Model: How will the model service be connected, and what does “free” actually mean for your budget?
The following sections address these questions in order.
2. Step One: Prepare the Environment
Runtime requirements vary between tools. The minimum versions in the official documentation are the only reliable source. The following checklist applies broadly:
- Runtime version: Terminal assistants are often built on JavaScript runtimes such as Node.js. Confirm that the installed version meets the minimum requirement.
- Git and version control: Assistants may read and modify many files. Commit a clean working state before using one so that changes can be reverted easily.
- Terminal support: Make sure the terminal displays Chinese characters and colored output correctly if your project requires them.
- Multiline terminal input: If
Shift+EnterorAlt+Enterdoes not insert a new line, enter a backslash (\) at the end of the line and press Enter to continue.
Example: Check the runtime and Git status
node -vgit status --short
After installation, run ai inside the project directory to enter the interactive interface. On the first launch, use help to inspect available commands and confirm that basic interaction works before proceeding.
3. Step Two: Initialize Project Context
Once a model service is connected, the assistant can respond, but it may not respond correctly. To consistently generate code that matches the project, establish explicit project context.
3.1 Generate a Project Instructions File
Run init from the project root:
> init
The assistant will scan the directory structure, build tools, and major dependencies, then generate an instructions file such as AGENTS.md in the project root. The file records the project purpose, technology stack, and common commands. In future sessions, relevant sections can be included as background information. This makes the file a one-time investment with long-term benefits.
Example: Project instructions file
# Project Instructions (AGENTS.md)## Technology Stack- Python 3.11, with uv for dependency management- FastAPI backend with SQLAlchemy for data access## Common Commands- Install dependencies: uv sync- Start the service: uv run uvicorn app.main:app --reload- Run tests: uv run pytest- Run linting: uv run ruff check .## Coding Standards- Follow PEP 8; functions must include type annotations and docstrings- New features must include unit tests with at least 80% coverage- Use Alembic for database migrations; do not modify existing migration files directly
3.2 Principles for Writing the Instructions File
To make the file useful, follow these principles:
- Describe how to work, not just what the project is: “This is a web project” is less useful than instructions for installing, running, testing, and checking the code.
- Keep it concise: The file may be included in context for every session. Unnecessary content consumes the available context budget. Keeping it under 100 lines is a practical goal.
- Maintain it as a team: Store it in version control. When coding standards change, update the file so the assistant does not rely on outdated rules.
- Keep commands accurate: Commands in the file should be the commands the team actually uses.
Example: Explicitly provide context in a task
Read @src/auth/token.py and analyze why token refresh fails after expiration.Fix the issue, add unit tests for the expired-token scenario, and runuv run pytest tests/auth to verify the change.
In addition to the project instructions file, you can use @ in a conversation to reference a specific file or directory. This forces the relevant code into the context and is more reliable than asking the assistant to guess which files matter.
4. Step Three: Connect a Model Service
4.1 Configuration Entry Points and Model Lists
During initial setup, the assistant will usually ask you to select a model provider. You can also enter the connect command at any time to open the configuration wizard:
- If credentials have already been configured, the wizard lists connected providers and their available models.
- If no credentials are configured, it guides you through selecting a provider and logging in or entering an API key.
Example: Model configuration menu
/ Connect a providerAdd provider----------------------------------------● Cloud Provider A default○ Cloud Provider B free tier available○ Cloud Provider C○ Custom endpoint compatible API
Each entry represents a different integration path:
| Provider type | Typical connection method | Suitable for | Considerations |
|---|---|---|---|
| Services with free or introductory quotas | Browser authorization or API key | Personal trials and prototypes | Usually subject to rate and usage limits; availability may vary |
| Mainstream commercial model services | API key; some support browser authorization | Daily development that requires strong code generation | Token-based billing requires cost monitoring |
| Local inference solutions | Local service address, usually without a key | Offline development, sensitive code, and predictable costs | Quality depends on local hardware; large models may be slow |
| Self-hosted gateways with a compatible API | Custom base_url and API key |
Enterprise governance and multi-model routing | The gateway’s availability and access control must be maintained |
After selecting a provider, press Enter and follow the prompts to complete authorization or paste the key. The configuration is then written to the appropriate file. Once setup is complete, use models to view available models and switch the default model with:
> models> model <provider>/<model-id>
4.2 The Cost Structure Behind “Free” Quotas
Free access is often the most attractive option, but it should be evaluated realistically. Most model services use usage-based pricing. Each request measures input and output tokens separately, and the corresponding prices are applied to each side. Making either side longer increases the cost.
When a conversation lasts dozens of turns and repeatedly reads several large files, a single request may quickly contain tens or even hundreds of thousands of tokens. This is one reason AI coding assistants can cost considerably more than ordinary chat usage.
“Free” access commonly comes from three sources:
- New-user trial credits: Usually limited by time or usage;
- Developer incentive programs: Free quotas available to developers in selected regions or under specific conditions;
- Bring-your-own-key access: Some terminal assistants include a set of freely callable models and also allow users to attach their own keys.
Each option has boundaries; none should be treated as unconditionally free. A more reliable cost strategy is to choose a model that is both affordable and capable enough for the task. If your team already uses a cloud platform such as Baidu AI Cloud, its API key can be connected through a compatible standard interface. For frequent and stable workloads, transparent usage-based billing may be easier to control than a limited free quota.
4.3 After Connecting: Verify Before Using
A configuration wizard reporting “connection successful” only confirms that the credentials were accepted. Perform a minimal verification to confirm that the model service is actually usable:
- Send a simple instruction: For example, ask the assistant to list the files in the current directory. Check whether the response is complete and clearly formatted.
- Check environment information: Some providers restrict available models by region or account status. If the list is empty or requests fail, return to the configuration wizard.
- Check the remaining quota: For free access, confirm the remaining amount and reset period so that a task is not interrupted halfway through.
- Prepare a backup model: Configure a second provider or select a lower-cost alternative in
models. You can switch when the primary model is rate-limited or unavailable.
Architecturally, this step is an integration test between the interaction layer and the model layer. The connection is complete only when credentials are valid, the model can be selected, and a usable response is returned.
Example: Configure a backup model
# List all providers and models> models# Switch when the primary model performs poorly or is rate-limited> model <provider>/<backup-model-id>
4.4 API Key Security: Four Rules
- Do not put keys in code: Store them in environment variables or credential files. Never hard-code them into source code or commit them to a repository. If a key has been committed, revoke it immediately and issue a new one.
Restrict file permissions: When the assistant handles login, credentials may be stored in a user directory such as
~/.local/share/opencode/auth.json. Make the file readable and writable only by the current user:chmod 600 ~/.local/share/opencode/auth.json
Use least privilege: Create restricted keys with only the permissions needed for model calls. Assign separate keys to projects or devices so they can be audited and revoked independently.
- Do not paste sensitive information into conversations: Redact real keys, internal addresses, and business data before sharing them. Conversation content is sent to the model service, so sensitive code and data must follow your organization’s data-classification policies.
5. Step Four: Use the Model Service Effectively
5.1 Example Configuration: Multiple Providers and Tiered Models
After connecting providers, manage them and their default models through a centralized configuration:
{// Centralize providers and models so conversations do not need to change"$schema": "https://opencode.ai/config.json","provider": {"<provider-a>": {"options": {"apiKey": "{env:PROVIDER_A_API_KEY}", // Read from an environment variable"baseURL": "https://<provider-a-endpoint>/v1"}},"<provider-b>": {"options": {"apiKey": "{env:PROVIDER_B_API_KEY}"}}},"model": "<provider-a>/<default-model>","small_model": "<provider-b>/<lightweight-model>" // Use a cheaper model for simple tasks}
small_model is an easily overlooked optimization. Tasks such as generating commit messages, summarizing files, or rewriting comments do not require the most expensive model. Routing them to a lightweight model can reduce spending with little impact on the user experience.
5.2 Match Models to Tasks
When faced with a long model list, choose models by task category:
- Complex logic and architecture design: Prefer models with strong coding capabilities, tool-calling support, and a sufficiently large context window.
- Routine questions and single-file edits: Choose a fast, lower-cost model.
- Document retrieval and summarization: A model with a large context window may process a long document in one request and reduce repeated reading.
- Privacy-sensitive work: Prefer local inference or a private deployment where appropriate.
5.3 Context Management: Precision and Continuity
The quality of an AI coding assistant depends heavily on how context is organized:
- Provide precise context: Use
@to reference relevant files and prevent unrelated content from consuming the context window. - Keep tasks focused: Use one task per conversation. Long sessions accumulate history, slow responses, and make the model more vulnerable to interference from outdated information.
- Compress context when needed: When the context approaches its limit, use
compact. The assistant summarizes the current session and starts a new context window, preserving key conclusions while removing redundant details. - Use subtasks: Delegate exploratory work, such as reading and summarizing a module, to a subtask. Keep only the conclusion in the main conversation. In terminal assistants of this type, the
taskcommand can start a subtask.
5.4 Integrate the Assistant into a Real Workflow
Understand the codebase → Plan the task → Apply changes → Build and test→ Review the code → Generate the commit message
- Understand the codebase: When joining an unfamiliar project, ask the assistant to map the directory structure, entry points, and data flow.
- Plan the task: For multi-step requirements, review the assistant’s plan first. After confirming it, execute the steps individually instead of applying a large change all at once.
- Apply changes: When only a few files are affected, use
diffto inspect the exact changes. For uncertain modifications, accept them in smaller blocks. - Build and test: Run
buildandtestcommands. Let the assistant inspect failures and help locate and fix the cause. - Review the code: Run
reviewbefore committing, then perform a human review as well. Never skip this step for security, concurrency, or data-consistency paths. - Generate a commit message: After completing the changes, use
committo generate a commit message that follows the project’s conventions.
Other useful commands include:
share: Share a session with colleagues for collaborative troubleshooting;unshare: Revoke sharing;export: Export the conversation for archiving;undo/redo: Undo or redo file changes.
6. Common Problems and Troubleshooting
6.1 Connection Problems
| Symptom | Possible cause | Recommended action |
|---|---|---|
The model list is empty after connect |
The account is not fully verified, or the service is unavailable in the current region | Sign in again or switch providers |
The request returns 401 or 403 |
The key is invalid, expired, or lacks permission | Issue a new key and confirm that it has model-call permissions |
| The request times out or the connection resets | Proxy or firewall restrictions | Check proxy settings and, if necessary, change the network environment |
| A local model does not respond | The model is not loaded or available memory is insufficient | Confirm that the local service is running or switch to a smaller model |
6.2 Quality Problems
- Changes do not follow project conventions: Check whether the project instructions file exists and includes the important rules.
- The assistant repeatedly changes the same code without converging: Stop asking follow-up questions, restate the goal, or split the task into smaller steps.
- Long sessions become slow and expensive: Compress the context or start a new session with only the necessary information.
- Command conflicts: Different tools may use similar commands with different meanings. After upgrading, check the help output before relying on a command.
A general troubleshooting approach is to first determine whether the problem is caused by model connectivity, context organization, or the tool itself. Then address the relevant layer instead of retrying blindly.
7. Cost Optimization and Security Boundaries
7.1 Five Ways to Reduce Costs
- Use lightweight models for lightweight tasks: Configure
small_modelto handle routine work. - Keep the project instructions file concise: Retain only conventions that genuinely help with coding.
- Close sessions promptly: Start a new session after finishing a task instead of processing unrelated requirements in the same conversation.
- Use caching when available: Many model services offer lower prices for repeated context prefixes. Keeping project instructions and stable prompts at the beginning of the context can improve cache reuse during related work.
- Review usage regularly: Inspect provider usage records, identify the largest sources of consumption, and optimize them specifically.
7.2 Four Security Boundaries
- Keep a human in the loop: Require manual confirmation for production changes, deletion operations, and permission changes.
- Make changes reversible: Work under version control and commit after each small, verifiable step.
- Redact sensitive information: Do not send keys, personal information, or critical business data in conversations.
- Follow compliance requirements: Before processing code or data, understand your organization’s rules for external model services. Use local inference or private deployment when necessary.
Conclusion
Getting started with a free AI coding assistant is not simply a matter of creating an account. It requires a complete setup across the environment, project context, and model service. Running connect only establishes the basic connection. The daily experience depends on the clarity of the project instructions file, the suitability of the selected model, the precision of the context, and deliberate cost management.
A practical progression is:
- Complete the first setup with a free quota or low-cost model;
- Build working habits in a real project;
- Gradually standardize configuration and workflows according to team requirements.
When the model service becomes a stable, controllable, and replaceable backend capability, the AI coding assistant can become a dependable productivity tool for everyday development.
