How to Configure AI Coding Assistants to Use Locally Hosted Models
Summary: A practical guide to replacing vendor-hosted cloud models with locally deployed ones in AI coding assistants. It explains the HTTP-based call mechanism (OpenAI-compatible chat completion and message-style protocols), how to stand up a local inference server (engine selection, model choice, quantization precision, startup parameters and a quick connectivity test), how to configure an assistant's provider settings and avoid common mistakes such as wrong base URL paths, mismatched model names, short timeouts and stale config caches, how to tune the model for code completion (FIM), chat and agent modes, plus a troubleshooting reference, a cost comparison between self-hosting and cloud APIs, deployment best practices, and a path for scaling from individual use to shared or production inference on Baidu AI Cloud
Many developers hit the same wall when setting up an AI coding assistant: the free tier included with a subscription runs out quickly, metered billing has to be budgeted carefully, and teams with strict data requirements would rather keep their code and conversations inside their own network. At the same time, the open-source model ecosystem has matured to the point where a reasonably capable, instruction-following model can run on a consumer graphics card or a single ordinary inference server.
That naturally leads to a question: can a coding assistant stop depending on the vendor’s cloud model and call a locally deployed model instead?
It can, and the change is not complicated. This article starts with how the call mechanism works, then walks through setting up a local inference service, configuring the assistant, practical guidance for typical scenarios such as code completion, and finally troubleshooting and cost evaluation.
1. How It Works: What Sits Between the Assistant and the Model
Most coding assistants expose only a “switch model” dropdown, which makes it look as if you have to choose one model for good. But if the configuration also exposes three things — a service endpoint, an API key, and a model name — you already have what you need to connect a local model.
The reason is that the vast majority of these assistants do not perform inference themselves. They send HTTP requests to a model service and receive text, code completions, or tool-call results back. So as long as your local inference service implements the API protocol the assistant expects, the assistant can use it exactly like a cloud model.
Two kinds of compatible protocols are common today:
- OpenAI-style chat completion API: the request body contains a
messagesarray (withroleandcontentfields) plus parameters such asmodel,temperature,max_tokens, andstream. Responses are usually JSON, and withstream=truethe content is pushed incrementally over SSE. This style is implemented by a large number of inference frameworks and hosting platforms, so its compatibility is the broadest. - Message-style inference API: requests are organized with fields such as
system,messages, andmax_tokens, and streaming is supported as well. Some assistants offer a “compatible mode” option under their “custom model” settings to specify which protocol to use.
+------------------+ HTTP / JSON (OpenAI-compatible or message-style) +------------------+| Coding Assistant| -------------------------------------------------> | Local Inference || (IDE plugin/CLI) | messages: [ {role:"system", ...}, | Service || | {role:"user", ...} ] | (Chat API compat)|| | <------------------------------------------------- | Quantized weights|+------------------+ streamed text / tool-call results +------------------+
Once this relationship is clear, three practical points follow:
- The protocol is what matters. When choosing an inference service, prioritize the interface style your assistant requires. Most tools support the OpenAI-compatible protocol, and message-style APIs usually work in “compatible mode” as well; some early or niche tools may support only a few fixed formats.
- Get the endpoint path right. Assistants differ in what they expect: some want the URL to stop at
/v1, some want it to point directly at/chat/completions, and some ask for both an “API URL” and an “endpoint URL”. A wrong path is the most common cause of “cannot connect” problems. - Local does not automatically mean weaker. Whether a model can handle the job depends on its size, reasoning ability, context length, and your hardware — not on whether it is labeled “local” or “cloud”.
2. Setting Up a Local Inference Service: Get the Model Running First
Before touching the assistant, you need a working model service. The steps below follow the order: pick an engine → pick a model → decide on quantization → start the server.
1) Choose an inference engine. Local inference engines fall into two broad categories:
- Consumer / general-purpose: supports CPUs and a range of GPUs, can run on consumer graphics cards or even machines without a discrete GPU, and is comparatively easy to install and configure. Suitable for individual developers.
- High-performance / production-oriented: targets multi-GPU and high-concurrency scenarios, with more optimization for throughput and parallelism strategies. Suitable for a shared inference service used by a team.
Choose based on your hardware and use case. There is no need to chase a high-performance engine at all costs, which can otherwise lead to compatibility problems on consumer hardware.
2) Choose a model. Coding tasks place higher demands on a model’s code ability. Focus on three things when selecting:
- Whether it has been specifically trained on code, or at least has strong instruction-following ability;
- Whether it supports tool calling (function calling) — the underlying mechanism that lets an assistant read files, run commands, and search a codebase;
- Whether the context length is sufficient. 32K is a reasonable starting point, because coding assistants pack multi-turn conversations, file contents, and retrieval results into the same context.
3) Decide on quantization precision. Taking the quantization formats commonly used for text generation as an example, the usual bit widths are 4-bit, 5-bit, 6-bit, and 8-bit:
- Lower bit width means smaller VRAM usage, but some loss in output quality;
- For demanding tasks such as code generation, prefer 6-bit precision or higher; 8-bit is fairly close to full precision in practice.
4) Start the service. Launch an inference service that exposes a chat-compatible API. Adjust the parameters as needed (and check them against the actual options of the engine you use):
python -m my_infer_server \--model ./models/code-7b-instruct \--quantization q6_k \--n-gpu-layers 99 \--ctx-size 16384 \--host 0.0.0.0 --port 8080 \--api-key local-token
What each parameter means:
--model: path to a local model file or directory;--quantization: quantization precision;--n-gpu-layers: number of layers offloaded to the GPU; with enough VRAM, offloading everything to the GPU gives a clear speedup;--ctx-size: context window size;--api-key: access token for the local service. Even for a service on your own machine, setting a key is advisable so that other hosts on the LAN cannot call it freely.
After starting, leave the assistant alone and send one request with the standard library to confirm the service itself is reachable:
import jsonimport urllib.requestBASE = "http://127.0.0.1:8080/v1/chat/completions"payload = {"model": "local-code-7b","messages": [{"role": "user", "content": "Write a quicksort in Python"}],"max_tokens": 512,"stream": False,}req = urllib.request.Request(BASE,data=json.dumps(payload).encode("utf-8"),headers={"Content-Type": "application/json","Authorization": "Bearer local-token"},)with urllib.request.urlopen(req, timeout=120) as resp:print(resp.read().decode("utf-8"))
This verifies the endpoint, the key, and the model name directly, and it lets you inspect the response format. If it responds normally, move on to configuring the assistant.
3. Configuring the Assistant: Point the Model Source at Local
Configuration option names and locations vary widely across coding assistants, but the core information is the same. If the assistant can import a configuration file, something like the following works as a reference (field names differ per assistant):
{"providers": [{"id": "local","type": "openai-compatible","base_url": "http://127.0.0.1:8080/v1","api_key": "local-token","models": ["local-code-7b"],"context_limit": 16384}],"request_timeout": 300,"stream": true}
If you can only fill in fields through a GUI, enter the service endpoint, the key, and the model name one by one, and select compatible mode.
A few details that commonly go wrong:
- The model name must match the server exactly. The name in the assistant is only a label; what actually selects a model is the
modelfield in the request body. If the name registered on the server differs from what you typed in the assistant, you will usually get a “model not found” error. - Allow enough timeout. Local inference is bounded by hardware performance, and time-to-first-token is often noticeably higher than with a mature cloud service. Project-level agent tasks take longer still, so setting the timeout to a few minutes is reasonable.
- Restart the assistant after changing configuration. Some assistants cache the model list at startup, so a changed config without a restart is easily mistaken for “the configuration is not taking effect”.
- Be careful about mixing local and cloud models. Some tools automatically pick a model per task — a large model for planning, a small one for quick edits. In that case it is best to point everything at your local service, or, when only a single configuration is supported, fix it to one option first, so you do not incur unexpected charges.
After configuring, verify in order: basic conversation → code generation → file reading and editing. Only when all three work has the integration really succeeded.
4. Practical Scenarios: Completion, Chat, and Agent Mode
The same local model faces different challenges in different modes, and the countermeasures differ accordingly.
(1) Code completion: prefer a dedicated model trained for fill-in-the-middle
Code completion and chat are two distinct task modes. A chat model receives an instruction plus context and is expected to produce a complete answer, whereas completion typically requires the model to understand the code before the cursor (the prefix) and after it (the suffix), and generate only the missing part in between. This capability is called fill-in-the-middle (FIM).
Using a general chat model for completion directly usually produces extra content, repeats code that is already written, or ignores the code files present in the context. Two more reliable options:
- Use a dedicated code completion model trained for FIM;
- Build a completion proxy on top of the chat-compatible API: when the assistant sends a completion request, assemble the prefix and suffix into a prompt with start/end markers, call the chat model, then trim the extra output and return it. The idea looks like this:
PREFIX_TOKEN, SUFFIX_TOKEN = "<fim_prefix>", "<fim_suffix>"def completion_proxy(prefix: str, suffix: str) -> str:# Assemble the code around the cursor into a fill-in-the-middle promptprompt = f"{PREFIX_TOKEN}{prefix}{SUFFIX_TOKEN}{suffix}"text = chat_complete(prompt, max_tokens=128, temperature=0.1)# Drop the repeated prefix and keep only the completionreturn text[len(prefix):] if text.startswith(prefix) else text
If the assistant lets you specify a “completion endpoint”, put the proxy’s address there. If it does not, the inference service itself needs to support a FIM endpoint.
(2) Code chat: accuracy comes first
In chat mode the model must understand the request, explain it, and produce complete code, which demands higher reasoning quality:
- Open the context window as wide as possible, at least 16K to start;
- A temperature of 0.1–0.3 is a reasonable choice, keeping output stable and closer to engineering conventions;
- If the model has a thinking/reasoning mode, it usually improves accuracy on complex programming tasks, but at a noticeable cost in latency — switch it on and off according to task difficulty.
(3) Agent mode: tool calling and context management are the core
Agent mode is the most capable and the most demanding mode for a local model: the assistant lets the model decide the next action — read a file, edit code, run a command, inspect the result — and loops until the task is done. This raises the bar for local models:
- Tool-call reliability: the model must emit structured call information. Some quantized models occasionally produce malformed JSON or missing fields, which is especially visible at very low bit widths. Choosing a model at 6-bit precision or higher is advisable.
- Attention over long contexts: as the context grows, models tend to miss key information in the middle. Enabling repository retrieval (RAG) in the assistant helps, so that only relevant snippets enter the context.
- Managing the reasoning trace: models with thinking mode enabled generate long reasoning blocks that consume both time and context. Make sure the assistant parses reasoning content correctly and separates it from the final reply, and turn thinking mode off when a long task does not need it.
- Small before large: validate the tool-calling chain on a small repository or a single file first, then scale up to the full project.
5. Troubleshooting Common Issues
| Symptom | Likely cause | What to do |
|---|---|---|
| Connection failed / endpoint unreachable | Wrong URL path, service not running, or cross-host access with the server not listening on 0.0.0.0 |
Check the /v1 path; check the --host parameter |
| Authentication failed | Empty or mismatched key | Make sure the assistant and the server use the same key |
| Model not found | Model name in the assistant differs from the server | Query the model list endpoint (for example /v1/models) to confirm the name |
| Rambling or irrelevant replies | Model does not match the task, or quantization is too aggressive | Switch to a code-specialized model, raise quantization precision |
| Reply cut off midway | max_tokens too small, or insufficient context left for the output |
Raise the output limit, trim the context |
| Tool calls fail frequently | Model does not support tool calling, or generates unstable formatting | Raise quantization precision, reduce the number of available tools |
| Slow or timing out | Insufficient compute, or too many concurrent tasks | Increase the timeout, lower concurrency, enable GPU acceleration |
| No change after editing config | Assistant cached the old configuration | Restart the assistant (and the IDE if necessary) |
Some inference services expose OpenAI-compatible management endpoints that are handy for a quick health check:
- Model list: query
GET /v1/modelsto confirm which models the service has loaded; - VRAM usage: query
GET /metricsto assess remaining room for context and concurrency; - Cache management: if the assistant repeatedly sends prompts with the same prefix, calling
POST /v1/cache/clearclears the prompt cache and avoids wasting resources on duplicates.
Exact paths and parameters depend on the inference service you use; refer to its documentation.
6. Cost Evaluation: Self-Hosted or Cloud?
“Free tier exhausted” is only the surface symptom. The real question is whether self-hosting a local model is more economical than continuing with cloud services. Three dimensions are worth comparing:
| Dimension | Self-hosted local model | Cloud hosted service |
|---|---|---|
| Upfront cost | Requires your own GPU or server | Close to zero |
| Cost profile | Mostly hardware investment; higher upfront, but low marginal cost whether usage is light or heavy | Pay per usage; fixed allowance with a monthly plan |
| Capability level | Depends on hardware and model choice; generally behind flagship cloud models | Access to the newest flagship models |
| Data compliance | Data stays on your machine or in your network | Data must be sent to the service provider |
A practical decision rule is average daily usage. Roughly:
- Daily token consumption = requests per day × tokens per request (input plus output)
- Annual token consumption = daily × 365
Self-hosting starts to pay off when the annual cost of metered billing consistently exceeds the hardware investment. Conversely, light users may find metered billing or a monthly plan more economical. Keep in mind, though, that self-hosted models usually trail flagship cloud models in capability. If your work relies heavily on complex code generation and long agent chains, quality deserves as much weight as cost.
For individual developers, reusing an idle graphics card is the lowest-cost starting point. For teams and organizations, a hybrid strategy is generally the better recommendation:
- Route everyday completion and simple Q&A to the local model to reduce API volume;
- Hand complex architecture design and difficult debugging to flagship cloud models;
- Keep tasks involving core code or sensitive data entirely local.
7. Getting the Most Out of a Local Model
How a local model performs depends not only on the model itself but also on how it is deployed:
- Hardware: memory must be enough to hold the model weights, and more VRAM allows longer cached contexts and faster responses. Platforms with shared memory architecture typically allocate VRAM as a proportion of system memory, so pay attention to this during configuration.
- Concurrency: a single user rarely runs into trouble, but for shared deployments it is worth capping concurrent requests so that queueing does not become excessive.
- API security: if the service needs to be reachable from other devices on the LAN, always set an access key and never expose it directly to the public internet.
- Version management: model files are large, so keep version records, and run a regression pass after upgrading a model or the inference engine to make sure quality has not suddenly degraded.
- Performance evaluation: measure tokens generated per second and time-to-first-token, and pay most attention to real tasks such as code generation, function explanation, and bug triage.
- Follow the ecosystem: the open-source community keeps releasing better-quantized and more capable models, so it is worth checking periodically and swapping in new ones, optionally with scheduled automatic updates.
8. Scaling to Teams and Enterprises: A General Architecture on Baidu AI Cloud
Once a local model moves from individual experimentation to team sharing or even production, a new set of problems appears: hardware maintenance, elastic scaling, and security governance. Using cloud services can remove a large part of that infrastructure work. Taking Baidu AI Cloud as an example, the following general-purpose products can be combined as needed:
- Elastic compute: deploy the inference service on cloud servers or GPU cloud servers, avoiding hardware procurement and data center maintenance while scaling with load.
- Baige platform: a computing platform built for large-model workloads, covering the full chain from model deployment and performance optimization to inference service delivery — a fit for teams that need to serve inference at scale.
- Qianfan platform: provides model development and application-building capabilities, making it convenient to package self-hosted or fine-tuned models as standardized APIs.
- Vector database VectorDB: backs semantic retrieval over a codebase, so the assistant loads only the code snippets relevant to the current task and gets more accurate results within a limited context.
- Object storage BOS: stores model files, code snapshots, and datasets, and can serve as a single entry point for model version management.
- Log service BLS and cloud monitoring BCM: collect inference service logs and runtime metrics centrally, with monitoring alerts that surface timeouts and VRAM overflows early and reduce operational overhead.
There is no need to adopt everything at once: for individual trials, start with a lightweight cloud server; for shared inference services, consider the Baige platform; and bring in a vector database when you need managed code retrieval. Availability and feature details vary by region, so check the official documentation for specifics.
9. Conclusion
“Free tier not enough” is essentially a symptom of how dependent developers are on the vendor’s way of calling a model. With a local inference service and a compatible API, control over the model moves back into your hands:
- Controllable cost: a one-time hardware investment, with no per-token billing afterwards;
- Controllable data: code and conversations stay on your machine or in your network;
- Replaceable capability: models keep improving along with the community ecosystem.
All of this rests on a correct understanding of the call mechanism: a local inference service implements a compatible API → the assistant is configured with the model endpoint and key → the model is tuned for the three scenarios of completion, chat, and agent work. When local resources are not enough for more demanding needs, combining them with the elastic compute and managed capabilities of a platform such as Baidu AI Cloud allows a smooth scale-out, and lets you find the balance point between cost and quality that suits you best.
