Connecting Multiple LLMs to AI Coding Assistants Through a Local Proxy
This article explains how a local proxy can connect an AI coding assistant to several large language model backends without changing the client. It covers the three-layer architecture, protocol adaptation, capability negotiation, routing based on task, cost, latency, and data boundaries, and security governance. It also lists practical guidelines for testing, configuration, streaming, retries, caching, and fallback visibility.
1. Problem Background: Decoupling Workflows from Model Services
AI coding assistants are useful not only because a particular model gives high-quality answers. They work because they form a complete workflow: reading the project structure, retrieving relevant files, editing code, running commands, checking the results, and correcting course across multiple turns of conversation. Once developers get used to this way of working, they do not want to relearn their tools every time they switch models. Enterprises, for their part, do not want to tie their development processes to a single model service.
In practice, teams often need to switch models. Chinese-language tasks may call for a model better suited to Chinese. Long-context tasks need a larger context window. Cost-sensitive scenarios need a lower unit price. When code cannot leave an internal network, a locally deployed model is required. Some tasks also need multimodal input or stronger tool-calling ability. If every model change requires a new client, updated configuration, and rewritten plugins, both developer experience and operations costs rise noticeably.
A more practical approach is to add a local proxy layer between the client and upstream model services. The client keeps its existing command-line or desktop interface. The proxy converts requests into the protocol each upstream understands, then converts streaming output, tool calls, reasoning information, and image results back into a format the client can use. In this setup, choosing a model becomes a routing change rather than a tool change. The assistant’s workflow is decoupled from the implementation of each model service.
2. Overall Architecture: Client, Proxy Layer, and Model Services
A typical architecture has three layers.
The first layer is the client layer. It includes command-line coding assistants, desktop coding assistants, editor plugins, and similar tools. These clients expose only a unified access endpoint to the proxy. They do not hold keys for upstream services directly, and they do not need to know how upstream protocols differ.
The second layer is the local proxy layer, which is the core of this approach. It typically contains an access gateway, protocol adapters, a capability negotiation module, model routing, session state management, streaming conversion, tool-call conversion, audit logging, and monitoring metrics. The proxy can run as a local process or as a container in a controlled environment.
The third layer is the upstream model service layer. It can include public model services, privately deployed inference services, or small models running locally. The proxy does not assume that all upstreams have the same capabilities. Instead, it adapts through capability declarations and runtime probing.
A simplified diagram:
Coding assistant client|vLocal proxy: access gateway -> protocol adaptation -> capability negotiation -> routing policy| |v vStreaming, tool, and multimodal result conversion Upstream model services A/B/C
The main value of this structure is a unified entry point with differences absorbed in one place. The client handles only one interaction protocol. The proxy centralizes the handling of upstream differences, and the routing layer centralizes policies for cost, latency, and data boundaries.
3. Protocol Adaptation: More Than Field Mapping
Protocol adaptation is not just renaming a request field and forwarding it. A complete adaptation includes at least the following steps.
Request mapping: Convert the client’s message structure into a conversation structure the upstream accepts. This covers system prompts, history trimming, sampling parameters such as temperature, and maximum output length. For parameters an upstream does not support, the proxy should explicitly ignore them or degrade gracefully. It should not silently pass them through and cause errors.
Response mapping: Convert the upstream’s text, finish reason, and usage information into the client’s unified format. Error codes also need normalization. For example, upstream rate limits, authentication failures, context length overflows, and content filtering should map to standard proxy errors. This lets clients show clear messages and lets the routing layer retry when appropriate.
Streaming event conversion: Upstream streaming events differ in meaning. Some push incremental text, some push typed events, and some return usage only at the end. The proxy must give clients a stable sequence of increments and handle cancellation, timeouts, and mid-stream disconnects correctly.
Tool-call conversion: Coding assistants often need the model to request tools such as reading files, searching, or running commands. The proxy must convert client tool definitions into a form the upstream recognizes. It must also convert the upstream’s tool-call arguments into structures the client can execute, and perform basic validation on those arguments. If an upstream does not support tool calling, capability negotiation should disable that path. The proxy should not let the model emit fake tool-call text.
Reasoning information and multimodal result conversion: Some tasks need reasoning summaries or image results. The proxy should separate reasoning information that can be shown to users from internal fields that should not be exposed. It should also wrap binary results such as images in a consistent format.
Session state management: In multi-turn conversations, the proxy must maintain upstream session identifiers, context trimming policies, and cache keys. Caches must respect tenant, project, and privacy boundaries. Results from different users must never be mixed.
4. Capability Negotiation: Not Every Model Can Be Connected Unconditionally
“Unified access to multiple models” can be misread as meaning any model can be plugged in seamlessly. In practice, each model must meet minimum capability requirements, and the proxy must negotiate capabilities with each one.
Maintain a capability matrix for each upstream. It should include at least streaming support, tool-calling support, multimodal input support, context window size, maximum output length, system prompt support, concurrent request support, and whether error code semantics are stable. Capability information can come from static declarations or runtime probing. Static declarations allow quick checks at startup. Runtime probing detects configuration drift or upstream degradation.
After receiving a client request, the proxy should choose an upstream that meets the task’s needs. A task that requires tool calling must not be routed to a model without tool-calling support. A long-context task must not go to a model with a window that is too small. A request containing images must not go to a text-only model. If no upstream fully matches, the proxy needs a clear fallback policy. It can reject the request with an explanation, trim the context, disable tool calling, or switch to a backup model. The goal of capability negotiation is not to make every model appear compatible. The goal is stable operation within each model’s capability limits.
5. Routing Strategy: Decisions Based on Task, Cost, Latency, and Data Boundaries
The routing layer is where most of the value in a multi-model setup sits. A good approach is rules first, scoring second, and circuit breaking as a safeguard.
Rules come first and handle hard constraints. Requests whose data must stay inside the internal network can only go to local or privately deployed models. Requests containing images can only enter the multimodal pool. Requests that need tool calling can only enter pools of models that support it. Hard constraints are not part of scoring, so cost or latency metrics cannot override them.
Scoring is secondary and handles soft goals. Among candidates that meet the hard constraints, the proxy scores models using weighted factors: task type, historical success rate, current latency, cost per unit, and context length needs. For example, short code completion prioritizes latency, long-document understanding prioritizes window size, and batch refactoring prioritizes cost and stability. Scoring weights should be configurable so teams can adjust them as their needs change.
Circuit breaking handles stability. When an upstream repeatedly times out, its error rate rises, or it hits rate limits, the proxy should trip the breaker quickly and switch to a backup model. When the upstream recovers, traffic should ramp up gradually to avoid overwhelming it. Degradation requires particular care for tool-calling requests. Tool-call formats vary across models, so a silent switch to a model without tool support can cause execution failures. In that case, returning a clear error is preferable to silent degradation.
A structured routing configuration example:
routes:- name: local-privateprotocol: generic-chatendpoint: http://127.0.0.1:8080/v1capabilities: [stream, tool_call]constraints:data_boundary: private- name: long-context-poolprotocol: generic-chatendpoint: http://127.0.0.1:8081/v1capabilities: [stream]constraints:min_context: 128k- name: cost-efficient-poolprotocol: generic-chatendpoint: http://127.0.0.1:8082/v1capabilities: [stream, tool_call]constraints:task_type: batch-refactor
6. Deployment and Security Governance
Although the local pro
