A Local Gateway for Programming Agents to Switch Between LLMs Freely

1. Background: Protocol Fragmentation Locks Models and Tools Together

LLM APIs are converging in capability but diverging in interface style. Anyone using AI programming tools today runs into this reality:

1. Chat Completions style: Built around a messages array, it has the widest ecosystem, and many model services offer compatible endpoints by default.
2. Responses style: Models a task as a response object containing tools, state, and output, which maps more naturally to agent workflows.
3. Messages style: Explicitly separates system prompts from conversation messages, with content organized as typed blocks (text, tool use, thinking).
4. Generative resource style: Routes directly by model resource name, with a request body structure that differs significantly from the others.

Command-line programming agents typically implement only one of these protocols completely, and by default send requests to a bound service address. This creates three typical pain points:

• Switching models means switching tools: To use another model service, you either give up your preferred tool or modify its source code.
• Scattered keys: Each tool’s config file stores its own copy of the key, making rotation and management costly.
• Missing observability: There is no global view of token usage, failed requests, or latency distribution, so troubleshooting relies on guesswork.

The solution is not complicated: run a gateway process on the local loopback address that speaks multiple protocols upstream and translates requests into the target upstream’s protocol downstream; then pair it with a configuration manager that redirects each agent’s config file to point at this gateway with minimal changes. The following sections cover capability design, configuration management, engineering, and verification.

2. Core Capability One: Multi-Protocol Coexistence and Semantic Conversion

1. Protocol Detection on the Same Port

The gateway listens on a fixed port on 127.0.0.1 (8787 for illustration) and identifies the protocol style by request path characteristics: paths containing chat/completions go to the chat completions parser, paths containing responses go to the responses parser, paths containing messages go to the messages parser, and those shaped around a model resource name go to the generative parser.

The key architectural decision is introducing an internal intermediate representation (IR): any protocol is first parsed into the IR, then serialized from the IR into the target protocol. This way, converting between n protocols requires only 2n adapters instead of n² pairwise converters, and the marginal cost of adding a new protocol is constant.

2. Key Request-Side Mappings

Semantic differences between protocols concentrate in five areas, each of which needs a mapping:

• System prompts: Some protocols use a top-level standalone field, while others inject them into the messages array as a system/developer role. This must never be dropped during conversion.
• Content structure: Plain text and multimodal blocks (images, tool results) are organized differently, requiring bidirectional conversion of typed blocks.
• Tool definitions: Parameters are almost always JSON Schema, but field naming and call ID formats differ. IDs need stable mapping (keeping the original ID prefix and maintaining a mapping table is recommended).
• Sampling parameters: max_tokens and max_completion_tokens are synonyms under different names; temperature and top_p share the same semantics and can be passed through directly.
• Stop reasons: stop, end_turn, tool_use, length, and others need a semantic alignment table—agents rely on this to decide whether it is their turn to execute a tool or whether the model is done.

3. Streaming Output Conversion

Streaming in the three mainstream protocols is based on SSE, but event granularity differs: some emit events by message lifecycle (start, delta, end), while others emit incremental JSON fragments directly. The correct approach is to maintain a session state machine inside the gateway: fold the upstream event stream into a unified internal delta sequence, then expand it into the target protocol’s event sequence, covering five lifecycle event categories—start, content delta, tool-call argument delta, end, and error. Send periodic heartbeat comments to prevent intermediary proxies from timing out and dropping the connection, and aggregate usage statistics back at the end.

A common incorrect implementation is “forward each event as it arrives.” Once upstream and downstream event granularities differ, fields get dropped or ordering breaks.

4. Tool Calls and Reasoning Content

• Tool calls under streaming: Arguments arrive as chunked JSON strings. They must be accumulated and assembled in the state machine, then validated for legality before deciding whether to pass through or repair.
• Reasoning content: Some models emit a thinking process. The correct approach is to map it to a dedicated content type in the target protocol (a thinking/reasoning block), never mixing it into the main body. Otherwise the agent may treat reasoning text as a reply or as tool arguments, triggering cascading errors.

3. Core Capability Two: Minimally Invasive Rewrites by the Configuration Manager

1. Four Design Principles

1. Change only necessary keys: Do not reorder, beautify, or “helpfully upgrade” other settings. Keep the scope of change minimal.
2. Preserve formatting: Keep comments, key order, and indentation intact. Once TOML/JSON is deserialized and re-serialized, comments and order are lost, and user-written notes vanish into thin air. The common approach is surgical modification based on line-level location or a format-preserving editing library.
3. Atomic writes: Write to a temporary file and flush to disk first, then rename within the same partition to replace. This avoids leaving a half-written config if the process is killed midway.
4. Rollback support: Back up a .bak copy before replacing, so a one-click restore is possible if something goes wrong.

2. What Each Agent Type Typically Needs Changed (Illustrative)

Agent Type	Config Location (Illustrative)	Keys to Modify
Messages-protocol programming agent	settings.json under its user directory	Service address, auth token, and default model in environment variables
Responses-protocol programming agent	config.toml under its user directory	Add a provider section pointing at the gateway and a default model; keep the available model list in a separate JSON file
Generative-protocol CLI tool	settings.json and .env	Point the service address at the gateway, switch auth to API key, set the default model
Open-source agent with pluggable providers	Its own config file	Add a gateway provider entry, with models written in provider/model composite form

“Changing config” beats “changing code” because agent upgrades won’t overwrite customizations, and the switch action can be scripted, audited, and rolled back.

3. Reference Atomic Write Implementation

import os, shutil
from pathlib import Path
def atomic_write(path, new_text: str):
    path = Path(path)
    tmp = path.with_suffix(path.suffix + ".tmp")
    with open(tmp, "w", encoding="utf-8") as f:
        f.write(new_text)
        f.flush()
        os.fsync(f.fileno())
    if path.exists():
        shutil.copy2(path, path.with_suffix(path.suffix + ".bak"))
    os.replace(tmp, path)  # rename within the same filesystem is atomic

4. Engineering and Security Considerations

1. Bind to loopback only: Default to 127.0.0.1, never listen on 0.0.0.0. The gateway holds multiple upstream keys, and exposing it to the LAN is like leaving the key vault door open.
2. Key consolidation: Upstream keys exist only on the gateway side, and agents hold just a local token. Rotating keys only touches the gateway, and all agents are unaffected.
3. Observability: Log upstream, model, latency, token usage, and error codes per request. Remember to redact logs, since prompts often contain business code.
4. Routing and fault tolerance: Route model names in provider/model composite form to different upstreams; configure multiple upstreams per provider for failover; apply exponential backoff retries for rate limits and 5xx errors, but ensure requests carrying tool calls remain idempotent.
5. Local vs. cloud trade-offs: A local gateway is sufficient for single-machine development. For team sharing, the same architecture can be deployed on an internal network, or you can reuse a cloud provider’s unified access capability—for example, Baidu AI Cloud’s model services typically provide endpoints compatible with mainstream protocols, which can be attached to the gateway as just another upstream, with the architecture unchanged.

5. Minimal Verification: From curl to Consistency Tests

curl -sS http://127.0.0.1:8787/v1/messages \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer <local-token>" \
  -d '{
    "model": "provider-a/model-x",
    "max_tokens": 1024,
    "stream": true,
    "messages": [{"role": "user", "content": "Introduce yourself in one sentence"}]
  }'

Checklist: Each protocol endpoint returns a structurally valid response; streaming event sequences are complete (with both start and end); requests with tool definitions reach the agent intact, and multi-turn continuation works normally after tool results are returned; reasoning content appears in a separate block; stop reasons map correctly.

Common pitfalls: System prompts get dropped during conversion, causing abrupt changes in agent behavior; tool call IDs are not stably mapped, so results don’t match up; an intermediary reverse proxy buffers SSE, producing “fake streaming”; synonymous parameter fields are not mapped and get rejected by the upstream.

6. Summary

The essential value of a local multi-protocol gateway is decoupling “tools” from “models”: consolidating protocol differences, key management, and observability/auditing into a single process. Two core capabilities are indispensable—protocol conversion pivoted on an intermediate representation (preserving streaming, tools, and reasoning), and minimally invasive, atomic, rollback-capable config rewriting. A gradual rollout is recommended: get a single protocol and single upstream working first, then expand the protocol and upstream list step by step, and solidify consistency tests as regression tests, so that “switching models” truly becomes a click rather than a rebuild.