A Local Gateway for Cross-Protocol LLM Access in Coding Agents

1. Background: How Protocol Silos Trap Coding Agents

1.1 The agent work loop

Command-line coding agents have evolved from “code completion” into “task executors”: they receive a goal (fix a bug, perform a refactor, add missing tests) and autonomously run a loop of “plan → call the model → parse the reply → execute tools (read/write files, run commands, run tests) → send tool results back to the model” until the task converges. Calls to model services in this loop rely on HTTP APIs, and the key problem is that the vast majority of agents implement only one interface protocol—endpoint paths, authentication methods, and request/response structures are all hardcoded around that protocol.

1.2 Four mainstream interface styles coexist

Large model service interfaces in the industry broadly fall into four styles:

• Chat completion protocol: the most widely compatible de facto standard. Requests center on a messages array (role + content), responses carry results in choices, and streaming output is delivered as incremental chunks; tool calls hang off the tool_calls field with arguments as JSON strings.
• Response event protocol: a newer interface aimed at agent scenarios. It returns a set of typed events, natively expresses reasoning state and tool execution state, and is semantically closer to a “state machine.”
• Message block protocol: centers on a list of content blocks, where text, tool calls, and tool results are different block types; streaming is delivered through block-level start/delta/stop events, and thinking processes can be carried as an independent block.
• Native generation protocol: organizes multi-turn conversations with a contents/parts structure, using separate fields for system instructions and tool declarations.

Model providers typically implement only one or two of these, and agents typically speak only one. Once the two sides mismatch, tools and models become hard-bound by protocol: even when a more suitable, cheaper, or enterprise-procured model exists, it cannot be used in a convenient agent.

1.3 Three pain points of protocol binding

• Limited model choice: instead of “choosing models by task” (strong models for refactoring, fast models for routine Q&A), the reality is “choosing models by agent.”
• Uncontrolled cost: some agents default to subscription channels, so high-frequency, low-difficulty requests also consume high-value quota, with no lever for task-tiered scheduling.
• Fragmented management: different tools and team members each have their own model egress, making enterprise-level key rotation, usage auditing, and content compliance difficult to implement uniformly.

2. Solution Overview: Protocol Translator + Configuration Manager

The idea is straightforward: run a gateway process on the local loopback address (e.g., 127.0.0.1:8000, any port), have it “play” the protocol the agent expects on the client side and call the protocol the target model actually supports on the upstream side, performing semantic-level translation in between. Then add a configuration manager that precisely repoints the agent’s configuration to the gateway. Together, the agent “thinks nothing has changed,” while the actual backend has been swapped.

2.1 Architecture layers

• Protocol service layer: implements endpoints and data models for each of the four protocol styles so that client-visible behavior matches the original vendor interface.
• Translation layer: the core of the solution, converting requests and responses in both directions, including a streaming converter.
• Routing layer: maps model aliases declared by the agent to actual upstream models and credentials, supporting task-tiered routing. Upstream can be a self-hosted inference service or a model service provided by a cloud platform such as Baidu AI Cloud—as long as it exposes a standard protocol, the gateway can manage it uniformly.
• Credential layer: a local key store. Real keys are stored only locally in the gateway, and the agent configuration holds only a placeholder token pointing to the gateway.
• Configuration manager: reads and writes agent configuration files (see Section 4).

2.2 How it differs from a reverse proxy

Changing only the service address to forward traffic is a reverse proxy; the gateway performs semantic-level translation: for the same conversation, the four protocols differ in message organization, streaming events, tool-call expression, and where thinking content is carried, and the gateway must ensure the converted semantics are lossless. Another often-overlooked detail is round-trip consistency: conversation history is maintained by the agent, and every turn sends the full history back; the tool-call blocks the gateway output last turn will be sent back this turn in the agent protocol and translated back to the upstream format, and the fields must line up exactly, otherwise multi-turn tool calls will inevitably break.

2.3 The full journey of one request

1. The agent sends a request to the gateway’s local address using the only protocol it speaks;
2. The gateway parses the request, and the routing layer locates the upstream model and credential by model alias;
3. The translation layer converts the request into the upstream protocol (system prompts, history messages, and tool declarations are all converted);
4. The upstream returns incremental chunks in streaming mode;
5. The streaming converter reassembles upstream chunks into the client protocol’s event sequence and sends them down;
6. The agent executes tools and sends results back in its own protocol; the gateway translates them back to the upstream format and enters the next turn, looping until the task ends.

3. Four Hard Problems in Protocol Conversion

3.1 Message structure: normalize first, then dialectize

The message structure differences among the four protocols go far beyond field names: system prompts sometimes use a separate field and sometimes are agreed to be the first system message; tool results are sometimes a separate block type and sometimes mixed into message content; role naming is also inconsistent. The correct engineering approach is to introduce an intermediate representation (IR): first normalize the inbound request into an internal model (unified abstractions for messages, tools, and thinking), then generate the target protocol payload from the IR. Direct pairwise conversion of four protocols would require N×(N-1) converters and is hard to keep consistent; with an IR, only 2N are needed (inbound normalization + outbound generation), and conversion correctness can be tested centrally.

3.2 Streaming output: rebuilding the event skeleton on a text stream

Streaming conversion is the hardest part. Clients of some protocols do not accept “plain text deltas” but rely on a typed event sequence (session start, block start, block delta, block stop, finish) to drive rendering. When the upstream provides only a text stream, the gateway must synthesize a complete event skeleton: emit a session event at the start; emit a block start event upon the first text delta; forward deltas segment by segment; and append block stop and finish events after the upstream ends. In addition, a buffering strategy must balance forwarding latency against event completeness, and upstream interruptions must produce a finish that the client can parse correctly (e.g., an event with error semantics) to avoid the agent hanging.

3.3 Tool calls: aggregation, splitting, and round-trip consistency

Tool calls are the lifeblood of agents, and three areas are most error-prone:

• Streaming aggregation: upstream tool arguments are JSON string fragments delivered in pieces; they must be aggregated by call index before a complete tool-call block can be produced. If the client requires incremental display while streaming, the aggregated result must be re-split into deltas.
• Round-trip consistency: call ID generation and passthrough, JSON argument serialization details, and the order of parallel calls must remain stable after two rounds of translation.
• Tool definition alignment: each protocol organizes tool schema fields differently; map field by field and handle semantic differences in defaults and required items.

In addition, token usage statistics and error codes also need translation so that quota prompts and retry logic on the agent side continue to work.

3.4 Thinking process: map when possible, never fabricate

Newer strong models commonly support outputting a thinking process, and some protocols carry it as a separate block or field. The conversion strategy should be honest mapping: if both sides have a corresponding carrier, map it (preserving the block type so the agent renders the reasoning chain as usual); if the upstream does not provide it, omit it truthfully and never fabricate placeholder content. Whether thinking content is sent back upstream with history must also be trimmed according to upstream protocol requirements to avoid unrecognized fields polluting the context.

4. Configuration Management: Four Design Principles for Seamless Switching

Once the protocol works, the agent still needs to send requests to the gateway. Editing configuration by hand is feasible, but agent config files often mix in a large number of user customizations, and breaking them is costly. Engineering the “config change” itself is key to making such a solution usable.

4.1 Minimal modification principle: change only the few keys necessary to implement the switch—service address, auth token, and model name—and leave everything else untouched. Existing user configuration is not “incidentally reformatted,” and problems can be rolled back in seconds.

4.2 Preserve comments, order, and indentation: do not reorder JSON keys, and do not drop TOML comments. The safe implementation abandons the crude “deserialize then reserialize” path and uses text-level targeted editing: locate the line containing the target key and replace only the value itself.

4.3 Atomic writes and backups: write to a temporary file first, verify it parses, then atomically replace the original file; automatically keep a backup before writing and provide one-click restore. In multi-file scenarios (address in one file, key in another), both writes must either succeed together or roll back together, eliminating “half-switched” states.

4.4 Common edits for four configuration forms:

Config form	Typical carrier	Keys to write	Notes
JSON settings file	settings.json in the user directory	service address variable, auth token variable, and model name variable in the env section	Preserve key order and indentation; do not reformat wholesale
TOML config file	config.toml	add a custom provider table and specify the model	Keep comments as-is; the model list can be stored in a separate local JSON
dotenv environment file	.env	gateway address variable and placeholder auth variable	Usually loaded at process start; restart the agent after switching
Multi-file combination	settings file + environment file	address and key in different files	Requires transactional writes to avoid partial success

An easily overlooked detail: some agents validate whether the model name is in a known list. The gateway must either maintain a local model list file for the agent to read, or pass any alias through as a valid model name; otherwise the switch will be blocked by the allowlist.

5. Illustrative Implementation: A Minimal Translation Kernel

The following simplified pseudocode illustrates the translation skeleton for “message block protocol inbound → chat completion protocol upstream” (type definitions and error handling omitted):

@app.post("/v1/messages")
async def handle_messages(req: dict):
    # 1. Model routing: agent-declared alias -> actual upstream model and credential
    model, credential = router.resolve(req["model"])
    # 2. Normalize into the intermediate representation (IR)
    ir = to_ir(system=req.get("system"),
               messages=req["messages"],   # includes text blocks / tool call blocks / tool result blocks
               tools=req.get("tools"))
    # 3. Generate the upstream protocol request from the IR
    payload = ir_to_chat(ir, model=model, stream=True)
    # 4. Streaming translation: upstream text deltas -> the event skeleton the client expects
    async def events():
        yield sse("message_start", meta())
        opened = False
        tool_buf = {}                      # aggregate tool argument fragments by call index
        async for chunk in call_upstream(payload, credential):
            delta = chunk["choices"][0]["delta"]
            if delta.get("content"):
                if not opened:
                    yield sse("content_block_start", type="text")
                    opened = True
                yield sse("content_block_delta", text=delta["content"])
            for tc in delta.get("tool_calls") or []:
                tool_buf.setdefault(tc["index"], []).append(
                    tc["function"]["arguments"])
        # Illustrative: emit complete tool-call blocks after aggregation is complete
        # (a production implementation should emit them as soon as streaming completes)
        for idx, parts in tool_buf.items():
            yield sse("content_block_start", type="tool_use")
            yield sse("content_block_delta", arguments="".join(parts))
        yield sse("message_stop")
    return sse_response(events())

Key points: steps 1–3 reflect the “normalize then dialectize” layering; step 4 reflects event skeleton synthesis and tool-call aggregation. A production implementation also needs usage passthrough, error events, cancellation propagation, and timeout handling.

The effect of targeted config editing looks like this (only the necessary lines change):

  // An agent's settings.json — all other keys, order, and indentation preserved
  {
    "env": {
-     "LLM_BASE_URL": "https://original-vendor-endpoint",
+     "LLM_BASE_URL": "http://127.0.0.1:8000",
+     "LLM_AUTH_TOKEN": "local-key"      // the real key lives only in the local gateway
    },
+   "model": "agent/main-model"           // gateway alias; the routing layer decides the actual model
  }

# An agent's config.toml — targeted addition of a gateway provider table
[model_providers.local_gateway]
name = "local-gateway"
base_url = "http://127.0.0.1:8000/v1"
protocol = "chat"          # this endpoint speaks the chat completion style
model = "agent/main-model"

6. Engineering and Security Checklist

• Listen on loopback only: bind to 127.0.0.1 by default to prevent credential capabilities from being accidentally exposed on the LAN; if sharing is truly required, add authentication and place it on a controlled network.
• Key management: inject real credentials via a local key store or environment variables; never put them in code, in config files that get synced, or in logs.
• Log redaction: record only model names, token usage, latency, and error codes; do not store full conversations, balancing troubleshooting with compliance.
• Timeouts, retries, and circuit breaking: when the upstream fluctuates, return a semantically correct error event to the agent rather than a truncated stream.
• Version compatibility: interface protocols keep evolving; parse leniently, generate with explicit versions, and add protocol detection when necessary.
• Regression testing: record and replay real traffic, covering edge cases such as “long conversations + multi-turn tool calls + streaming interruptions + parallel tools.”

7. Conclusion

A local gateway removes a constraint that should never have existed—protocol binding—from model selection: agents are chosen for convenience, and models are chosen by task and cost. What determines whether a solution is “workable” or “good” are two pillars: semantic-level protocol conversion (lossless round-trips of streaming, tool calls, and thinking processes) and minimal, atomic configuration editing (comment-preserving, order-preserving, rollback-capable). With these two points mastered, you have a clear technical yardstick whether you build a lightweight gateway yourself or evaluate comparable general-purpose solutions; for enterprises, it also provides a practical lever for unified key management and unified usage auditing.