Building a Universal Local LLM Gateway: Unifying Multiple AI API Protocols

I. Background: Why a Local Protocol Gateway is Needed

With the growing adoption of AI coding assistants, CLI development tools, and various automation agents, a typical engineering challenge has emerged: different tools often support only one specific Large Language Model (LLM) API protocol, while different model services offer varying interface styles. Common scenarios include:

• A coding assistant that only supports the Anthropic-style Messages API;
• An agent tool that only supports the OpenAI-style Responses API;
• Some clients that only support the Gemini-style generation interface;
• A target model we actually want to call that only provides an OpenAI Chat Completions compatible interface.

If switching models means changing tools, modifying configurations, adapting protocols, or even rewriting calling logic, development costs will skyrocket. A more rational approach is to introduce a local gateway to handle protocol translation, request forwarding, response normalization, and configuration updates.

This article abstracts this class of problems into a universal technical solution: building a local protocol gateway that enables different AI tools to access various model backends using their respective familiar API protocols.

II. Overall Architecture: Gateway + Configuration Manager

The entire solution can be divided into two core modules.

2.1 Local Gateway

The local gateway typically runs on the loopback address of the development machine, such as 127.0.0.1:3425. It does not rely on external services and does not require exposing API keys to the public internet. All client requests first reach the gateway, which then transforms them into the request format recognized by the target model service.

A functional gateway should possess at least the following capabilities:

1. Multi-Protocol Input
   Simultaneously provide common interface formats such as OpenAI Chat Completions, OpenAI Responses, Anthropic Messages, and Gemini.

2. Protocol Interconversion
   When a client makes a request using Protocol A, the gateway can convert it to Protocol B required by the backend model; conversely, it converts the backend response back to Protocol A.

3. Streaming Output Preservation
   For typewriter-style outputs, the gateway must not wait for the complete response before forwarding. It should convert and forward data as it is received. Otherwise, the user experience will noticeably degrade.

4. Tool Call Preservation
   Function calls, tool selection, and parameter generation must be fully preserved during protocol conversion. Since different protocols express tool calls differently, the gateway must explicitly handle these fields rather than simply passing them through.

5. Reasoning Process Preservation
   If the backend model returns a chain of thought, intermediate reasoning steps, or internal reasoning fields, the protocol mapping must consider whether these fields should be retained, how they are named, and in which layer of the structure they belong.

2.2 Configuration Manager

Simply running a gateway is not enough. Each client has its own configuration file structure, and manual modification is highly error-prone. Therefore, the configuration manager automatically modifies the corresponding client’s configuration after a user selects a model.

The goal of the configuration manager is not to “rewrite the entire configuration file” but to achieve minimal changes:

• Only modify keys related to connecting to the gateway;
• Preserve original comments, key order, and indentation style;
• Use atomic writes to prevent client startup failures caused by half-written configurations;
• Leave unsupported keys or unknown configurations as they are.

This design significantly reduces maintenance costs and allows users to roll back to the original configuration at any time.

III. Key Points of Protocol Conversion

Protocol conversion is the most complex part of the gateway. Although different protocols serve the same LLM generation task, they exhibit significant differences in field design, request structure, role hierarchy, and response format.

3.1 Request Structure Mapping

Taking a standard conversation as an example, a client might send a request in the Anthropic Messages style:

{
  "model": "example-model",
  "max_tokens": 1024,
  "system": "You are a helpful assistant.",
  "messages": [
    {
      "role": "user",
      "content": "Explain what a local gateway is"
    }
  ]
}

If the backend model only accepts the OpenAI Chat Completions style, the gateway needs to convert it into a format like this:

{
  "model": "example-model",
  "max_tokens": 1024,
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "Explain what a local gateway is"
    }
  ]
}

This example only shows a simple structural change. In real-world scenarios involving multi-turn messages, image content, tool definitions, and response format requirements, the field mapping relationships will be much more complex.

3.2 Response Normalization

Request conversion is just the first step; a greater challenge lies in converting the backend response into a format the client can recognize.

For instance, some protocols use stop_reason to indicate the end of generation, while others use finish_reason. Some protocols place tool calls inside the content structure, while others place them at the top level of the message. The gateway must process these field by field:

• Text content;
• Tool calls;
• Stop reasons;
• Usage statistics;
• Streaming chunks;
• Error messages.

If the response format mapping is incomplete, clients might experience issues like “successful request but no output,” “missing tool calls,” or “interrupted streaming output.”

3.3 Stream Processing

Streaming output requires the gateway to perform incremental conversions. The streaming chunk structures often differ across protocols.

In some protocols, each chunk contains only an incremental text fragment; in others, the first chunk might return role information, with subsequent chunks returning content. The gateway needs to understand these state machine differences and generate the correct chunk sequence for different clients.

During implementation, an “Intermediate Representation (IR) layer” approach can be adopted:

1. The client request enters the gateway;
2. The gateway converts the client protocol into a unified internal representation;
3. The internal representation is converted into the backend model protocol;
4. The streaming chunks returned by the backend model are converted into internal events;
5. The internal events are then converted into streaming chunks of the client protocol.

The benefit of this approach is that adding a new protocol only requires implementing the mapping between it and the internal representation. The complexity does not grow exponentially with the number of protocols.

IV. Client Configuration Write Strategy

Different clients use different configuration file formats, field naming, and authentication methods. The configuration manager needs to handle these targetedly.

4.1 Configuration File Types

Common configuration formats include JSON, TOML, and environment variable files. Each format handles comments, ordering, and indentation differently.

• JSON configurations generally do not support comments, so modifications should aim to keep key order stable;
• TOML configurations support comments and table structures, making them suitable for adding new provider blocks;
• Environment variable files are typically simple KEY=VALUE pairs, so care must be taken not to overwrite unrelated variables.

A robust approach is to first parse the original file, locate the target position, execute minimal modifications, and finally serialize it back to the original format.

4.2 Minimal Change Example

Assume a client uses a JSON configuration that already contains several environment variables. After a user selects a model, the configuration manager only wants to add or modify three fields:

{
  "env": {
    "EXAMPLE_BASE_URL": "http://127.0.0.1:3425",
    "EXAMPLE_AUTH_TOKEN": "local-token",
    "EXAMPLE_MODEL": "selected-model"
  }
}

Ideally, other fields, comments, and indentation remain unchanged, and even a Git diff would only show the addition or modification of these few keys.

For clients using TOML configuration, a new provider table can be added:

[model_providers.local_gateway]
name = "Local Gateway"
base_url = "http://127.0.0.1:3425"
api_key = "local-token"

Meanwhile, the model list can be written to an independent model configuration file to prevent the main configuration from becoming too complex.

4.3 Atomic Writes

Atomic writes are typically achieved using a “temporary file + rename” method:

1. Write the modified content to a temporary file in the same directory;
2. Ensure the write is complete and flushed to disk;
3. Use an atomic rename operation to replace the original file.

This prevents configuration file corruption caused by process interruptions, abnormal exits, or disk issues during the write process.

V. Practical Usage Workflow

For developers, a typical usage workflow can be summarized in the following steps:

1. Start the local gateway and confirm the listening address and port;
2. Select the desired backend model;
3. The configuration manager automatically updates the current client’s configuration file;
4. Restart the client or reload the configuration;
5. Initiate conversations, generation, or tool calls normally;
6. To switch models, simply repeat steps 2 through 4.

The core advantage of this workflow is that the client itself requires no modification. It continues to use its familiar protocol, configuration format, and calling method, merely directing its requests to the local gateway.

VI. Security and Debugging Recommendations

6.1 Key Management

The local gateway typically needs to store the API key for the backend model service. It is recommended to separate this from the client configuration, for example, by using local environment variables, dedicated credential files, or the system keychain. Avoid writing all keys in plaintext across multiple client configurations.

6.2 Access Control

Although the gateway listens on a loopback address, other processes on the same machine can still access that port. A local token can be injected into the gateway, which the client must include in its requests. This way, even if another local process accidentally accesses the port, it cannot directly utilize the backend resources.

6.3 Debugging Methods

When debugging protocol conversion issues, you can start from the following aspects:

• Check the raw request actually sent by the client;
• Check the backend request transformed by the gateway;
• Check the complete response returned by the backend;
• Check the final response returned to the client by the gateway.

It is recommended that the gateway provides detailed logs, but avoid outputting complete keys and sensitive content in the logs.

For streaming issues, you can record the timestamp, type, fields, and content length of each chunk to help pinpoint whether it is a delay, packet loss, or field mapping error.

VII. Scalability and Practical Value

Once the local gateway runs stably, it becomes a lightweight yet critical piece of infrastructure in the development environment. Its direct values include:

• Developers can use the same model backend across different AI coding tools;
• No need to adjust client code when switching models;
• Configuration changes are centrally managed, making rollback and troubleshooting easier;
• When integrating a new client, simply confirm which protocol it supports and bridge it via the gateway.

From a broader perspective, this “local protocol gateway + configuration manager” approach is not limited to the coding assistant domain. Any scenario involving multi-protocol clients and multi-model backends can adopt a similar architecture to reduce integration costs.

Finally, it is important to emphasize that the value of the gateway solution lies not in replacing any specific protocol, but in providing a stable layer of abstraction. It converges protocol differences, configuration differences, and model-switching complexity to a single local layer, allowing upper-layer clients and lower-layer model services to evolve independently.