Building an LLM Gateway: Automatic Model Routing for Coding Assistants Based on Request Content

1. Why an LLM Gateway is Needed

1.1 Common Pain Points in a Multi-Provider Environment

As the large language model (LLM) ecosystem rapidly evolves, developers frequently use APIs from multiple model providers alongside various AI-powered coding tools. This introduces several typical challenges:

Inconsistent Protocol Styles: The industry predominantly uses two API styles: the OpenAI-style Chat Completions protocol and the Anthropic-style Messages protocol. Different tools and providers support these to varying degrees. Some tools only support one style; some providers offer both, while others offer only one. Consequently, combining a specific tool with a specific provider often results in protocol incompatibilities, requiring additional adaptation work.

High Switching Costs: Most intelligent coding tools hardcode model configurations in their settings files or UI. Switching a model typically requires modifying configurations, restarting processes, or even re-authenticating. For developers looking to flexibly use different models for different tasks, the accumulated friction is significant.

Capability and Requirement Mismatches: Certain models excel at code generation but do not support image inputs, while multimodal models might not be the optimal choice for pure code tasks. When tools themselves do not support dynamic model selection based on content, users are forced to choose between “versatile but mediocre” and “specialized but limited.”

1.2 The Core Concept of a Gateway

The solution to these problems is introducing an intermediate layer—an LLM Gateway. Positioned between coding tools and model providers, it exposes a unified interface compatible with mainstream calling styles. Internally, it dynamically forwards requests to different providers and models based on configurable routing rules.

This approach ensures:

• The tool side simply points to the gateway address, unaware of backend model changes.
• The gateway handles protocol style conversion and adaptation, smoothing out provider differences.
• Routing rules are user-defined; changing models only requires adding or removing rules, which take effect instantly without restarting the tool.

Model selection is thus taken out of the tool’s hands and returned to the user.

2. Routing Rule Design and Typical Use Cases

2.1 Matching Based on Request Content

The essence of routing rules is “conditional distribution based on request content.” Before forwarding, the gateway can parse the request body and extract key features for matching. Common matching dimensions include:

Presence of Image Input: If the request’s messages contain an image type content block, it can be identified as a multimodal request. A rule can be configured to route default requests to a cost-effective, strong text coding model, but automatically switch to a multimodal model with vision capabilities once an image is detected. This process is completely transparent to the user; the moment an image is pasted, the request is sent to the correct model.

{
  "name": "multimodal-fallback",
  "match": {
    "has_image_content": true
  },
  "target": {
    "provider": "provider-a",
    "model": "vision-model-x"
  }
}

Text Matching Based on Prompts: Many coding tools concatenate project paths, system prompts, and other information into the request prompt. Leveraging this, keyword or regex matching rules can be written: requests containing a company project path prefix use the company’s API token, while requests matching personal project characteristics use a personal token. This achieves usage management separating work and personal projects without modifying any tool configurations.

Matching Based on Model Name or Headers: Some tools allow specifying a model name. The gateway can also use the model name as a routing key, mapping requests to the actual backend model to achieve “alias mapping.”

2.2 Rule Management and Lifecycle

Routing rules should support one-click enable, disable, and sorting within a management interface. The priority order of the rules determines the matching result—matching top-to-bottom, forwarding upon the first hit. This design offers excellent flexibility: for instance, to temporarily switch to another provider’s model, simply disable or adjust the corresponding rule without altering any tool configuration or restarting the process. Rule changes take effect immediately, making them ideal for A/B comparison testing between different models.

3. Engineering Implementation Highlights

3.1 Protocol Conversion Layer

The gateway must perform bidirectional protocol conversion for both requests and responses. Core tasks include:

• Request body structure mapping: Different style message formats have varying field names and content block structures, requiring field-by-field conversion.
• Streaming response handling: The Server-Sent Events (SSE) formats differ between the two styles. Upstream streaming events must be converted into the format expected by the tool side while maintaining real-time streaming.
• Error code and cancellation reason semantic alignment: This prevents exceptions caused by parsing failures on the tool side.

3.2 Technology Stack Selection

This type of gateway is a typical I/O-intensive application. Its core work is request forwarding and stream proxying, with minimal CPU computation. Choosing a runtime that starts quickly, has few dependencies, and is easy to package as a single-file distribution (such as one built on an emerging high-performance JavaScript runtime) is a sound engineering decision. The single-file executable distribution allows users to download and run it immediately, significantly lowering the deployment barrier.

3.3 Observability

A practical gateway should feature a dashboard displaying key metrics in real time:

• Current concurrent request count
• Output token rate (tokens/s)
• Hit count and forwarding distribution for each rule
• Request latency and error rate

These metrics not only help troubleshoot issues but also intuitively verify whether routing rules work as expected.

4. Trade-offs to Consider

No intermediate layer is free. The primary trade-off is the degradation of cache hit rates. Model providers typically use request prefixes for KV caching to reduce costs and accelerate responses. The gateway’s presence might alter certain request characteristics (such as injected fields or minor differences from protocol conversion), causing previously hit prefix caches to become invalid. In practice, this impact is usually manageable—the hit rate only drops slightly, and the flexibility benefits gained often far outweigh this cost. However, if a business is highly sensitive to caching costs, rule designs should strive to maintain the stability of request prefixes.

5. Conclusion

The value of an LLM Gateway lies in liberating model selection from a tool’s hardcoded configurations. Through declarative routing rules, it enables dynamic content-based distribution—such as routing images to multimodal models and code to strong coding models—while smoothing out protocol differences across providers. For developers using multiple model providers who want maximum model freedom within their coding tools, this is an architectural improvement with low investment and obvious returns. Its core concept—proxy layer + configurable routing + protocol adaptation—can naturally extend to enterprise scenarios for unified model access and governance.