Design and Implementation of a Local Proxy Gateway for Unified Multi-Model Access

    This article explores the practice of designing a local proxy gateway to unify access to multiple Large Language Models (LLMs). It covers the core architecture, protocol conversion, model routing strategies, account quota scheduling, and security boundaries. By introducing a local proxy layer, developers can seamlessly switch between different models without modifying client tools, optimizing for cost, context length, and compliance while maintaining a consistent client workflow.

    I. Background: Practical Constraints of Single-Model Access

    Developers typically form a stable workflow in their daily coding: reading project structures, modifying files, running commands, and checking results. The value of this workflow depends heavily on how developers interact with AI coding assistants, rather than just the capability of a specific model.

    However, different tasks have distinct requirements for models:

    • Understanding specific language requirements and generating technical documentation may be better suited for certain models.
    • Analyzing codebases with long contexts requires a larger context window.
    • Batch, repetitive micro-tasks are more sensitive to cost.
    • Tasks involving sensitive code may require the model to be deployed in an intranet environment.

    If switching models means changing tools, modifying configurations, and re-adapting every time, the efficiency loss for the team will accumulate. A more reasonable approach is to introduce a unified access layer between the client and the model services. This way, the client remains unchanged, and models can be switched according to the task.

    II. Core Concept: The Local Proxy Layer

    The local proxy layer sits between the client and the upstream model services. The client always sends requests to the local address. The proxy is responsible for converting the request into the protocol required by the upstream service, and then converting the response back into a format the client can parse.

    This design provides three main benefits:

    1. The client requires no modifications, preserving existing operational habits and automation scripts.
    2. Upstream services can be replaced or coexist, as switching only occurs within the proxy configuration.
    3. API keys, routing rules, and access logs are centrally managed, facilitating auditing and troubleshooting.

    It is important to note that the proxy layer solves access and scheduling problems. The inference capabilities and the reliability of tool calls still depend on the upstream model itself. The proxy layer cannot enable a less capable model to perform tasks beyond its scope; this boundary should be clearly defined during design.

    III. Key Aspects of Protocol Conversion

    The differences in interface formats across services are primarily reflected in the following areas:

    • Request Structure: How messages are organized, the placement of system instructions, and the field names for tool definitions.
    • Streaming Output: How incremental content is chunked and how the end marker is represented.
    • Tool Calling: Function names, the JSON structure of parameters, and the format for returning call results.
    • Multimodal Content: The encoding method and referencing format for images.

    The proxy layer needs to perform bidirectional mapping. Taking tool calls as an example, the tool definitions sent by the client must be converted into the format accepted by the upstream service, and the call instructions returned by the upstream must be restored into a structure the client can parse. Three things must be ensured during the conversion process:

    • Parameter types are not lost; numbers, booleans, and nested objects remain intact.
    • The order of streaming events is not disrupted, otherwise, the client may assemble incorrect content.
    • Error codes are mapped to a range recognizable by the client, while the original upstream error information is retained for troubleshooting.

    Below is a simplified illustration of the request conversion logic, omitting the implementation of helper functions:

    1. function to_upstream_request(client_req, provider_type):
    2. messages = client_req.messages
    3. tools = client_req.tools
    4. if provider_type == "style_a":
    5. return {model: client_req.model, messages: messages, tools: tools}
    6. if provider_type == "style_b":
    7. return {
    8. model: client_req.model,
    9. system: extract_system(messages),
    10. messages: strip_system(messages),
    11. tools: map(convert_tool, tools),
    12. max_tokens: client_req.max_tokens or 1024
    13. }
    14. raise UnknownProviderError(provider_type)

    IV. Model Routing Strategies

    The routing layer determines which upstream service will handle a specific request. There are three common strategies:

    1. Static Task Mapping: A virtual model name is fixed to a specific upstream model. For example, “Chinese documentation” maps to one set of services, while “long text analysis” maps to another.
    2. Failover: When the primary service returns a rate limit or timeout error, candidate services are tried sequentially by priority.
    3. Weighted Round Robin: Requests are distributed to multiple services with similar capabilities based on weights, used to alleviate quota pressure.

    When designing routing, note the following:

    • Failover should only target retryable errors, such as rate limits and temporary unavailability. Parameter errors should not trigger a failover, as this would mask the real problem.
    • Capability differences between services can be significant, especially regarding tool calling support. The candidate list should include capability tags for each service to prevent silent feature loss after a switch.
    • Switching models mid-session may cause inconsistencies in response style or tool behavior. A more robust approach is session-level binding, where only new sessions participate in reallocation.

    A general configuration structure looks like this:

    1. virtual_models:
    2. coding-default:
    3. strategy: failover
    4. candidates:
    5. - provider: primary-service
    6. priority: 1
    7. - provider: backup-service
    8. priority: 2
    9. batch-economy:
    10. strategy: weighted_round_robin
    11. candidates:
    12. - provider: service-a
    13. weight: 3
    14. - provider: service-b
    15. weight: 1

    The core logic for failover is similarly straightforward, with the key being accurate error classification:

    1. function route_with_failover(req, candidates):
    2. last_error = null
    3. for c in sort_by_priority(candidates):
    4. try:
    5. return c.call(req)
    6. catch RateLimitError as e:
    7. last_error = e
    8. catch TransientError as e:
    9. last_error = e
    10. // Other errors (e.g., parameter errors) are thrown directly without switching
    11. raise last_error

    V. Account and Quota Scheduling

    If a team uses multiple accounts to support the same type of service, the proxy layer can handle quota visualization and scheduling:

    • Display the remaining quota and reset time for each account to help determine when to switch.
    • Select an available account for new sessions.
    • Keep existing sessions tied to their original account to avoid context inconsistency caused by mid-conversation switches.

    This principle aligns with the session binding mentioned in Section IV. The granularity of scheduling should be the session, not the individual request. If accounts are assigned at the request level, a single conversation might jump between multiple accounts, which not only complicates troubleshooting but can also skew quota statistics.

    VI. Security and Data Boundaries

    The access layer centrally holds API keys and comes into contact with code and prompt content; therefore, security requirements must be prioritized:

    • Key Management: Keys should be stored in environment variables or a dedicated key management service, not written into configuration files or committed to version control repositories.
    • Network Exposure: The local proxy should only listen on the loopback address by default. If it needs to provide external services, authentication and access control must be added.
    • Log Desensitization: By default, do not log complete code content, prompt text, or credentials. Only retain metadata such as request time, routing results, status codes, and duration.
    • Compliance Constraints: For code that cannot leave the intranet, routing rules should restrict traffic to internally deployed model services. A mandatory validation step should be added during configuration loading to prevent accidental routing to external services.

    VII. Implementation Steps

    1. Inventory Task Scenarios: List the team’s common tasks and annotate each task type’s requirements for language, context length, cost, and data sensitivity.
    2. Select Candidate Services: Prepare one or two primary services and a backup service for each task type, and record the interface capabilities supported by each service.
    3. Write Routing Configurations: Start with static mapping. Once stable, introduce failover and weighted strategies.
    4. Canary Validation: Use a fixed set of test tasks to compare the accuracy, duration, and cost performance of different services before deciding on the default route.
    5. Establish Monitoring: Continuously monitor error rates, switch counts, and quota usage across accounts, and adjust weights and priorities promptly.

    VIII. Implementation Checklist

    • The client requires no changes to complete the full workflow of reading, modifying, executing, and validating.
    • Every virtual model is configured with a clear primary and backup service.
    • Tool calling functionality has undergone regression testing on every candidate service.
    • No API keys appear in configuration files or logs.
    • The routing scope for sensitive tasks is strictly confined to compliant environments.
    • Failover is only triggered for retryable errors.
    • Session-level binding is active, ensuring existing sessions do not switch models or accounts mid-conversation.