画板 1

    A ~50-Line Model Switching Layer for Command-Line Coding Agents

    3 hr ago0 views

    This article explains how to add a lightweight model switching layer to command-line coding agents. By inserting an HTTP protocol adapter between the agent and model backend, you can expose chat-completions-style and event-stream-style endpoints, map request fields, convert tool definitions, translate streaming events, and switch model backends through environment variables. The example uses a .NET 10 file-based C# app, Microsoft.Extensions.AI, and ASP.NET Core minimal APIs. It also covers configuration, verification, troubleshooting, protocol limitations, security boundaries, and practical engineering improvements.

    1. Why command-line coding agents get locked to one model

    Writing code in the terminal is no longer unusual. Command-line coding agents are becoming part of everyday engineering work. They use the terminal as the interaction interface and usually provide three core capabilities:

    • Shell tools: run builds, tests, scripts, and other commands in a controlled environment.
    • Patch application: modify code files safely using a unified patch format, making review and rollback easier.
    • Plan tracking: split complex tasks into step lists and update completion status item by item.

    Compared with editor plugins that only provide code completion, these agents are better suited to end-to-end tasks, such as locating the root cause of an error and fixing it, refactoring across files, or adding unit tests in bulk.

    Stronger capabilities impose stricter requirements on models: stable structured tool calls, strict adherence to patch formats, and consistency in long contexts. As a result, many agents are deeply bound to the native interface of a particular model provider.

    In an event-style interface, for example, after the agent establishes a connection, the server continuously sends incremental text, reasoning summaries, tool-call arguments, and completion signals over SSE (Server-Sent Events). In contrast, many open-source inference services, local runners, and model aggregation platforms only expose chat-completion-style interfaces: a request carries a set of messages, and the response is returned at once or streamed in chunks.

    This mismatch creates three pain points:

    1. Cost pressure: expensive subscriptions can be heavy for high-frequency users and individual developers.
    2. Limited experience: it is hard to choose the best model separately for code understanding, tool calling, and long-context handling.
    3. Data compliance: when private enterprise code is involved, inference may need to run locally or inside a private network.

    Common workarounds all have drawbacks:

    • Directly modifying the agent client: deep source coupling, frequent upgrades, and hard-to-maintain patches.
    • Switching to another tool: gives up the interaction and patch workflow users are already used to.
    • Building an entire service: authentication, stream forwarding, and multi-protocol adaptation are all required, which is substantial work.

    A lighter and more general approach is to insert a protocol adapter layer between the agent and the model backend, wrapping any model backend into the protocol shape the agent needs.

    2. Strategy: turn model switching into configuration

    2.1 Core insight: change the agent into implementing a translator

    Most command-line coding agents allow a configuration block to specify the service address, model name, and API key. This means that if you provide a protocol-compatible HTTP service, the agent will treat it as a native backend. The problem therefore changes from “modifying the agent” to “implementing a translator.”

    The main differences to translate fall into two interface styles:

    Aspect Chat-completions style Agent event-stream style
    Message structure messages array separated by role input array, with historical turns represented as item collections
    System instruction separate system message instructions field
    Tool result message with role tool function call output item
    Streaming output incremental chunks named event stream: text deltas, function argument deltas, completion events, etc.
    End semantics finish_reason field separate completion event
    Usage statistics prompt, completion, and total token counts input and output token counts

    2.2 Selection criteria: provider neutrality first, then runnable versions

    The first selection criterion is provider neutrality: core code should not depend on any specific model provider, and switching backends should only require configuration changes.

    Mainstream generic model client abstractions reduce differences between providers into a unified interface. Typical capabilities include:

    • representing multi-turn conversations with a unified message model;
    • expressing parameters such as temperature, maximum output tokens, and tool definitions with a unified options object;
    • providing both one-shot and streaming calls;
    • returning usage statistics uniformly, with basic error and cancellation propagation.

    With this abstraction, the adapter only needs to do two things: parse the incoming protocol and map fields, then re-encode the outgoing protocol. This is why about 50 lines of core code can be enough to get it running.

    The example in this article is based on the file-based app feature in .NET 10, which can run a single .cs file directly without a project file. The public SDKs are also provided as bundles. Keep the runnable boundaries in mind:

    • File-based apps depend on the .NET 10 SDK. For versions before .NET 10, use a standard console project.
    • The exact versions of extension packages depend on the compatible versions available on NuGet when you install them. If API names change, follow the documentation for the corresponding version.
    • The example uses ASP.NET Core minimal APIs. It does not switch to another host framework.

    2.3 Architecture and data flow

    1. ┌──────────────────────┐
    2. │ Command-line coding agent │
    3. └──────────┬───────────┘
    4.            │ POST /v1/responses (event stream)
    5.            ▼
    6. ┌──────────────────────┐   ┌──────────────────────────┐
    7. │ POST /v1/responses   │   │ POST /v1/chat/completions │
    8. │ (adapter ingress)    │   │ (adapter ingress)         │
    9. └──────────┬───────────┘   └────────────┬─────────────┘
    10.            └────────────┬────────────────┘
    11.                         ▼
    12.              Unified model client abstraction
    13.                         ▼
    14.       ┌─────────────────┼─────────────────┐
    15.       ▼                 ▼                 ▼
    16.  Cloud aggregation gateway  Cloud model service  Local inference service

    The design has three layers:

    • Ingress protocol layer: parses requests from agents or general tools.
    • Abstract client layer: hides backend differences and handles authentication, request construction, and stream reading.
    • Backend model layer: can be replaced freely with an aggregation gateway, a cloud model service, or a local inference service.

    Many mainstream cloud platforms and inference services provide OpenAI-compatible unified inference interfaces and can serve as a standard type of backend for the adapter. Refer to each product’s official documentation for details.

    3. Core implementation: build the ~50-line skeleton

    3.1 Runtime form and minimal skeleton

    The following single file can be saved as proxy.cs and run. It contains host creation, port binding, client construction, and two endpoints. Later sections expand the key parts.

    1. #:package Microsoft.Extensions.AI
    2. #:package Microsoft.Extensions.AI.OpenAI
    3. #:package Microsoft.Extensions.AI.Ollama
    5. // 1. Configuration: read everything from environment variables; secrets are not embedded in source or logged
    6. string? Env(string key, string? fallback = null) =>
    7.     Environment.GetEnvironmentVariable(key) is { Length: > 0 } v ? v : fallback;
    8. var baseUrl   = Env("MODEL_BASE_URL");
    9. var apiKey    = Env("MODEL_API_KEY");
    10. var modelName = Env("MODEL_NAME", "qwen2.5-coder:14b")!;
    11. var port      = Env("PORT", "18080")!;
    13. // 2. Client: use the Ollama client for local addresses; use the OpenAI-compatible client for the rest
    14. IChatClient ai = string.IsNullOrWhiteSpace(baseUrl)
    15.                  || baseUrl.Contains("11434", StringComparison.Ordinal)
    16.     ? new OllamaChatClient(new Uri(baseUrl ?? "http://localhost:11434"), modelName)
    17.     : new OpenAIClient(
    18.           new Uri(baseUrl.EndsWith("/v1") ? baseUrl : baseUrl.TrimEnd('/') + "/v1"),
    19.           string.IsNullOrWhiteSpace(apiKey)
    20.               ? throw new InvalidOperationException("MODEL_API_KEY must be set when connecting to a cloud backend")
    21.               : apiKey)
    22.       .AsIChatClient(modelName);
    24. // 3. Host: bind to loopback by default; PORT can adjust it
    25. var builder = WebApplication.CreateBuilder();
    26. builder.WebHost.UseUrls($"http://127.0.0.1:{port}");
    27. var app = builder.Build();
    29. app.MapPost("/v1/chat/completions", async (HttpRequest req, CancellationToken ct) =>
    30. {
    31.     using var doc = await JsonDocument.ParseAsync(req.Body, cancellationToken: ct);
    32.     var (chat, opts) = BuildChat(doc.RootElement, modelName);
    33.     var res = req.HttpContext.Response;
    34.     res.Headers.ContentType = "application/json";
    36.     if (doc.RootElement.TryGetProperty("stream", out var s) && s.GetBoolean())
    37.     {
    38.         res.Headers.ContentType = "text/event-stream";
    39.         res.Headers.CacheControl = "no-cache";
    40.         var id = "chatcmpl-" + Guid.NewGuid().ToString("N")[..24];
    41.         await foreach (var u in ai.GetStreamingResponseAsync(chat, opts, ct))
    42.             await WriteSse(res, JsonSerializer.Serialize(new
    43.             {
    44.                 id,
    45.                 choices = new[] { new { delta = new { content = u.Text } } }
    46.             }), ct);
    47.         await WriteSse(res, "[DONE]", ct);
    48.         return Results.Empty;
    49.     }
    51.     var r = await ai.GetResponseAsync(chat, opts, ct);
    52.     return Results.Json(new
    53.     {
    54.         id = "chatcmpl-" + Guid.NewGuid().ToString("N")[..24],
    55.         choices = new[]
    56.         {
    57.             new { finish_reason = "stop", message = new { role = "assistant", content = r.Text } }
    58.         },
    59.         usage = ToUsage(r.Usage)
    60.     });
    61. });
    63. app.MapPost("/v1/responses", async (ResponsesRequest req, CancellationToken ct) =>
    64. {
    65.     var (chat, opts) = req.ToChat(modelName);
    66.     var res = req.HttpContext.Response;
    67.     res.Headers.ContentType = "text/event-stream";
    68.     res.Headers.CacheControl = "no-cache";
    69.     await foreach (var evt in StreamEvents(ai, chat, opts, ct))
    70.         await res.WriteAsync(evt, ct);
    71.     return Results.Empty;
    72. });
    74. app.Run();
    76. // 4. Helper methods
    77. static (List<ChatMessage>, ChatOptions) BuildChat(JsonElement root, string model)
    78. {
    79.     var chat = new List<ChatMessage>();
    80.     foreach (var m in root.GetProperty("messages").EnumerateArray())
    81.     {
    82.         var role = m.GetProperty("role").GetString() ?? "user";
    83.         var text = m.TryGetProperty("content", out var c)
    84.             ? c.ValueKind == JsonValueKind.String ? c.GetString() ?? "" : c.GetRawText()
    85.             : "";
    86.         if (role == "tool")
    87.             chat.Add(new ChatMessage(ChatRole.Tool, text));
    88.         else
    89.             chat.Add(new ChatMessage(new ChatRole(role), text));
    90.     }
    91.     var opts = new ChatOptions { ModelId = model };
    92.     if (root.TryGetProperty("temperature", out var t) && t.TryGetDouble(out var tv)) opts.Temperature = tv;
    93.     if (root.TryGetProperty("max_tokens", out var m2) && m2.TryGetInt32(out var mv)) opts.MaxOutputTokens = mv;
    94.     if (root.TryGetProperty("tools", out var tools) && tools.ValueKind == JsonValueKind.Array)
    95.         opts.Tools = tools.EnumerateArray().Select(ToTool).ToList();
    96.     return (chat, opts);
    97. }
    99. static AITool ToTool(JsonElement t)
    100. {
    101.     var f = t.GetProperty("function");
    102.     var name = f.GetProperty("name").GetString() ?? "tool";
    103.     if (!f.TryGetProperty("parameters", out var schema) || schema.ValueKind != JsonValueKind.Object)
    104.         schema = JsonDocument.Parse("{\"type\":\"object\"}").RootElement;
    105.     // The overload name depends on the package version you use
    106.     return AIFunctionFactory.Create(name, f.TryGetProperty("description", out var d) ? d.GetString() : null, schema);
    107. }
    109. static async Task WriteSse(HttpResponse res, string data, CancellationToken ct)
    110. {
    111.     await res.WriteAsync("data: " + data + "\n\n", ct);
    112.     await res.Body.FlushAsync(ct);
    113. }
    115. static object ToUsage(UsageDetails? u) => new
    116. {
    117.     prompt_tokens = u?.InputTokenCount ?? 0,
    118.     completion_tokens = u?.OutputTokenCount ?? 0,
    119.     total_tokens = u?.TotalTokenCount ?? 0
    120. };

    3.2 Client construction: only one place is backend-specific

    The only backend-specific part of the script is the construction of the unified client. It reads environment variables into string variables first. Missing values and missing keys fail at startup, and downstream methods such as BuildChat do not read environment variables directly.

    Environment variable Meaning Example
    MODEL_BASE_URL Backend service address; leave empty to use the default Ollama address http://localhost:11434 or https://<gateway-endpoint>/v1
    MODEL_API_KEY Access key; required when connecting to a cloud backend <your-key>
    MODEL_NAME Model identifier qwen2.5-coder:14b
    PORT Adapter listening port; bound to 127.0.0.1 by default 18080 
    1. // Equivalent to section 2 in the skeleton
    2. IChatClient ai = string.IsNullOrWhiteSpace(baseUrl)
    3.                  || baseUrl.Contains("11434", StringComparison.Ordinal)
    4.     ? new OllamaChatClient(new Uri(baseUrl ?? "http://localhost:11434"), modelName)
    5.     : new OpenAIClient(
    6.           new Uri(baseUrl.EndsWith("/v1") ? baseUrl : baseUrl.TrimEnd('/') + "/v1"),
    7.           string.IsNullOrWhiteSpace(apiKey)
    8.               ? throw new InvalidOperationException("MODEL_API_KEY must be set when connecting to a cloud backend")
    9.               : apiKey)
    10.       .AsIChatClient(modelName);

    The three branches cover local inference services, OpenAI-compatible cloud aggregation gateways, and cases where a compatible endpoint does not include /v1. When the backend is upgraded or replaced, you usually only need to add one branch. AsIChatClient comes from the Microsoft.Extensions.AI namespace.

    3.3 Endpoint one: chat-completions style /v1/chat/completions

    This endpoint serves other editing tools, scripts, and SDKs. It needs to do three things:

    • map messages and parameter dictionaries into a unified message list and options object;
    • convert tool definitions one by one;
    • in streaming mode, wrap each incremental chunk as an SSE record and finally send [DONE].
    1. app.MapPost("/v1/chat/completions", async (HttpRequest req, CancellationToken ct) =>
    2. {
    3.     using var doc = await JsonDocument.ParseAsync(req.Body, cancellationToken: ct);
    4.     var (chat, opts) = BuildChat(doc.RootElement, modelName);
    5.     var res = req.HttpContext.Response;
    6.     res.Headers.ContentType = "application/json";
    8.     if (doc.RootElement.TryGetProperty("stream", out var s) && s.GetBoolean())
    9.     {
    10.         res.Headers.ContentType = "text/event-stream";
    11.         res.Headers.CacheControl = "no-cache";
    12.         var id = "chatcmpl-" + Guid.NewGuid().ToString("N")[..24];
    13.         await foreach (var u in ai.GetStreamingResponseAsync(chat, opts, ct))
    14.             await WriteSse(res, JsonSerializer.Serialize(new
    15.             {
    16.                 id,
    17.                 choices = new[] { new { delta = new { content = u.Text } } }
    18.             }), ct);
    19.         await WriteSse(res, "[DONE]", ct);
    20.         return Results.Empty;
    21.     }
    23.     var r = await ai.GetResponseAsync(chat, opts, ct);
    24.     return Results.Json(new
    25.     {
    26.         id = "chatcmpl-" + Guid.NewGuid().ToString("N")[..24],
    27.         choices = new[]
    28.         {
    29.             new { finish_reason = "stop", message = new { role = "assistant", content = r.Text } }
    30.         },
    31.         usage = ToUsage(r.Usage)
    32.     });
    33. });

    BuildChat, ToTool, WriteSse, and ToUsage are helper methods. They are responsible for mapping messages and options, converting tool definitions, writing SSE responses, and converting usage statistics. They have compilable implementations in the skeleton in Section 3.1. WriteSse flushes the response stream immediately after each write to prevent buffering of incremental content.

    3.4 Endpoint two: agent event stream /v1/responses

    This is the most critical part of the adapter. The agent relies on complete event semantics to determine generation status, so the mapping must be rigorous:

    • instructions: convert to a system message and place it at the beginning of the message list.
    • input items: convert by type into user messages, assistant messages, function calls, and function call outputs.
    • streaming events: map text deltas to text delta events and function argument deltas to function call argument delta events.
    • termination event: regardless of normal completion, cancellation, or exception, a completion or failure event must be sent.

    First, define the request structure and mapping method. Fields are explicitly declared with [JsonPropertyName], and property names use normal PascalCase to avoid binding failures caused by style confusion:

    1. using System.Text.Json.Serialization;
    3. sealed class ResponsesRequest
    4. {
    5.     [JsonPropertyName("instructions")] public string? Instructions { get; set; }
    6.     [JsonPropertyName("input")] public List<Item>? Input { get; set; }
    7.     [JsonPropertyName("tools")] public List<ToolDef>? Tools { get; set; }
    8.     [JsonPropertyName("temperature")] public double? Temperature { get; set; }
    9.     [JsonPropertyName("max_output_tokens")] public int? MaxOutputTokens { get; set; }
    11.     public (List<ChatMessage>, ChatOptions) ToChat(string model)
    12.     {
    13.         var chat = new List<ChatMessage>();
    14.         if (!string.IsNullOrWhiteSpace(Instructions))
    15.             chat.Add(new ChatMessage(ChatRole.System, Instructions!));
    16.         foreach (var it in Input ?? [])
    17.             switch (it.type)
    18.             {
    19.                 case "message":
    20.                     var text = string.Concat(it.content?
    21.                         .Where(p => p.type is "input_text" or "output_text")
    22.                         .Select(p => p.text) ?? []);
    23.                     chat.Add(new ChatMessage(new ChatRole(it.role ?? "user"), text));
    24.                     break;
    25.                 case "function_call":
    26.                     chat.Add(new ChatMessage(ChatRole.Assistant,
    27.                     [
    28.                         new FunctionCallContent(it.call_id!, it.name!,
    29.                             it.arguments is { Length: > 0 } a
    30.                                 ? JsonDocument.Parse(a).RootElement.Clone() : null)
    31.                     ]));
    32.                     break;
    33.                 case "function_call_output":
    34.                     chat.Add(new ChatMessage(ChatRole.Tool,
    35.                         [new FunctionResultContent(it.call_id!, it.output ?? "")]));
    36.                     break;
    37.             }
    38.         var opts = new ChatOptions { ModelId = model };
    39.         if (Tools is { Count: > 0 })
    40.             opts.Tools = Tools.Select(t => ToTool(t.function!)).ToList();
    41.         if (Temperature is { } t) opts.Temperature = t;
    42.         if (MaxOutputTokens is { } m) opts.MaxOutputTokens = m;
    43.         return (chat, opts);
    44.     }
    45. }
    47. sealed class Item
    48. {
    49.     public string? type { get; set; }
    50.     public string? role { get; set; }
    51.     public List<Part>? content { get; set; }
    52.     public string? call_id { get; set; }
    53.     public string? name { get; set; }
    54.     public string? arguments { get; set; }
    55.     public string? output { get; set; }
    56. }
    57. sealed class Part { public string? type { get; set; } public string? text { get; set; } }
    58. sealed class ToolDef { public string? type { get; set; } public FnDef? function { get; set; } }
    59. sealed class FnDef
    60. {
    61.     public string? name { get; set; }
    62.     public string? description { get; set; }
    63.     public JsonElement parameters { get; set; }
    64. }

    ToTool in the skeleton already accepts tool declarations in both chat-completions style and event-stream style, in the form {"type":"function","function":{...}}, so both endpoints can reuse it. Function calls and their output items must appear in pairs and keep the same call_id. The example does not validate this; missing items should be fixed during integration testing.

    Next is the event stream generator. C# does not support object spread syntax. For each event, business fields are first placed into a dictionary, and then type and sequence_number are added uniformly:

    1. static async IAsyncEnumerable<string> StreamEvents(
    2.     IChatClient ai, List<ChatMessage> chat, ChatOptions o,
    3.     [EnumeratorCancellation] CancellationToken ct = default)
    4. {
    5.     var seq = 0;
    6.     yield return Sse("response.created", seq++, new() { ["status"] = "in_progress" });
    8.     try
    9.     {
    10.         await foreach (var u in ai.GetStreamingResponseAsync(chat, o, ct))
    11.         {
    12.             if (!string.IsNullOrEmpty(u.Text))
    13.                 yield return Sse("response.output_text.delta", seq++,
    14.                     new() { ["delta"] = u.Text });
    15.             foreach (var c in u.Contents.OfType<FunctionCallContent>())
    16.                 yield return Sse("response.function_call_arguments.delta", seq++,
    17.                     new()
    18.                     {
    19.                         ["call_id"] = c.CallId,
    20.                         ["name"] = c.FunctionName,
    21.                         ["delta"] = c.Arguments?.GetRawText() ?? ""
    22.                     });
    23.         }
    24.         yield return Sse("response.completed", seq++, new() { ["status"] = "completed" });
    25.     }
    26.     catch (Exception ex) when (ex is not OutOfMemoryException)
    27.     {
    28.         // Cancellation is also handled here; no additional event is needed after the client disconnects
    29.         yield return Sse("response.failed", seq++, new()
    30.         {
    31.             ["status"] = "failed",
    32.             ["error"] = new { message = ex is OperationCanceledException ? "cancelled" : ex.Message }
    33.         });
    34.     }
    35. }
    37. static string Sse(string type, int seq, Dictionary<string, object?> payload)
    38. {
    39.     payload["type"] = type;
    40.     payload["sequence_number"] = seq;
    41.     return $"event: {type}\ndata: {JsonSerializer.Serialize(payload)}\n\n";
    42. }

    The route itself is short. Minimal APIs bind ResponsesRequest from the JSON request body, then the mapping is passed to the iterator and written back one record at a time:

    1. app.MapPost("/v1/responses", async (ResponsesRequest req, CancellationToken ct) =>
    2. {
    3.     var (chat, opts) = req.ToChat(modelName);
    4.     var res = req.HttpContext.Response;
    5.     res.Headers.ContentType = "text/event-stream";
    6.     res.Headers.CacheControl = "no-cache";
    7.     await foreach (var evt in StreamEvents(ai, chat, opts, ct))
    8.         await res.WriteAsync(evt, ct);
    9.     return Results.Empty;
    10. });

    Exceptions are converted into response.failed inside the generator because once an exception escapes the handler, the response headers have already been sent and the framework cannot change it to a JSON error code. Exceptions thrown inside the generator are thrown when the iterator pulls the next record, so this handling still works.

    3.5 Key mappings for the event-style protocol

    Event-style element Unified abstraction representation Location in the example implementation
    instructions System message ToChat, placed at the beginning of the message list
    message items in input User/assistant messages case "message" in ToChat
    function call items Function call content FunctionCallContent, carrying call_id, function name, and arguments
    function call output items Function call result content FunctionResultContent, associated with the call by call_id
    tool definitions Function-type tools (AIFunction) ToTool, constructed from JSON Schema
    text deltas Text in streaming responses response.output_text.delta event
    function argument deltas Function call content in streaming responses response.function_call_arguments.delta event
    completion status End of stream or one-shot response response.completed / response.failed events

    The current coverage is limited. The example event stream implements only five event types: created, text delta, argument delta, completed, and failed. This is enough to run ordinary conversations and common tool calls. If the agent depends on finer-grained state, the following events still need to be added:

    • call_id and function name should appear completely in the first argument delta; later deltas should only carry argument fragments.
    • tool argument end events, allowing the agent to determine that the arguments are complete;
    • structured error events, distinguishing authentication failures, model refusals, context overflow, and other cases;
    • reasoning summary events, used to display the thinking process.

    Special attention is needed for streaming function call arguments. The backend model may return complete arguments at once, or return them in multiple segments. Chat-completion-style wrappers concatenate the argument segments into complete JSON and parse them into a structured object. In contrast, agents expect the original argument deltas. A concatenated event can only be sent as a single delta. For complex tool calls, such as multiple file reads and writes, an agent may treat concatenated arguments as invalid fragments. Therefore, a low-level client that preserves raw streaming data is more suitable for this scenario. These differences explain why “selection neutrality” does not mean “identical behavior.” Each behavior needs to be verified item by item during integration testing.

    3.6 Where the ~50 lines come from

    A complete solution usually includes reading the request body, parsing instructions and input, converting tool definitions, stream encoding, and error handling. Compressing the core logic to about 50 lines relies on four points:

    • Abstraction reuse: message and option construction is handled by the abstraction layer, so no backend-specific branch is needed.
    • Logic merging: both ingress endpoints share the same tool conversion and SSE writing logic.
    • Runtime simplification: file-based apps remove boilerplate such as class declarations and entry methods.
    • Capability deferral: cross-cutting capabilities such as authentication, rate limiting, and multi-backend routing are not built in yet.

    Line count only reflects implementation compactness and is not a quality metric. The standard for evaluating the adapter is the completeness and stability of protocol semantics.

    4. From startup to switching: connect a command-line agent to a local model

    Step 1: configure environment variables.

    1. # Example 1: connect to a local inference service
    2. export MODEL_BASE_URL="http://localhost:11434"
    3. export MODEL_API_KEY=""
    4. export MODEL_NAME="qwen2.5-coder:14b"
    5. export PORT=18080
    7. # Example 2: connect to a cloud aggregation gateway
    8. export MODEL_BASE_URL="https://<aggregator-endpoint>/v1"
    9. export MODEL_API_KEY="<your-key>"
    10. export MODEL_NAME="<model-identifier>"

    Step 2: run the script. Save it as a single file and execute it. The adapter listens on http://127.0.0.1:18080 by default, and can be adjusted with PORT.

    Step 3: verify the chain with curl.

    1. # Chat-completions endpoint
    2. curl -sS http://127.0.0.1:18080/v1/chat/completions \
    3.   -H "Content-Type: application/json" \
    4.   -d '{"model":"x","messages":[{"role":"user","content":"Hello"}],"stream":false}'
    6. # Agent event-stream endpoint; observe whether event lines and sequence_number increase
    7. curl -sS -N http://127.0.0.1:18080/v1/responses \
    8.   -H "Content-Type: application/json" \
    9.   -d '{"model":"x","instructions":"You are a coding assistant","input":[{"type":"message","role":"user","content":[{"type":"input_text","text":"Hello"}]}]}'

    Troubleshooting signals: HTTP 401 usually means a backend key error or missing key; 404 means the path or agent base address is wrong; HTTP 200 but no data: line usually means the backend returned an empty stream or the first byte timed out; only response.created without response.completed means the backend stream was interrupted midway and has been captured as response.failed.

    Step 4: modify the agent configuration. Point the agent’s model service base address to the adapter. The model name can be arbitrary; the actual model is decided by the adapter based on environment variables.

    Step 5: verify agent capabilities. First ask the agent to “read the README and summarize the project purpose” to confirm the ordinary conversation path works. Then ask it to “fix the failing assertion in the specified test” to verify tool calling and patch application.

    Step 6: switch the model. Only environment variables need to be changed and the adapter restarted; the agent configuration and calling code remain unchanged.

    Model selection suggestions:

    • Context window: agents often carry a large amount of file content at once, so long-context models or versions with context compression support are preferred.
    • Tool calling: prefer models with mature function-calling support and stable output formatting.
    • Local models: due to GPU memory and compute limits, choose models with moderate parameter size, optimization for code corpora, and mature quantized versions.
    • Reasoning depth: complex defect localization can use models with extended reasoning support, while small daily changes can use faster general-purpose models.

    5. Engineering enhancements and common pitfalls

    5.1 Enhancement directions

    • Multi-backend routing: choose backends by task type or model capability, for example using a reasoning model for planning and a code model for patch generation.
    • Fallback and retry: switch to a backup model when the primary model is rate-limited or times out. Apply this only to retryable errors to avoid duplicate billing.
    • Observability: record request latency, input/output token counts, and error distribution; connect to logging and monitoring alerts to support cost optimization.
    • Transport robustness: adjust Kestrel request body size limits to accommodate large file contexts, set first-byte timeouts for backends, and send SSE heartbeats regularly to prevent reverse proxies from closing idle connections.

    5.2 Common issues

    1. Missing termination event: the agent keeps waiting and the interface stays at “generating.”
    2. Cancellation not propagated: after the user interrupts, the backend continues generating and billing.
    3. Inconsistent function call identifiers: returned function outputs cannot be matched with calls, causing tool execution failures.
    4. Argument escaping issues: if patch content contains newlines, quotes, or other special characters and JSON is concatenated improperly, escaping errors can occur.
    5. Context overflow: too much file content causes the backend to reject the request, requiring truncation or compression of historical messages.
    6. Key leakage: request headers printed in exception stack traces or logs must be masked uniformly.

    6. Security boundaries and production advice

    Script-style adapters like this are intended for local use by default and do not have production-grade capabilities. The example binds to 127.0.0.1 by default, so the agent configuration only needs http://127.0.0.1:18080. When moving toward real deployments, note the following:

    • Listening scope: if external exposure is truly required, authentication and transport encryption must be handled in a front gateway. Do not simply change the bind address.
    • Secret management: inject secrets with environment variables or a secret management service. Do not hard-code them or commit them to version control. Mask logs and exception messages uniformly.
    • Least privilege: agents can execute shell commands and modify files. Run them in containers or dedicated virtual machines and limit accessible directories.
    • Chain records: record which code snippets were sent to which model backend to satisfy code asset compliance audits.
    • Service evolution: for team use, split the adapter into an independent service and add multi-tenant isolation, quota management, and distributed tracing.

    7. Use cases and summary

    The value of this approach is that a small amount of code can turn an agent limited to a single model service into a unified frontend that can switch backends freely.

    Use cases:

    • Daily coding and code review by developers;
    • Code analysis in offline or intranet environments;
    • Teaching demonstrations and protocol research;
    • Low-cost unified access for small and medium-sized teams.

    It cannot replace a formal gateway with enterprise-grade governance capabilities, but as a lightweight adapter it is enough for most needs of individuals and small teams. For further practice, proceed in three directions:

    1. Complete event types and error semantics.
    2. Verify tool calling one by one for target models.
    3. Gradually add routing, observability, and fallback capabilities.

    When switching models changes from modifying code to modifying configuration, developers can truly return their focus to engineering.

    09040904

    Top articles