Protocol Conversion in Practice: Connecting Free LLM APIs to AI Coding Assistants

    This article shows how to build a protocol conversion gateway that connects free or low-cost LLM inference capacity to AI coding assistants. It explains why the two dominant API styles—completion-style and message-style—are not directly interoperable, compares them across authentication, system prompts, message structure, tool definitions, tool calls, stop reasons, usage fields, and streaming, and then lays out a bidirectional conversion design with Python examples. You'll learn to normalize requests, map tool definitions, handle tool-result submission, implement the stateful streaming conversion that breaks most easily, and address engineering concerns such as key management, retries, timeouts, circuit breaking, observability, and compliance. The goal: turn idle free capacity into a repeatable part of daily development instead of a one-off experiment.

    Off-Peak Free-Riding on LLM APIs: A Protocol Conversion Practice for AI Coding Assistants

    1. Introduction: The Real Barrier Behind “Free Usage”

    As AI-assisted programming tools spread quickly, coding assistants have become a “second pair of hands” for many developers—useful for everything from code completion and unit-test generation to cross-file refactoring. Most of these tools, though, depend on cloud-hosted LLMs, and once token usage climbs, API calls turn into a fixed expense. Individual developers and students feel this most: they run business tasks during the day, yet compute sits idle at night and on weekends.

    The idea of “off-peak free-riding” grew out of that gap—hunting for models suited to coding within free or low-cost inference quotas. Many cloud vendors and model platforms periodically open free quotas, trial credits, or time- and quantity-limited free models, letting developers try different model capabilities at near-zero cost. But when people actually get to work, many stall at step one: the endpoint runs a simple Q&A example fine, yet the moment they connect it to an AI coding assistant, they hit message-format errors, broken tool calls, and stuttering streaming output.

    Behind this is a general engineering problem: the model service’s API spec and the client tool’s expected API spec don’t match. There are usually two fixes:

    1. Modify the client to support multiple APIs. This isn’t realistic for most closed-source or deeply encapsulated products.
    2. Put a protocol conversion gateway between them—exposing the API the client expects on the outside and calling the model service’s actual API on the inside.

    The second option involves minimal changes, is reusable, and is easy to observe, and it’s the focus of this article. The rest of the piece works toward these goals:

    • Clarify the differences between the two API specs;
    • Design a bidirectional conversion scheme for requests and responses;
    • Plug idle model capacity into the everyday workflow of an AI coding assistant.

    2. The Root Problem: Why the API Specs Don’t Interoperate

    The LLM service market today has roughly two dominant API shapes. Each is optimized for different things, but their field structures differ substantially:

    Aspect Shape 1: Completion-style API (one industry de-facto standard) Shape 2: Message-style API (default for some coding assistants)
    Auth Bearer key carried in a request header Dedicated API-key header, usually plus an API-version header
    System prompt A system-role message placed inside messages A standalone top-level system field (string or content-block array)
    Message content Plain-text content is usually a string content is always a content-block array: text, image, tool_use, tool_result
    Adjacent same-role messages Generally unrestricted Must merge adjacent same-role messages
    Tool definition tools array; function def nested in a function object, params described with JSON Schema tools array; name and description at top level, params in an input_schema field
    Tool call tool_calls array on the assistant message, arguments is a JSON string A tool_use block in the assistant’s content, input is an object
    Tool result submission A new role-tool message linked via tool_call_id A new user-role message containing a tool_result block
    Stop reason finish_reason: stop, length, tool_calls, etc. stop_reason: end_turn, max_tokens, tool_use, etc.
    Usage stats prompt_tokens, completion_tokens input_tokens, output_tokens
    Streaming Standard chunked transfer, incremental deltas inside choices Named event stream: message start, content-block deltas, message deltas, message stop, etc.

    For an AI coding assistant, the problem lands squarely on tool calling. Coding tasks almost inevitably involve reading files, editing code, running commands, and searching a codebase—all done through tool calls. So whether tool calls convert correctly directly determines whether tools can actually get work done. Get only plain conversation working, and the assistant stays stuck at “chat advice.”

    Beyond structural differences, real use throws up a few “soft differences”:

    • Model aliases: the model name in the client request may not match a model name that actually exists upstream, so the gateway needs to map it;
    • Parameter support: thinking budgets, sampling parameters, response formats, and so on are supported to different degrees across APIs and need to be trimmed or passed through as needed;
    • Token-counting endpoint: some message-style APIs offer a dedicated token-counting endpoint that the client uses to estimate context usage, so the gateway needs a compatible implementation or a sensible fallback.

    So protocol conversion can’t stop at field substitution—it should be designed as a complete semantic adaptation layer.

    3. Solution Design: The Overall Architecture of the Conversion Gateway

    Logically, the conversion gateway splits into four layers:

    1. Access layer: exposes the API the client expects and handles auth, request-body validation, and rate limiting;
    2. Conversion layer: does the bidirectional mapping of requests and responses between the two API shapes, with the hard parts concentrated in system prompts, tool calls, and streaming;
    3. Routing layer: dispatches requests to different upstream services based on headers or model aliases, and maintains an “alias → real model” map;
    4. Observability layer: logs each call’s latency, token usage, stop reason, and error info, for debugging and cost tracking.

    Three design principles should hold:

    • One-way proxy, convert on demand: only transform incompatible fields and pass the rest through untouched, avoiding new uncertainty;
    • Semantic fidelity: the structure, order, and linkage of tool calls must be preserved completely;
    • Observability: any anomaly can be traced back to a specific request, upstream response, and conversion step.

    The code below illustrates the key conversion logic in Python style. The HTTP service can use any thread-safe lightweight framework—a plain synchronous handler plugs in easily.

    4. Key Implementation: Converting Requests, Responses, and Streaming

    4.1 Foundation: System Prompts and Message Normalization

    The completion-style API usually needs system messages extracted into a top-level field of the message-style API. Adjacent same-role messages must be merged, or upstream rejects them. Plain-text messages need wrapping into content-block arrays.

    1. from copy import deepcopy
    2. def normalize(messages):
    3. """Extract system prompts and merge adjacent same-role messages"""
    4. systems, merged = [], []
    5. for msg in messages:
    6. role = msg.get("role", "user")
    7. if role == "system":
    8. systems.append(msg["content"])
    9. continue
    10. # Merge adjacent same-role messages to satisfy the message-style API constraint
    11. if merged and merged[-1]["role"] == role:
    12. merged[-1]["content"].extend(to_blocks(msg["content"]))
    13. else:
    14. merged.append({"role": role, "content": to_blocks(msg["content"])})
    15. return "\n\n".join(s for s in systems if s), merged
    16. def to_blocks(content):
    17. """Normalize a string or content-block array into a list of blocks"""
    18. if isinstance(content, str):
    19. return [{"type": "text", "text": content}]
    20. return deepcopy(content)

    4.2 Tool Definition Mapping

    The function-style API nests tool descriptions inside a function object, which needs to be “flattened” to the top level; the parameter description sits in a parameters field, renamed to input_schema during conversion. Both are JSON Schema, so their semantics align directly.

    1. def convert_tool(tool):
    2. fn = tool["function"]
    3. return {
    4. "name": fn["name"],
    5. "description": fn.get("description", ""),
    6. "input_schema": fn.get("parameters", {"type": "object", "properties": {}}),
    7. }

    4.3 Full Request Conversion

    Combine the snippets above and you get the main request-conversion flow from client to upstream:

    1. def build_upstream_request(req, model):
    2. system, messages = normalize(req["messages"])
    3. if req.get("tools"):
    4. # Tool defs present but no explicit tool-call mode set: default to auto
    5. tool_mode = req.get("tool_choice", "auto")
    6. body = {"model": model, "messages": messages,
    7. "max_tokens": req.get("max_tokens", 4096)}
    8. if system:
    9. body["system"] = system
    10. if req.get("tools"):
    11. body["tools"] = [convert_tool(t) for t in req["tools"]]
    12. body["tool_choice"] = {"type": tool_mode} if tool_mode != "auto" else {"type": "auto"}
    13. if req.get("temperature") is not None:
    14. body["temperature"] = req["temperature"]
    15. return body

    Two details that are easy to miss:

    • Image input: image_url in the completion-style API (a URL or a base64 data URL) needs converting to the message-style API’s base64-based image source block;
    • Parameter trimming: parameters upstream doesn’t support (certain response-format params) are dropped outright and logged, not allowed to fail the whole request.

    4.4 Non-Streaming Response Conversion

    Non-streaming responses mainly handle three parts: body text, tool calls, and stop-reason mapping.

    1. def convert_response(resp):
    2. choice = resp["choices"][0]
    3. message = choice.get("message", {})
    4. content = [{"type": "text", "text": message.get("content") or ""}]
    5. for call in message.get("tool_calls") or []:
    6. fn = call["function"]
    7. content.append({
    8. "type": "tool_use",
    9. "id": call["id"],
    10. "name": fn["name"],
    11. "input": json.loads(fn.get("arguments") or "{}"),
    12. })
    13. return {
    14. "id": resp.get("id", ""),
    15. "type": "message",
    16. "role": "assistant",
    17. "content": content,
    18. "stop_reason": map_stop_reason(choice.get("finish_reason")),
    19. "usage": {
    20. "input_tokens": resp["usage"]["prompt_tokens"],
    21. "output_tokens": resp["usage"]["completion_tokens"],
    22. },
    23. }
    24. def map_stop_reason(reason):
    25. return {
    26. "stop": "end_turn",
    27. "length": "max_tokens",
    28. "tool_calls": "tool_use",
    29. "content_filter": "end_turn",
    30. }.get(reason, "end_turn")

    4.5 Streaming Conversion: The Most Error-Prone Part

    AI coding assistants almost always enable streaming during interaction to show “thinking” and code snippets in real time. The message-style API’s stream is a stateful event sequence, while the completion-style API’s stream is append-only text deltas. Conversion needs a state machine:

    1. On the upstream’s first chunk, emit a message-start event and a text content-block-start event;
    2. Pass each delta through as a text content-block-delta event;
    3. On a tool-call delta, enter a tool-call content block and accumulate the argument fragments;
    4. On the end chunk, emit content-block-stop events and a “message delta” event carrying the stop reason, then finally a message-stop event.
    1. def stream_convert(upstream_lines, msg_id="msg_proxy"):
    2. block_opened = False
    3. tool_args = {} # Accumulate argument fragments per tool-call index
    4. tool_started = {} # Track the content-block index for each tool-call index
    5. next_index = 0
    6. yield ev("message_start", {"type": "message", "id": msg_id, "role": "assistant",
    7. "content": [], "usage": {"input_tokens": 0, "output_tokens": 0}})
    8. for line in upstream_lines:
    9. if not line.startswith("data:") or line.strip() == "data: [DONE]":
    10. continue
    11. delta = json.loads(line[5:])["choices"][0].get("delta", {})
    12. if delta.get("content"):
    13. if not block_opened:
    14. next_index, block_opened = open_text_block(next_index)
    15. yield ev("content_block_delta",
    16. {"index": next_index, "type": "content_block_delta",
    17. "delta": {"type": "text_delta", "text": delta["content"]}})
    18. for call in delta.get("tool_calls") or []:
    19. i, fn = call.get("index", 0), call.get("function", {})
    20. if i not in tool_started:
    21. tool_started[i], next_index = next_index, next_index + 1
    22. block_opened = False
    23. yield ev("content_block_start",
    24. {"index": tool_started[i], "type": "content_block_start",
    25. "content_block": {"type": "tool_use", "id": call.get("id", ""),
    26. "name": fn.get("name", ""), "input": {}}})
    27. if fn.get("arguments"):
    28. tool_args[i] = tool_args.get(i, "") + fn["arguments"]
    29. yield ev("content_block_delta",
    30. {"index": tool_started[i], "type": "content_block_delta",
    31. "delta": {"type": "input_json_delta",
    32. "partial_json": fn["arguments"]}})
    33. if delta.get("finish_reason"):
    34. for i, idx in tool_started.items():
    35. yield ev("content_block_stop", {"index": idx, "type": "content_block_stop"})
    36. yield ev("message_delta", {"type": "message_delta",
    37. "delta": {"stop_reason": map_stop_reason(delta["finish_reason"])},
    38. "usage": {"output_tokens": 0}})
    39. break
    40. yield ev("message_stop", {"type": "message_stop"})

    Three things to watch when implementing this state machine:

    • Start and stop must pair strictly: every content-block start needs a matching stop, or some clients wait forever;
    • Don’t emit the end event early: only after upstream sends the end chunk should you send the message-delta event—only then does the client treat the round as complete;
    • Forward tool arguments as raw fragments: forward upstream’s argument fragments directly instead of waiting to join the full JSON and sending it all at once. The client needs to preview content as it’s being generated.

    5. Tool Calling: From “Can Chat” to “Can Work”

    Tool calling is the core capability of an AI coding assistant, so pay special attention to:

    • Tool-result submission (request side): when the client sends tool results back, convert role-tool messages into tool_result blocks inside a user message, linking via tool_use_id. When the same tool is called in parallel, result order gets shuffled, so you must rely on unique IDs, not name matching;
    • Tool arguments (response side): arguments in non-streaming responses is a JSON string and needs parsing into an object. On parse failure, fall back to an empty object and log, rather than throwing;
    • Tool-call mode: the completion-style API’s tool_choice values (e.g., auto, none) map to corresponding modes in the message-style API. If upstream can’t force a specific tool precisely, degrade to “must use a tool” or “auto”;
    • Tool definition size: many coding assistants register dozens of tools at once, and some models grasp long tool definitions poorly. At the prompt-template level, control how many tools are exposed per round and trim dynamically by task.

    6. Integrating with an AI Coding Assistant: Config and Routing

    Once the gateway is ready, connecting a coding assistant usually takes two steps: point the client’s API base URL to the gateway, and swap the key for the gateway’s key. Most tools do this via config files or env vars:

    1. # Point to the local gateway; use the gateway's own verification token as the key
    2. export CODING_ASSISTANT_BASE_URL="http://127.0.0.1:8787"
    3. export CODING_ASSISTANT_API_KEY="<your gateway token>"

    Config item names vary by tool, but the goal is the same: send the client’s requests to the gateway, not straight to the model service.

    If the gateway fronts multiple models, “model aliases” enable task routing:

    Task type Suggested routing target Why
    Planning, architecture discussion A model with stronger long-context ability Needs to take in a lot of background at once
    Code completion, multi-file edits A model strong at code with fast responses Prioritizes low latency and high accuracy
    Code review, math/logic reasoning A model with stronger reasoning Needs deeper chain-of-thought analysis

    On flow-control params, set the client’s timeout slightly longer than the gateway-plus-upstream timeout sum, so the client doesn’t disconnect first while the gateway is still waiting on upstream, wasting resources.

    7. Engineering and Compliance: The “Invisible Requirements”

    7.1 Keys and Access Control

    • Environment isolation: have the gateway listen only on the loopback address, or add auth behind a reverse proxy—don’t expose the gateway directly to the public internet;
    • Key management: inject upstream keys via env vars or a secret manager, never into the repo; when hot-reload is supported, record a key fingerprint instead of plaintext;
    • Least privilege: give the assistant’s key its own quota and call scope.

    7.2 Stability: Retries, Timeouts, and Circuit Breaking

    • Idempotency check: only back off and retry clearly retryable errors such as connection timeouts and rate limiting, up to two or three times; errors like request-body parse failures shouldn’t be retried;
    • Tiered timeouts: distinguish “first-byte timeout” from “overall timeout.” Reasoning models can have markedly longer first-byte times, so don’t set the overall timeout too short;
    • Connection reuse: use a connection pool and keep-alive for upstream to cut handshake latency;
    • Circuit breaking / fallback: when upstream keeps erroring, briefly trip the breaker and switch to a backup model to keep the assistant usable.

    7.3 Observability and Cost Control

    The gateway should log each call’s model, input/output tokens, first-byte time, and stop reason, and tally usage by task. Even when upstream quota is free, set per-user, per-day call quotas at the gateway to prevent runaway loops or scripts from draining the quota.

    7.4 Privacy and Compliance

    • Code egress risk: coding assistants send source snippets to the model service, so before connecting, confirm whether the terms allow that use and whether the data is used for training; for company code or sensitive data, redact beforehand or choose a compliant enterprise-grade service;
    • Credential leakage risk: config files and shell history may contain real keys, so fold a key fingerprint into content hashes to catch accidental leaks;
    • Nature of free quotas: free or trial quotas often carry rate limits, context-length limits, and commercial-use restrictions—read the terms before use;
    • Contract stability: third-party APIs are a “contract surface”—fields and behavior can change across versions, so centralize the conversion logic and add contract tests.

    7.5 Troubleshooting Common Problems

    Symptom Common cause Where to look
    401/403 auth failure Auth header format mismatch, or expired key Check the gateway’s header and key validity
    400 message-format error Adjacent same-role messages not merged, system message not extracted Check the normalization logic, print the converted request body
    Tool calls ineffective Tool definitions not flattened, args not converted from string to object Compare tool defs and tool_use blocks before/after conversion
    Streaming stalls or cuts out Wrong end-event order, missing content-block-stop event Capture and compare raw event streams from gateway and upstream line by line
    Output truncated max_tokens too small, or mapped to stop reason length/max_tokens Raise max_tokens, confirm the stop-reason mapping is right
    Model name not recognized Alias not configured or misspelled Check the alias map

    8. Conclusion

    The essence of “off-peak free-riding” is turning idle, low-cost compute into everyday productivity. But making it truly usable isn’t about “finding free endpoints”—it’s about bridging the last mile from API spec to workflow. Three takeaways:

    1. Align API semantics before connecting. Auth, system prompts, tool calls, streaming events—leave any one conversion incomplete and tools go from “can chat” to “can’t work”;
    2. Make the conversion layer observable and testable. Use alias routing, retry/circuit-breaking, and usage stats to turn scattered calls into stable infrastructure;
    3. Cost and efficiency ultimately depend on workflow design. Let different models do what they’re best at—humans handle requirement judgment and result acceptance, models handle repetitive coding and checks.

    To add: for more reliable service guarantees and fuller engineering support, beyond building your own conversion layer you can also evaluate LLM services from cloud vendors such as Baidu AI Cloud, choosing based on your own requirements for context length, tool calling, compliance, and cost. Once these technical details are handled well, “free-riding” stops being an occasional experiment and becomes a sustainable development aid.