LLM Integration for Coding Tools: The Protocol Is the Real Substance

1. Starting from a familiar frustration

Most developers start with a simple goal: you hold an API key or some quota for one model provider, but what you actually use every day is a different terminal coding agent — one that feels right, can read and write files, and can run commands. The two don’t speak the same language. The coding tool only understands its own native protocol, the model only exposes an endpoint that mimics a popular API style, and a translation layer sits in between. So “how do I plug this in” becomes the first hurdle.

My first instinct — and probably yours — was to grab an off-the-shelf protocol conversion tool and change a few environment variables. Instead, the problems multiplied:

• Simple Q&A looked fine, but the moment the agent was asked to edit code it fell apart, either refusing to call tools or emitting a wall of text nothing could parse;
• Streaming stuttered; long answers regularly stopped mid-sentence;
• Multi-turn tool calling died halfway with a 400 complaining about missing parameters or an invalid message structure;
• Occasionally it worked, then a slightly harder task broke it again, with no pattern in sight.

Three principles got me out: trust only the protocol, never the adapter’s promise of universality. When something breaks, put the raw request and raw response from both sides side by side and compare field by field, instead of retrying configuration options. What follows walks through that in the order of concepts, mechanics, practice, and debugging.

2. Why “universal adapters” always fail

2.1 A protocol conversion layer is stateless structural mapping

Gateway, proxy, or router — whatever you call it, the conversion layer does exactly one thing: translate one protocol’s message structure, tool definitions, streaming events, and error semantics into a form the other protocol can recognize. It has no reasoning ability of its own. It cannot conjure capabilities the model lacks, and it cannot guess how the model should call a tool.

Only three things are genuinely solvable at the translation layer:

• Field mapping: what each field is called and where it belongs;
• Format conversion: how text formats, structured JSON, and binary content convert into each other;
• Semantic alignment: how concepts that are close but not identical on the two sides correspond.

Three model-side limitations are unfixable:

• whether instruction-following ability is good enough;
• how large the context window is;
• whether tool calling is natively supported.

If the model itself cannot work with a complex system prompt, no amount of polish in the translation layer will rescue the actual coding quality.

2.2 The adapter’s black-box nature amplifies failures

The real problem with third-party adapters isn’t weakness — it’s opacity:

• Requests get rewritten without you knowing. Most adapters trim or rewrite fields for compatibility: dropping unknown parameters, merging system messages, renaming tool parameters. When things break you cannot tell whether the model answered wrong or the request was mangled on the way in.
• Errors get swallowed or flattened. Detailed upstream error messages get wrapped into a single “request failed”, leaving you with no lead.
• Streaming responses get buffered. Some implementations collect the entire response before parsing SSE and forwarding, which shows up in the UI as “hangs for ages, then dumps everything at once”. Worse, some coding tools treat a long silence as a dropped connection, producing unexplained retries and even duplicated execution.

3. What a coding agent actually needs from a model

To understand the pitfalls, you first have to understand how a coding agent works. Terminal coding tools built around “read code, edit code, run tests” are all essentially an agent loop:

1. Assemble the system prompt, project rules, available tool list, and current conversation into a request and send it to the model;
2. If the model returns plain text, show it to the user; if it returns tool calls, execute them locally — read a file, search the codebase, run a shell command, run tests;
3. Append the tool results as new messages and request the model again, until it stops asking for tools or the turn limit is reached.

┌──────────────┐   request (system + tools + messages)    ┌──────────────┐
│  Agent shell │ ───────────────────────────────────────▶ │ Model server │
│              │ ◀─────────────────────────────────────── │              │
└──────┬───────┘   response (text / tool_use list / stop reason) ──────────┘
       │ tool_use: read files, search code, run commands, run tests
       ▼
┌──────────────────┐
│ Local execution  │
└──────────────────┘

The “shell” is the part the coding tool owns: the system prompt, tool definitions, context management, and permission confirmations. The model does exactly one thing here — decide, based on the current conversation, whether to speak next or which tool to call with which arguments. Seen this way, “integration” means satisfying every contract requirement this loop places on the endpoint. There are five:

Contract	Requirement
Capability	The model must natively support tool calling. “Teaching” a model via prompts to emit JSON almost certainly fails with a large tool set
Structure	Tool definitions, tool calls, and tool results must be represented in messages exactly as the shell’s protocol expects
Streaming	Incremental output must be supported; tool call arguments typically arrive as JSON fragments in pieces and must be reassembled completely
Context	The input limit must accommodate a full round of the request, with a clear error before the limit is exceeded
Stability	Error semantics must be clear — distinguishing rate limiting, authentication, invalid parameters, and server overload — so the shell can retry or degrade

Shift your view from “find a tool that works” to “satisfy this contract” and the causes of most problems become obvious.

4. The protocol is the real substance: structural differences between two API paradigms

The compatible endpoints most model services offer borrow from the chat-completions design: a messages array, a role field distinguishing speakers, choices organizing candidate results. The native protocol a coding tool expects uses a message + content block design: system is a standalone parameter, and content can contain text blocks, image blocks, tool use blocks, and tool result blocks. The structures differ; the concepts are similar; the conversion must be aligned item by item.

The field names below only illustrate protocol structure. The authoritative definitions live in the official API documentation of whatever service you integrate with.

4.1 Message structure: where system goes, which message carries tool results

The chat-completions style has no separate system field. The system prompt is usually a message with role set to system, placed at the head of the array. Multiple system messages may be merged or kept separately, depending on the server implementation. The native protocol makes system a standalone request parameter, and tool results must be wrapped in a message whose role is user, expressed as tool_result content blocks and linked to the corresponding call with tool_use_id.

The easiest place to get this wrong is parallel tool calls. A model may request several tools in one turn, and all tool results must be gathered in the next user message, with count and order matching the previous turn’s calls exactly:

• omit one result and the next turn may be rejected as “tool call with no result”;
• splitting tool results into separate messages, one per message, can also fail structural validation.

What the conversion layer does:

# Illustration: structural conversion of tool calls and tool results
# Left: an array of assistant/tool role messages (chat-completions style)
# Right: a standalone system field plus a message list whose content is an array of content blocks (native protocol)
TOOL_RESULT_TO_USER_MSG = True   # Merge tool results into tool_result blocks inside a user message
MERGE_SYSTEM_MESSAGES = True     # Concatenate multiple system messages into one system parameter
def normalize_request(req: dict) -> dict:
    systems, messages = [], []
    for m in req.get("messages", []):
        role = m["role"]
        if role == "system":
            systems.append(m["content"])
            if not MERGE_SYSTEM_MESSAGES:
                messages.append(m)            # keep as-is, let upstream decide
            continue
        if role == "user":
            messages.append(m)
            continue
        if role == "assistant":
            blocks = []
            if m.get("content"):
                blocks.append({"type": "text", "text": m["content"]})
            for call in m.get("tool_calls") or []:
                args = call["function"]["arguments"] or "{}"
                blocks.append({
                    "type": "tool_use",
                    "id": call["id"],
                    "name": call["function"]["name"],
                    "input": json.loads(args),  # must be an object, not a JSON string
                })
            messages.append({"role": "assistant", "content": blocks})
            continue
        if role == "tool":
            # Parallel calls: multiple results from the same round must be merged into one user message
            result = {
                "type": "tool_result",
                "tool_use_id": m["tool_call_id"],
                "content": [{"type": "text", "text": m["content"]}],
                "is_error": m.get("is_error", False),
            }
            if TOOL_RESULT_TO_USER_MSG:
                if messages and messages[-1]["role"] == "user" \
                        and isinstance(messages[-1]["content"], list) \
                        and messages[-1]["content"][0].get("type") == "tool_result":
                    messages[-1]["content"].append(result)    # append to the previous user message
                else:
                    messages.append({"role": "user", "content": [result]})
            else:
                messages.append({"role": "user", "content": m["content"]})
    out = {"messages": messages}
    if systems:
        out["system"] = "".join(systems)
    return out
def normalize_tools(tools: list[dict]) -> list[dict]:
    # Tool definitions need restructuring too: function/parameters -> name/input_schema
    return [{
        "name": t["function"]["name"],
        "description": t["function"].get("description", ""),
        "input_schema": t["function"].get("parameters") or {"type": "object", "properties": {}},
    } for t in tools or []]

4.2 Tool calls: from an arguments string to an input object

In the native protocol a tool use block consists of id, name, and input, where input is already a JSON object. The chat-completions style stuffs the name and parameters into a function field, with parameters as a serialized string. Converting the response takes two steps:

1. Parse that string — failure means the model emitted invalid JSON;
2. Convert into the native structure.

Streaming makes it harder: arguments arrive as several delta fragments that must be accumulated by index and concatenated, then parsed once at the end.

Stop reason mapping is the most overlooked link. The two sides use different enum values with subtly different semantics, so you need an explicit mapping table:

chat-completions finish_reason	native protocol stop_reason	What the shell should do
stop	end_turn	The model is done; show the text to the user
tool_calls	tool_use	Execute all tool_use blocks in content, return results, continue the loop
length	max_tokens	The answer was truncated: tool call JSON is often incomplete and must never be treated as a normal ending
content_filter	refusal	Content was blocked; ask the user to rephrase

The length / max_tokens pairing is a hot spot for hidden failures: the model is just about to call a tool when output hits the ceiling and is cut off, leaving half a JSON object. If the adapter hands that back as a normal response, the shell either executes a tool call with broken arguments or falls into a loop where the model keeps calling tools and the tools keep failing.

# Illustration: response conversion — tool calls, stop reason, and usage
FINISH_REASON_MAP = {"stop": "end_turn", "tool_calls": "tool_use", "length": "max_tokens"}
USAGE_MAP = {
    "prompt_tokens": "input_tokens",
    "completion_tokens": "output_tokens",
    "total_tokens": "total_tokens",   # rely on the fields each side actually returns
}
def normalize_response(resp: dict) -> dict:
    choice = (resp.get("choices") or [{}])[0]
    message = choice.get("message") or {}
    blocks = []
    if message.get("reasoning_content"):
        # Some models return thinking content in a separate field; restore it as a thinking block
        blocks.append({"type": "thinking", "thinking": message["reasoning_content"]})
    if message.get("content"):
        blocks.append({"type": "text", "text": message["content"]})
    tool_uses = []
    for call in message.get("tool_calls") or []:
        try:
            args = json.loads(call["function"]["arguments"] or "{}")
        except json.JSONDecodeError:
            args = {}   # don't raise on incomplete arguments; let the stop reason decide
        tool_uses.append({
            "type": "tool_use", "id": call["id"], "name": call["function"]["name"], "input": args,
        })
    blocks.extend(tool_uses)
    return {
        "content": blocks,
        # Only tool calls with no accompanying text count as a genuine request to call tools
        "stop_reason": "tool_use" if tool_uses and not message.get("content")
                       else FINISH_REASON_MAP.get(choice.get("finish_reason"), "end_turn"),
        "usage": {USAGE_MAP[k]: v for k, v in (resp.get("usage") or {}).items() if k in USAGE_MAP},
    }

4.3 Streaming responses: event types, incremental fields, and the closing frame

The chat-completions style streams as an SSE (Server-Sent Events) stream: every line starts with data: and choices[0].delta carries the increment. The native protocol organizes its stream by event type. Common types are:

• message_start: the message begins;
• content_block_start / content_block_delta / content_block_stop: the start, increment, and end of a content block;
• message_delta: carries the final stop reason and usage;
• message_stop: the whole message ends.

A text increment is usually a short phrase, but a tool call increment is a scattered JSON fragment — first {"file_pa, then th": "src/ma. Judging only by “is there content” and ignoring the type will glue a tool call into the UI as ordinary text.

# Chat-completions style (illustrative)
data: {"choices":[{"index":0,"delta":{"content":"I'll help you with that"}}]}
data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_1",
      "function":{"name":"read_file","arguments":"{\"path\":"}}]}}]}
data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,
      "function":{"arguments":"\"src/main.py\"}"}}]}}]}
data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}]}
# Native protocol (illustrative)
event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"I'll help you with that"}}
event: content_block_start
data: {"type":"content_block_start","index":1,"content_block":{"type":"tool_use","id":"call_1","name":"read_file","input":{}}}
event: content_block_delta
data: {"type":"content_block_delta","index":1,"delta":{"type":"input_json_delta","partial_json":"{\"path\":"}}
event: content_block_stop
data: {"type":"content_block_stop","index":1}
event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"tool_use"},"usage":{"output_tokens":42}}
event: message_stop
data: {"type":"message_stop"}

Three things matter in streaming conversion:

1. Route increments to the right place: text increments, thinking increments, and tool argument increments each accumulate in their own container;
2. Wait for the closing frame: parse accumulated tool arguments only when message_stop or finish_reason arrives — parsing mid-stream is guaranteed to fail;
3. Give every upstream event a matching downstream event: the native protocol requires each content block to have a paired start and stop, and a missing event will wedge the client’s state machine.

Also note that the native protocol lets clients send ping events to keep the connection alive, and servers may inject ping frames. The adapter should pass these through untouched rather than mistaking them for content.

4.4 Context, multimodality, and “thinking” content: three underrated traps

Context overflow is the most common hidden failure. Every turn, the coding agent sends the system prompt, tool definitions, project rules, and the entire conversation history. Long tasks push the input close to the model’s ceiling. A rejected request, a missing error code, or a response truncated mid-generation all show up as “tools suddenly stopped working”. The conversion layer should do three things:

• read the total request length and compare it against the upstream’s stated limit, returning a semantically clear error when exceeded instead of forwarding blindly;
• map upstream limit errors to a unified error code;
• report truncation honestly in the stop reason, and never disguise a truncated response as a normal result.

Multimodal input has to be split by block. Coding tools frequently ask the model to “look at a screenshot and fix the page” or “read this architecture diagram”. If the model doesn’t support images, silently dropping image blocks is worse than returning an explicit “image input not supported” error — the model would otherwise face a silently degraded context.

Returning thinking content from a reasoning model is stricter. In extended thinking mode, the thinking blocks a model returns carry a signature. On the next request, signed thinking blocks must be placed back verbatim in the content of the corresponding assistant message, ideally preserving the original order relative to the body text and tool call blocks, or validation may fail. Thinking content should not be extracted on its own and displayed or concatenated as ordinary text. If the underlying model returns internal reasoning in a separate field, the conversion must restore it as a signed thinking block rather than dropping it.

Keep in mind too that extended thinking generally requires the output budget to exceed the thinking budget — setting the budget too low will error out the same way.

4.5 Prompt caching: one system prompt, two billing models

A coding tool’s system prompt often runs to tens of thousands of tokens and is re-sent every turn. The native protocol uses cache_control to mark the content prefix you want cached; on a cache hit, billing for that portion of the input drops sharply — though caches have a minimum lifetime and can expire under very frequent requests. Chat-completions style caching is mostly decided server-side, typically hitting the longest common prefix, and asks very little of the client.

Don’t forward cache_control verbatim to a service that only does automatic caching, and don’t assume automatic caching will cover your tool definitions. Treating the difference in caching mechanics between the two sides as the focus of cost optimization usually beats endlessly tuning model parameters.

5. A practical path: from picking a model to getting the loop running

5.1 Step one: check whether the model meets the basic bar for a coding model

Before integrating, verify the model service against this checklist:

• Tool calling: natively supported? Are arguments objects or strings? Is parallel calling possible?
• Instruction following: with a system prompt of hundreds to a thousand-plus lines and a dozen or more tool definitions, can it pick the right tool and generate valid arguments?
• Context length: can the input ceiling cover the context of a typical coding task? Does it error clearly when exceeded?
• Streaming: does it return standard SSE? Are tool argument increments complete?
• Multimodality and thinking: do you need image input or a deep reasoning mode? Are the corresponding fields present?
• Structured output: do you need a strict JSON mode to lower the argument error rate?

This checklist doubles as a selection guide. Different versions and deployment tiers of the same model can differ substantially in capability — judge by the actual API, not by marketing claims.

5.2 Step two: choose the integration form

There are three common approaches:

1. Direct connection: when the protocol the coding tool natively supports matches the endpoint protocol the model service offers, you only need to change configuration. This is the most stable option and should be verified first.
2. A thin conversion layer: when the protocols differ, use a few dozen to a few hundred lines of code for one-way conversion, mapping only the fields you use and keeping complete request and response logs. It offers the most control and is the recommended default.
3. A general-purpose routing tool: aimed at multi-model scenarios, feature-rich, with an active community. Its strength is unified multi-model access and failover; for an individual developer, though, the many configuration options and hard-to-observe behavior mean you often can’t tell which layer broke.

Whichever you pick, make “being able to see the raw payloads” a hard requirement. Running the conversion layer in your own environment — a container platform or a cloud VM — makes debugging far easier than depending on someone else’s hosted black box. On Baidu AI Cloud, for instance, you can use the container platform for version management, the log service to retain the raw text of every request and response, and monitoring alerts to watch error rates and latency.

5.3 Step three: three-stage integration testing that keeps problems contained

Don’t start by experimenting inside the full coding tool. Verify in the following order, so each stage can be confirmed independently:

Stage one: a single conversation. Use a minimal request to verify basic connectivity, confirming auth, model, and endpoint are fine.

# Illustration: minimal non-streaming request to verify auth, model name, and basic conversation
curl -sS -X POST "$BASE_URL/chat/completions" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "model": "'"$MODEL"'",
        "messages": [{"role": "user", "content": "Explain the average time complexity of quicksort in one sentence"}],
        "max_tokens": 128,
        "stream": false
      }'

Stage two: streaming plus a single tool call. Verify SSE increment parsing and one complete “call — execute — answer” cycle. Three checkpoints matter: whether the tool argument JSON fragments are fully concatenated, whether the stop reason is mapped to “call tools requested”, and whether the model can answer from the results after they are returned.

Stage three: multi-turn tool calling plus long context. Build a multi-turn task like “read file → edit code → run tests → fix based on the error”, and watch whether parallel call results are merged correctly, whether context overflow is detected, and whether thinking content is returned legally. Many “intermittent” problems only surface at this layer.

For each stage, send the same request body to both the native endpoint and the converted endpoint and compare responses field by field. This pinpoints whether the problem is a mapping issue or a model issue within minutes.

5.4 Step four: write the agent loop itself

Once the conversion layer works, use a few dozen lines of loop to test whether the model can really do the work. Its core logic is identical to that of a coding tool, and it is the most direct yardstick for “can this model actually do the job”:

# Illustration: a minimal agent loop for evaluating multi-turn tool calling
def run_agent(user_input: str, max_turns: int = 20) -> str:
    messages = [{"role": "user", "content": user_input}]
    for turn in range(max_turns):
        # normalize_request / normalize_response handle protocol conversion
        resp = chat(messages, tools=TOOL_SCHEMAS, stream=True)
        # 1) Truncation takes priority over tool calls: tool arguments cut off by max_tokens are untrustworthy
        if resp["stop_reason"] == "max_tokens":
            return "[output truncated at the limit] " + resp["text"]
        # 2) Done: display the text and exit the loop
        if resp["stop_reason"] != "tool_use":
            return resp["text"]
        # 3) Write back in original order: signed thinking blocks, text blocks, and tool use blocks must all be kept
        messages.append({"role": "assistant", "content": resp["content"]})
        # 4) Execute tools locally; merge parallel call results into a single user message
        results = [execute_tool(block) for block in resp["content"] if block["type"] == "tool_use"]
        messages.append({"role": "user", "content": results})
    return "[max turns reached, possible infinite loop]"

Test it with at least three kinds of task: a small single-file change, a cross-file refactor, and a test loop that fails before it passes. The first checks basic tool calling, the second checks instruction following under long context, and the third checks multi-turn reasoning and self-correction. Passing all three is a reasonable basis for judging whether a model is suitable as a coding model.

5.5 Cost and performance: measure first, optimize second

Token consumption in coding work has a very different shape from ordinary chat:

• Input far outweighs output: every turn re-sends the system prompt, tool definitions, and conversation history, so input tokens are often several times — sometimes tens of times — the output;
• Caching is highly valuable: a stable, unchanging prefix is the best caching candidate and has the most direct effect on cost;
• Parallelism amplifies rate-limit pressure: one tool call can trigger parallel reads of several files, so requests bunch up in a short window and easily hit requests-per-minute or tokens-per-minute limits;
• Long output is capped: an output limit that’s too small truncates code generation, while one that’s too large may burn tokens on nothing useful.

So the optimization order should be: measure the per-turn input/output breakdown, then enable caching and verify hit rates, then tune concurrency and timeouts, and only then consider switching models. Judge every effect by task completion rate and token consumption on real work, not by feel.

6. Quick troubleshooting reference

Symptom	Most likely cause	What to do
The model only produces text and never calls tools	The model doesn’t support tool calling, or the tool definitions weren’t converted correctly	Check that tool definitions reach upstream intact; confirm the model natively supports tool calling
Tool arguments fail to parse	Streaming increments weren’t fully concatenated, or output was cut off by max_tokens	Parse arguments at the closing frame; raise the output limit; inspect the stop reason
Invalid structure after several turns	Parallel call results weren’t merged into a single user message	Match calls to results by id and merge them uniformly
“Thinking block signature invalid”	The thinking block was dropped, rewritten, or reordered	Return thinking blocks verbatim and preserve content block order
Response hangs, then times out and retries	The adapter buffered the stream, or ping frames were swallowed	Switch to frame-by-frame passthrough and shorten the no-output timeout
Errors appear as soon as context grows	The input exceeds the limit	Add a length pre-check, trimming above the limit, and a clear error
The same request fails intermittently	Server overload or rate limiting	Distinguish error codes and apply exponential backoff only to retryable errors

The overall debugging principle is a single sentence: always look at the raw payloads first. Log upstream requests and responses verbatim, compare them field by field against both protocols’ documentation, and the vast majority of problems can be located within ten minutes.

7. Turning hard-won experience into engineering capability

Looking back, the value of the whole exercise isn’t “we got some model working” but three shifts in thinking:

• From tool hunting to protocol reading. An adapter is just a protocol translator. Once you understand both protocols’ fields, events, and error semantics, you hold the initiative in debugging. The protocol is the substance; the tool is only a means.
• From “it runs” to “it’s observable”. Only by logging requests and responses in full, classifying errors, and monitoring latency and success rates can intermittent problems become reproducible and locatable.
• From saving money to staying sustainable. Free-tier or quota-farming access can deliver low cost for a while, but only when caching, concurrency control, and usage analysis turn cost and effectiveness into measurable metrics can such a setup stay stable long term.

The same approach applies to teams building their own coding assistant or a unified model gateway: define the agent loop’s contract with the model API clearly first, then pick the model service whose protocol is closest, and finally close the remaining gap with a thin conversion layer. With most model services now offering endpoints compatible with mainstream API styles, this work is far easier than it was a few years ago. But the hidden costs of protocol differences remain, and they deserve serious attention at the selection stage.

8. Conclusion

The detour from believing in a “universal adapter” to accepting that “the protocol is the real substance” wasn’t wasted. It makes clear that integrating a model isn’t a matter of editing a few config values — it’s understanding every difference between two API designs in message structure, tool calling, streaming events, and context management. Once you’ve internalized that, switching to any model or any coding tool takes far fewer wrong turns.