Access LLM APIs on a Free Tier with a Protocol-Gateway: OpenAI-to-Anthropic Adapter in Practice

    A practical guide to accessing large language model (LLM) APIs within free tiers or at low cost while keeping AI coding tools flexible. It explains how to decouple front-end coding tools from back-end models, choose inference platforms by free quota, capability, compatibility, rate limits, and compliance, and bridge the protocol gap between OpenAI-style and Anthropic-style APIs. The article includes a runnable Python gateway built on FastAPI and httpx that accepts Anthropic-style /v1/messages requests, converts them to OpenAI-style /chat/completions, and returns complete non-streaming and streaming event sequences. It also covers key security, free-tier compliance, backoff and retries, error mapping, secure deployment, observability, and a checklist of common integration mistakes.

    1. Background: The Cost and Compatibility Problems of AI Coding Tools

    AI coding assistants have become part of many developers’ daily workflow. Code completion, error explanation, refactoring suggestions, test generation, and cross-file edits all deliver meaningful productivity gains. At the same time, the cost of calling large model APIs puts pressure on individual developers and small teams.

    A common misconception is that you must use a specific official coding tool, or bind yourself to one particular model service, to get the full experience. In reality, many tools support a custom backend model address. As long as you provide an interface that is compatible with a given protocol, you can decouple the front-end tool from the back-end model.

    This article addresses two problems:

    1. How to obtain model inference resources within a free tier or at low cost.
    2. How to adapt model services with different protocol styles into an interface that common AI coding tools can recognize.

    One point deserves special emphasis: this article discusses accessing within a free tier or at low cost, not a promise of zero cost. Free tiers, billing methods, rate-limit rules, and terms of service are all determined by the specific platform and change with its operating strategy. Always defer to the official documentation before use.

    2. The Resource Layer: Choosing Free-Tier or Low-Cost Models

    Most mainstream cloud providers and inference platforms offer OpenAI-style compatible APIs. Common configuration fields include:

    • base_url: The inference service address.
    • api_key: The access key.
    • model: The model name.

    When choosing an inference platform, evaluate it across these dimensions:

    Evaluation Dimension Recommendation Notes
    Free tier Look for trials, free models, or time-limited free access Free tiers are limited; long-term zero cost cannot be guaranteed
    Model capability Combine coding, reasoning, long-context, and math strengths Different models excel at different tasks; there is no need to force a single model
    API compatibility Prefer OpenAI-style APIs Higher compatibility means lower migration cost
    Rate limiting Check QPS, RPM, TPM, and concurrency High-frequency scenarios should reserve room for backoff and retries
    Stability Check availability, response latency, and failure rate Free resources may involve queuing or fluctuations
    Compliance Read the terms of service and avoid abusing the free tier No high-frequency requests, no bypassing rate limits, no reselling

    For individual developers, a few practical strategies work well:

    • Use the free tier for prototype validation, lightweight Q&A, and scripting assistance.
    • Use low-cost, pay-as-you-go resources for everyday code completion.
    • Configure “coding-oriented” models and “reasoning-oriented” models separately, and call them per scenario.
    • Wrap different backends through a single gateway to avoid repeatedly changing configuration on the tool side.

    3. The Protocol Layer: OpenAI Style vs. Anthropic Style

    Once you have a model service address, configuring it directly into a coding tool often fails, and the core reason is that the API protocols are not unified. There are currently two common styles for large-model APIs: the OpenAI style and the Anthropic style.

    Aspect OpenAI Style Anthropic Style What the Gateway Must Do
    Request endpoint Usually chat/completions Usually messages Forward inbound requests to the corresponding endpoint
    System prompt Inside the messages array, with the system role Usually a separate top-level field Convert the top-level system into a system message
    Message roles May include system, user, assistant, etc. Usually only user and assistant Filter or convert illegal roles
    Response body choices[].message.content content array Map the response body back to the Anthropic structure
    Stop reason finish_reason stop_reason Perform field mapping
    Streaming output OpenAI-style SSE events Anthropic-style event sequence Convert events chunk by chunk and ensure complete termination

    Many excellent AI coding tools use the Anthropic-style interface. If all you have is an OpenAI-style model service, you can insert a protocol-conversion gateway in the middle:

    • Outward-facing, it disguises itself as an Anthropic-style interface.
    • Inward-facing, it converts requests to the OpenAI style.
    • It reconverts the upstream response back to the Anthropic style before returning.

    4. A Runnable Minimal Gateway: Converting an OpenAI Interface to an Anthropic Interface

    The following is a runnable minimal Python example based on FastAPI and httpx. It can:

    • Receive Anthropic-style POST /v1/messages requests.
    • Convert requests into OpenAI-style POST /chat/completions.
    • Support non-streaming responses.
    • Support streaming responses and emit a complete Anthropic event sequence.
    • Support simple gateway authentication.

    4.1 Dependencies

    Python 3.10 or later is recommended.

    1. pip install fastapi uvicorn httpx

    4.2 Environment Variable Configuration

    1. export UPSTREAM_BASE_URL="https://your-provider.example/v1"
    2. export UPSTREAM_API_KEY="sk-your-upstream-key"
    3. export TARGET_MODEL="your-model-name"
    4. export GATEWAY_API_KEY="local-gateway-key"

    Notes:

    • UPSTREAM_BASE_URL: The upstream OpenAI-style service address, which usually includes /v1.
    • UPSTREAM_API_KEY: The upstream model service key.
    • TARGET_MODEL: The model name the gateway ultimately uses when forwarding to the upstream.
    • GATEWAY_API_KEY: The gateway’s own authentication key, used to prevent arbitrary calls to the local port.

    4.3 Gateway Code

    1. import json
    2. import os
    3. import uuid
    4. from typing import Any, AsyncGenerator
    5. import httpx
    6. from fastapi import FastAPI, Request
    7. from fastapi.responses import JSONResponse, StreamingResponse
    8. app = FastAPI()
    9. UPSTREAM_BASE_URL = os.getenv("UPSTREAM_BASE_URL", "").rstrip("/")
    10. UPSTREAM_API_KEY = os.getenv("UPSTREAM_API_KEY", "")
    11. TARGET_MODEL = os.getenv("TARGET_MODEL", "default-model")
    12. GATEWAY_API_KEY = os.getenv("GATEWAY_API_KEY", "")
    13. def auth_ok(request: Request) -> bool:
    14. if not GATEWAY_API_KEY:
    15. return True
    16. if request.headers.get("x-api-key") == GATEWAY_API_KEY:
    17. return True
    18. auth = request.headers.get("authorization", "")
    19. return auth == f"Bearer {GATEWAY_API_KEY}"
    20. def text_from_content(content: Any) -> str:
    21. if isinstance(content, str):
    22. return content
    23. if isinstance(content, list):
    24. parts = []
    25. for block in content:
    26. if isinstance(block, dict) and block.get("type") == "text":
    27. parts.append(block.get("text", ""))
    28. elif isinstance(block, str):
    29. parts.append(block)
    30. return "\n".join(parts)
    31. return str(content)
    32. def to_openai_payload(req: dict[str, Any]) -> dict[str, Any]:
    33. messages = []
    34. system = req.get("system")
    35. if system:
    36. messages.append({
    37. "role": "system",
    38. "content": text_from_content(system)
    39. })
    40. for msg in req.get("messages", []):
    41. role = msg.get("role")
    42. if role not in {"user", "assistant"}:
    43. continue
    44. messages.append({
    45. "role": role,
    46. "content": text_from_content(msg.get("content", ""))
    47. })
    48. payload = {
    49. "model": TARGET_MODEL,
    50. "messages": messages,
    51. "stream": bool(req.get("stream", False)),
    52. "max_tokens": int(req.get("max_tokens", 4096)),
    53. }
    54. if "temperature" in req:
    55. payload["temperature"] = req["temperature"]
    56. if "top_p" in req:
    57. payload["top_p"] = req["top_p"]
    58. return payload
    59. def map_stop_reason(reason: str | None) -> str:
    60. return {
    61. "stop": "end_turn",
    62. "length": "max_tokens",
    63. "tool_calls": "tool_use",
    64. "content_filter": "refusal",
    65. }.get(reason or "", "end_turn")
    66. def sse(event: str, obj: dict[str, Any]) -> str:
    67. return f"event: {event}\ndata: {json.dumps(obj, ensure_ascii=False)}\n\n"
    68. @app.post("/v1/messages")
    69. async def messages(request: Request):
    70. if not auth_ok(request):
    71. return JSONResponse(
    72. status_code=401,
    73. content={
    74. "type": "error",
    75. "error": {
    76. "type": "authentication_error",
    77. "message": "invalid x-api-key"
    78. }
    79. }
    80. )
    81. if not UPSTREAM_BASE_URL or not UPSTREAM_API_KEY:
    82. return JSONResponse(
    83. status_code=500,
    84. content={
    85. "type": "error",
    86. "error": {
    87. "type": "configuration_error",
    88. "message": "UPSTREAM_BASE_URL or UPSTREAM_API_KEY missing"
    89. }
    90. }
    91. )
    92. req = await request.json()
    93. openai_payload = to_openai_payload(req)
    94. if not openai_payload["stream"]:
    95. async with httpx.AsyncClient(timeout=httpx.Timeout(90)) as client:
    96. resp = await client.post(
    97. f"{UPSTREAM_BASE_URL}/chat/completions",
    98. headers={"Authorization": f"Bearer {UPSTREAM_API_KEY}"},
    99. json=openai_payload,
    100. )
    101. if resp.status_code >= 400:
    102. return JSONResponse(
    103. status_code=resp.status_code,
    104. content={
    105. "type": "error",
    106. "error": {
    107. "type": "upstream_error",
    108. "message": resp.text
    109. }
    110. }
    111. )
    112. data = resp.json()
    113. choices = data.get("choices") or [{}]
    114. choice = choices[0]
    115. message = choice.get("message", {})
    116. usage = data.get("usage", {})
    117. return {
    118. "id": data.get("id", f"msg_{uuid.uuid4().hex}"),
    119. "type": "message",
    120. "role": "assistant",
    121. "model": data.get("model", TARGET_MODEL),
    122. "content": [
    123. {
    124. "type": "text",
    125. "text": message.get("content", "")
    126. }
    127. ],
    128. "stop_reason": map_stop_reason(choice.get("finish_reason")),
    129. "stop_sequence": None,
    130. "usage": {
    131. "input_tokens": usage.get("prompt_tokens", 0),
    132. "output_tokens": usage.get("completion_tokens", 0),
    133. }
    134. }
    135. async def stream() -> AsyncGenerator[str, None]:
    136. msg_id = f"msg_{uuid.uuid4().hex}"
    137. try:
    138. async with httpx.AsyncClient(timeout=httpx.Timeout(120)) as client:
    139. async with client.stream(
    140. "POST",
    141. f"{UPSTREAM_BASE_URL}/chat/completions",
    142. headers={"Authorization": f"Bearer {UPSTREAM_API_KEY}"},
    143. json=openai_payload,
    144. ) as up:
    145. if up.status_code >= 400:
    146. body = await up.aread()
    147. yield sse("error", {
    148. "type": "error",
    149. "error": {
    150. "type": "upstream_error",
    151. "message": body.decode("utf-8", "ignore")
    152. }
    153. })
    154. yield sse("message_stop", {"type": "message_stop"})
    155. return
    156. yield sse("message_start", {
    157. "type": "message_start",
    158. "message": {
    159. "id": msg_id,
    160. "type": "message",
    161. "role": "assistant",
    162. "model": TARGET_MODEL,
    163. "content": [],
    164. "stop_reason": None,
    165. "stop_sequence": None,
    166. "usage": {
    167. "input_tokens": 0,
    168. "output_tokens": 0,
    169. }
    170. }
    171. })
    172. yield sse("content_block_start", {
    173. "type": "content_block_start",
    174. "index": 0,
    175. "content_block": {
    176. "type": "text",
    177. "text": ""
    178. }
    179. })
    180. finish_reason = None
    181. output_chars = 0
    182. async for line in up.aiter_lines():
    183. if not line.startswith("data:"):
    184. continue
    185. chunk = line[5:].strip()
    186. if chunk == "[DONE]":
    187. break
    188. try:
    189. obj = json.loads(chunk)
    190. except json.JSONDecodeError:
    191. continue
    192. choices = obj.get("choices") or []
    193. if not choices:
    194. continue
    195. choice = choices[0]
    196. delta = choice.get("delta") or {}
    197. piece = delta.get("content")
    198. if piece:
    199. output_chars += len(piece)
    200. yield sse("content_block_delta", {
    201. "type": "content_block_delta",
    202. "index": 0,
    203. "delta": {
    204. "type": "text_delta",
    205. "text": piece
    206. }
    207. })
    208. if choice.get("finish_reason"):
    209. finish_reason = choice.get("finish_reason")
    210. break
    211. yield sse("content_block_stop", {
    212. "type": "content_block_stop",
    213. "index": 0
    214. })
    215. yield sse("message_delta", {
    216. "type": "message_delta",
    217. "delta": {
    218. "stop_reason": map_stop_reason(finish_reason),
    219. "stop_sequence": None
    220. },
    221. "usage": {
    222. "output_tokens": max(1, output_chars // 4)
    223. }
    224. })
    225. yield sse("message_stop", {
    226. "type": "message_stop"
    227. })
    228. except Exception as e:
    229. yield sse("error", {
    230. "type": "error",
    231. "error": {
    232. "type": "api_error",
    233. "message": str(e)
    234. }
    235. })
    236. yield sse("message_stop", {
    237. "type": "message_stop"
    238. })
    239. return StreamingResponse(stream(), media_type="text/event-stream")

    4.4 Starting the Gateway

    1. uvicorn app:app --host 127.0.0.1 --port 8080

    By default it listens only on the local machine at 127.0.0.1, which is the safer approach. If you genuinely need LAN access, bind to an internal network address instead, and pair it with a reverse proxy, TLS, and authentication.

    4.5 Non-Streaming Call Example

    1. curl -X POST http://127.0.0.1:8080/v1/messages \
    2. -H "content-type: application/json" \
    3. -H "x-api-key: local-gateway-key" \
    4. -d '{
    5. "model": "claude-style",
    6. "max_tokens": 512,
    7. "system": "You are a rigorous programming assistant.",
    8. "messages": [
    9. {
    10. "role": "user",
    11. "content": "Write a quicksort in Python."
    12. }
    13. ]
    14. }'

    4.6 Streaming Call Example

    1. curl -N -X POST http://127.0.0.1:8080/v1/messages \
    2. -H "content-type: application/json" \
    3. -H "x-api-key: local-gateway-key" \
    4. -d '{
    5. "model": "claude-style",
    6. "max_tokens": 512,
    7. "stream": true,
    8. "system": "You are a rigorous programming assistant.",
    9. "messages": [
    10. {
    11. "role": "user",
    12. "content": "Explain HTTP status code 429."
    13. }
    14. ]
    15. }'

    4.7 AI Coding Tool Configuration Example

    If the tool asks for an Anthropic-style interface address, configure it like this:

    1. Base URL: http://127.0.0.1:8080/v1
    2. API Key: local-gateway-key
    3. Model: any recognizable name; the final model is determined by TARGET_MODEL in the gateway

    If the tool asks for the full interface path, you can also enter:

    1. http://127.0.0.1:8080/v1/messages

    The integration steps can be summarized as follows:

    1. Configure the upstream UPSTREAM_BASE_URL, UPSTREAM_API_KEY, and TARGET_MODEL.
    2. Start the local gateway, which listens on 127.0.0.1:8080 by default.
    3. Enter the gateway address and GATEWAY_API_KEY in the coding tool.
    4. Test the non-streaming call first, then the streaming call.
    5. After confirming the tool displays output correctly, connect it to your real development workflow.

    5. Streaming Responses Must Cover the Complete Event Sequence

    Many compatibility problems do not appear in ordinary requests but in streaming output. If the gateway converts only part of the content and does not send the complete event sequence, the front end may exhibit issues such as “stuck loading,” “only the beginning is shown,” or “the session is never closed after finishing.”

    A minimal complete Anthropic-style streaming event sequence should usually include:

    1. message_start
    2. content_block_start
    3. content_block_delta
    4. content_block_stop
    5. message_delta
    6. message_stop

    If the upstream encounters an exception, you should still emit an error event and follow it with message_stop, so the front end does not wait forever.

    Common streaming problems include:

    • Failing to pass stream: true, so the gateway never enters the streaming branch.
    • Not converting delta.content, so the front end receives no text increments.
    • Not sending message_stop, so the tool cannot finish cleanly.
    • Not mapping finish_reason to stop_reason, so the tool’s state becomes abnormal.
    • A missing or malformed usage, so token accounting is inaccurate.

    6. Engineering Best Practices

    A self-hosted gateway can solve compatibility problems at low cost, but long-term stability requires filling in the engineering capabilities.

    6.1 Key Security

    • The upstream api_key must be injected through environment variables, a key management system, or a local configuration center.
    • Never hardcode keys in the code.
    • Never commit keys to a Git repository.
    • Do not print full keys in logs; keep only the first and last few characters.
    • Set an expiry for keys and support rotation.

    6.2 Compliant Use of Free Tiers

    Free tiers and trial resources must strictly comply with the platform’s terms of service. Recommendations:

    • Do not bypass rate limits or run high-frequency bulk requests.
    • Do not use the free tier for commercial resale or large-scale proxying.
    • Do not keep retrying after the quota is exhausted, which creates useless load.
    • Set call caps and a local budget for free resources.
    • Prioritize prototype validation, development debugging, and low-frequency tasks.

    6.3 Rate Limiting and Retries

    For upstream rate limits, we recommend:

    • Applying exponential backoff on 429.
    • Starting waits of 1, 2, and 4 seconds.
    • Setting a maximum of 3 to 5 retries.
    • Setting request timeouts to avoid connections hanging indefinitely.
    • Adding a local queue or cache for high-frequency tools.

    A simple backoff strategy could be:

    1. 1st failure: wait 1 second
    2. 2nd failure: wait 2 seconds
    3. 3rd failure: wait 4 seconds
    4. 4th failure: wait 8 seconds
    5. Beyond the limit: return a diagnosable error

    6.4 Error Mapping

    The gateway should convert upstream errors into error structures the front end can understand. Common mappings include:

    Upstream Situation Recommended Gateway Behavior
    Authentication failure Return authentication_error
    Quota exhausted Return rate_limit_error or quota_exceeded
    Model not found Return not_found_error
    Content too long Return invalid_request_error
    Upstream timeout Return timeout_error and preserve trace information

    Error responses should preserve as much diagnosable context as possible, such as request ID, elapsed time, and error type, but must not leak keys or sensitive content.

    6.5 Secure Deployment

    If the gateway is used only locally, bind to 127.0.0.1 by default. If other devices need to access it, we recommend:

    • Using a reverse proxy such as Nginx or Caddy.
    • Enabling HTTPS or TLS.
    • Enabling gateway authentication, such as x-api-key.
    • Restricting accessible IPs or internal network ranges.
    • Applying a least-privilege policy to upstream keys.
    • Turning off debug logs before exposing the service externally.

    6.6 Observability

    At a minimum, record these metrics:

    • Total request count.
    • Successful request count.
    • Failed request count.
    • Number of upstream 429s.
    • P95 latency.
    • Average time to first byte.
    • Average streaming duration.
    • Remaining upstream quota.

    Simple alert thresholds could reference:

    • Alert when the failure rate over 5 minutes exceeds 5%.
    • Alert when the 429 ratio over 5 minutes exceeds 10%.
    • Investigate the upstream or network when P95 latency exceeds 15 seconds.
    • Check the upstream connection when a streaming request sends no data for more than 60 seconds.

    7. Common Errors Checklist

    The following issues are extremely frequent in practice:

    1. Misplaced system field
      Not converting the Anthropic top-level system into an OpenAI system message, causing the model to ignore the system prompt.

    2. Missing stream flag
      The front end requests streaming, but the gateway does not enable stream when building the payload, so only a single response is received.

    3. Incorrect finish_reason mapping
      OpenAI’s finish_reason is not converted to Anthropic’s stop_reason, causing abnormal tool behavior.

    4. Missing usage fields
      The gateway does not return input_tokens and output_tokens, so some tools cannot count tokens.

    5. Missing message_stop
      No termination event is sent after streaming ends, so the front end may wait indefinitely.

    6. Insufficient multimodal content handling
      If a request contains complex content blocks such as images or documents, you need to extract the text separately or fall back on formatting.

    7. Model name mismatch
      The model name entered in the tool does not match the model actually supported by the upstream, resulting in model not found.

    8. Misunderstanding the free tier
      Assuming the free tier can be used without limit, ignoring the platform’s rate limits, terms of service, or account restrictions.

    8. Summary

    The key to accessing LLM APIs at low cost is not to bind yourself to a single tool or a fixed model, but to build a decoupled “resource layer + adapter layer” design:

    • The resource layer is responsible for choosing model services within a free tier, at low cost, or on a pay-as-you-go basis.
    • The adapter layer smooths over the interface differences between the OpenAI style and the Anthropic style through a protocol-conversion gateway.

    The value of this approach lies in being reusable, portable, and extensible:

    • Switching models requires only changing the gateway configuration.
    • Connecting a new tool requires only pointing it at the gateway.
    • Any upstream service compatible with the OpenAI style can be brought under unified management.
    • A local gateway reduces the risk of key exposure and makes it easier to add authentication, rate limiting, and logging.

    For individual developers and small teams, as long as you control the boundaries of the free tier, comply with the platform’s terms of service, and complete the streaming conversion and error handling, you can build a stable, maintainable, and reusable AI coding integration setup at low cost.