Connecting Free Cloud LLMs to Coding Tools via a Local Proxy

    If a coding tool lets you set a custom API base URL, you can route it to free cloud models through a local proxy gateway. This guide covers model choice, when a gateway is needed, credential checks, a FastAPI proxy, client field mapping, and a curl connectivity test.

    1. Takeaway first

    If a coding tool allows a custom endpoint, you can point it at free models from a cloud platform. In most cases a local proxy gateway is enough to bridge the request. Free quotas have expiry dates and usage caps. Before you rely on them, confirm the terms in the console.

    2. Choosing between two model types

    The platform currently offers two model families with different strengths.

    Reasoning models emphasize logic and math. They are useful for algorithms, edge cases, and pseudocode before you write production code. Platform docs claim math-reasoning accuracy above 95%. That figure is vendor-reported and not independently verified here; trust your own tests.

    Mixture-of-experts (MoE) models have about 230 billion total parameters and activate about 10 billion per inference. The design trades lower per-request compute for larger capacity, which often fits multi-file edits, context understanding, and debugging from error messages. Parameter counts also come from platform materials; use the official model card as the source of truth.

    Dimension Reasoning model MoE model
    Best for Algorithm design, requirement breakdown, boundary analysis Multi-file edits, error repair, code review
    Suggested use Think through the approach before coding Coding and refactoring
    What to measure Latency, output length Long-context stability, output quality

    3. Decide whether you need a gateway

    If the tool supports an OpenAI-compatible base URL and arbitrary model names, fill in the platform URL and key directly. Add a proxy only when the tool requires a fixed request format, extra headers, or routing by task to different models. A proxy adds latency and needs maintenance.

    4. Checks before you create credentials

    Sign in to the platform console, create an API key, and record each model’s base URL and model ID. Before production use, verify in the console or terms of service:

    • Free-quota validity period and total volume
    • Rate limits (requests per minute and concurrency)
    • Commercial use and any identity-verification requirements
    • Whether the free policy can change

    “Zero cost” in this article means cost inside the free quota. Fees and throttling after the quota are defined by the platform.

    5. Deploy the proxy gateway

    Install dependencies:

    1. pip install fastapi uvicorn httpx

    Gateway code below. All secrets come from environment variables; the source contains no usable keys.

    1. import os
    2. import httpx
    3. from fastapi import FastAPI, HTTPException, Request
    4. from fastapi.responses import Response
    5. UPSTREAM_BASE_URL = os.environ["UPSTREAM_BASE_URL"].rstrip("/")
    6. PROXY_TOKEN = os.environ["PROXY_TOKEN"]
    7. # Client-facing model aliases -> upstream model IDs and keys
    8. MODEL_ROUTES = {
    9. "reasoning": {
    10. "model": os.environ["REASONING_MODEL_ID"],
    11. "api_key": os.environ["REASONING_API_KEY"],
    12. },
    13. "moe": {
    14. "model": os.environ["MOE_MODEL_ID"],
    15. "api_key": os.environ["MOE_API_KEY"],
    16. },
    17. }
    18. app = FastAPI()
    19. @app.post("/v1/chat/completions")
    20. async def chat_completions(request: Request):
    21. if request.headers.get("authorization") != f"Bearer {PROXY_TOKEN}":
    22. raise HTTPException(status_code=401, detail="invalid proxy token")
    23. data = await request.json()
    24. alias = data.get("model", "moe")
    25. route = MODEL_ROUTES.get(alias)
    26. if route is None:
    27. raise HTTPException(status_code=400, detail=f"unknown model alias: {alias}")
    28. # Replace the alias with the real upstream model ID and use that model's key
    29. data["model"] = route["model"]
    30. headers = {
    31. "Authorization": f"Bearer {route['api_key']}",
    32. "Content-Type": "application/json",
    33. }
    34. try:
    35. async with httpx.AsyncClient(timeout=60.0) as client:
    36. upstream = await client.post(
    37. f"{UPSTREAM_BASE_URL}/chat/completions",
    38. json=data,
    39. headers=headers,
    40. )
    41. except httpx.TimeoutException:
    42. raise HTTPException(status_code=504, detail="upstream timeout")
    43. except httpx.HTTPError:
    44. raise HTTPException(status_code=502, detail="upstream unreachable")
    45. # Pass through upstream status and body so clients see real errors
    46. return Response(
    47. content=upstream.content,
    48. status_code=upstream.status_code,
    49. media_type=upstream.headers.get("content-type", "application/json"),
    50. )

    Set environment variables and start the server. Replace values with what you obtained from the console:

    1. export UPSTREAM_BASE_URL="https://platform-provided-address/v1"
    2. export PROXY_TOKEN="a random string you generate for local auth"
    3. export REASONING_MODEL_ID="real reasoning model ID"
    4. export REASONING_API_KEY="cloud key for the reasoning model"
    5. export MOE_MODEL_ID="real MoE model ID"
    6. export MOE_API_KEY="cloud key for the MoE model"
    7. uvicorn gateway:app --host 127.0.0.1 --port 8000

    Limitation: this code does not stream. If the client defaults to stream=true, turn streaming off first. For streaming, switch to FastAPI StreamingResponse and forward chunks.

    6. Client settings and field mapping

    In the tool’s model settings, choose a custom provider or OpenAI-compatible mode and fill in the table below.

    Client field Value Gateway behavior Upstream field
    Base URL http://127.0.0.1:8000/v1 Accepts the request Platform UPSTREAM_BASE_URL
    API Key value of PROXY_TOKEN Validates the local token Not exposed to the client
    Model name reasoning or moe Looks up the real model ID REASONING_MODEL_ID or MOE_MODEL_ID

    The client must send aliases, not upstream model IDs. Alias-to-ID mapping lives only in the gateway.

    7. Verify with curl

    Before wiring the coding tool, confirm the gateway forwards correctly:

    1. curl http://127.0.0.1:8000/v1/chat/completions \
    2. -H "Authorization: Bearer $PROXY_TOKEN" \
    3. -H "Content-Type: application/json" \
    4. -d '{"model":"moe","messages":[{"role":"user","content":"Explain in one sentence"}]}'