How LLM API Relay Gateways Work: Request Architecture and Profit Model Breakdown

Why “Cheap LLM Access” Is Everywhere

In developer communities, you will keep running into variations of the same pitch: extremely low per-token pricing, upstream capabilities advertised as “full-power,” and integration options marketed as “plug in and go.” Almost all of them describe the same service shape — an LLM API relay gateway.

The essence of this kind of service is packaging. Whatever usage barriers the official APIs impose on some users get bundled up and hidden, and what is exposed outward is a single base URL and an API key. You paste them into your usual client or script and it runs. Whether such a service is worth using has little to do with “how it is built.” What matters is understanding the request path behind it and the profit structure that path enables, because without a clear path there is no reference point for what “low price” even means.

Taking One Request Apart: The Full Technical Path

The best way to understand a relay gateway is to start from a real call.

Client Side: The Request You Send

Suppose you start a streaming chat completion in your client. The request body is usually the industry-standard shape:

{
  "model": "some upstream model identifier",
  "messages": [
    { "role": "user", "content": "Help me find the bug in this code" }
  ],
  "stream": true
}

Notice that this request does not go directly to the model provider. It is sent to the base URL the relay hands you, roughly like this:

POST https://<your-relay-gateway-host>/v1/chat/completions
Authorization: Bearer <your-api-key>

From this point on, the request enters a deliberately engineered multi-layer pipeline.

Layer 1: The Access Layer

The access layer’s job is to catch the request: parse the client payload and validate its format, extract the API key to identify the caller, check whether the request hits an allowed path and group policy, and apply basic rate limiting and concurrency control so a single user cannot overwhelm the backend. Its value is collapsing messy integration details into one uniform entry point — no matter how many different upstream model services sit behind it, the client only ever sees one address.

Layer 2: Authentication and Quota

Once the key is parsed, the server has several questions to answer: who owns this key? Is there enough balance left? Is the caller’s group allowed to use this model? Has a rate limit or quota ceiling been triggered?

This layer is the core of quota management. It maintains the mapping between users and quotas and updates counters before and after each request: a “is there enough?” check on the way in, and a “how much was consumed?” accounting entry on the way out. Your usable boundary is effectively defined here.

Layer 3: Routing and Forwarding

After authentication, the request has to be forwarded to a real upstream. This is often the most consequential layer, and the least transparent. It picks an upstream channel based on the model name, the user’s group, and cost policy; it may rewrite the request; it normalizes responses from different upstreams into a structure the client can parse; and in streaming scenarios it also handles incremental token forwarding and error retries.

What the user sees in the client is “a model.” Which upstream was actually hit, which version, and which specification is usually impossible to verify. This is the technical root of the “third revenue stream” discussed below.

Layer 4: Billing Estimation

Before the request goes out, the system estimates how much quota the call will consume; once a response comes back, it settles up based on the actual input and output token counts. The billing layer typically has to handle differentiated pricing for input versus output, the counting rules for streaming versus batch requests, the policy on whether failed, timed-out, or retried calls are billed, and quota reservation and release when requests queue under concurrency. Whether the billing rules are transparent, and whether they match the upstream’s own accounting, directly determines whether the price you pay for quota is reasonable.

Behind the “Low Price”: Where the Money Comes From

Once the path is broken apart, the revenue sources become obvious. A relay gateway usually does not make money on the model itself. It makes money on three things.

1. Payment and On-Platform Credit Conversion

Many users top up in their local currency while the dashboard prices usage in a credit balance, sometimes with a deliberately round conversion such as “1 unit of local currency ≈ 1 credit unit.” The key detail: this “dollar” is usually not a dollar in the foreign-exchange sense but an on-platform credit unit defined by the service itself. The conversion basis, the ratio applied, and whether fees are layered on top are all controllable margin. The fuzzier the threshold, the harder it is for users to compare prices across providers.

2. The Credit Pool

There is a natural time gap and a natural accumulation between what users top up and what they actually consume. Light users buy a large balance, use a small fraction of it, and stop. Heavy users wire a coding assistant into their projects and run batch jobs overnight, and those users are handled with rate limits, queues, and upstream switching. Below the average, a portion of idle credit simply becomes revenue for the operator — the same logic as the float on prepaid cards and prepaid mobile top-ups.

3. Opacity in Models and Channels

What users are really buying is the outcome “I can use a given model,” and they can rarely verify which channel or which specification sits behind it. When “full-power” versus “degraded,” or “direct from the official API” versus “a second-hand relay,” cannot be falsified, pricing acquires elasticity. Information asymmetry is itself a monetizable asset.

From the User’s Side: Building a Verifiable Judgment

Once you understand the path and the revenue structure, evaluation has a method:

1. Verify the model is real. Run controlled comparisons with fixed prompts and fixed parameters, and observe whether response characteristics, context length handling, and refusal behavior stay consistent. Stable, measurable differences are far more convincing than something that merely “looks like” the advertised model.

2. Take data and key security seriously. Your requests pass through four checkpoints: access, authentication, routing, and billing. Whether plaintext is logged, whether request logs are retained, and how keys are stored all belong in the evaluation. Before sending sensitive data through, work out how many hands it passes through.

3. Clarify the billing basis. Establish how the credit unit maps to real token-based billing, and ask explicitly whether failures, timeouts, and retries are charged. Only once the opaque conversion is made explicit can you actually compare prices.

4. Assess stability and compliance. Streaming forwarding, concurrency queuing, and upstream switching all affect latency and success rate; when enterprise data is involved, compliance and traceability matter too.

Self-Hosting a Gateway: Turning Opacity into Control

If your team is large enough and your call volume is high enough, rather than passively accepting a third party whose path you cannot pin down, it often makes sense to own the gateway layer yourself. A minimally viable self-hosted gateway only needs to cover the four layers described above:

Client → Access layer (parsing / rate limiting) → Auth layer (keys / quota)
       → Routing layer (multi-upstream selection / format normalization) → Billing layer (token counting / accounting) → Upstream model service

In the cloud, these capabilities can be assembled from general-purpose components: a gateway or load balancer to provide the uniform entry point and rate limiting; key management and access control for authentication and permission isolation; a message queue to absorb spikes and relieve concurrency queuing; logging, monitoring, and alerting to retain call data and observe latency and error rates; and object storage to archive the request records needed for audits.

The value of self-hosting is not just eliminating the middleman’s markup. It is a clear path, billing rules you control, and well-defined data boundaries. Once you know what every layer does, whether a price is “low” stops being a matter of faith.

Conclusion

The “low price” of an LLM API relay gateway is essentially three capabilities sold as a bundle: hiding the access barriers, monetizing the float in the credit pool, and putting a price on channel opacity. That is not necessarily a bad thing, but as a user you need to be able to take the path apart — how many layers a request crosses, whose hands it passes through on the way back, and whether the billing basis is clear. Answer those questions and you can genuinely judge whether a quote is good value or simply too cheap to mean anything. For engineering teams, owning the gateway layer of abstraction means owning the initiative over both cost and risk.