AI API Relay Services: Where the Money Comes From and How to Evaluate Them

1. What the Low-Cost Claims Mean

Developers increasingly encounter marketing messages such as “low-cost model service,” “full-capability models without downgrade,” and “simple connectivity.” A single account’s revenue may not represent the industry average, but the fact that these models can appear in batches suggests they meet a real and sizable demand: teams and individual developers want to use frontier large language models without being blocked by access barriers or complex billing.

To judge whether such a service deserves long-term investment, it is useful to treat it as a software system. First, examine what money it receives from users. Then examine what costs and risks it bears in order to obtain that money. Only when the request path, pricing model, and data sources are clear can “low cost” be evaluated meaningfully.

2. What Happens to a Single Request

Start with the simplest possible call:

{
  "model": "your-model-id",
  "messages": [
    { "role": "user", "content": "Help me analyze the problem in this code snippet" }
  ],
  "stream": true
}

If you call the model provider directly, this request only passes through three stages: client → provider gateway → inference service. The authentication credential is issued by the model provider, the response includes an identifier generated by the provider, and usage is measured in the official token unit defined by the provider.

When using a relay service instead, the client only receives an endpoint and a key:

POST https://api.example-relay.com/v1/chat/completions
Authorization: Bearer sk-relay-user-key

The request first lands on the relay service’s entry point. It then passes through at least the following layers:

Processing layer	Main responsibility	Typical technical concerns
Access layer	Identify the user, validate the key, check balance and group, determine model permission	Authentication, multi-tenant isolation
Routing layer	Map a unified model name to an actual upstream channel, and select among channels when necessary	Channel pool management, load balancing, timeout and retry, failover
Billing estimation layer	Estimate token usage from input length, check and reserve the corresponding quota	Tokenizer estimation, quota reservation, concurrency control
Forwarding layer	Call the upstream and stream results back to the client chunk by chunk	Streaming forwarding, connection keep-alive, resumable handling
Logging and settlement layer	Record usage, deduct quota based on actual consumption, and generate a bill	Usage details, reconciliation, anomaly alerts

A few points deserve special attention:

• A unified model name is only a mapping. Which channel and version it actually corresponds to is decided by the routing layer.
• Users usually cannot tell how many upstream providers a request passed through, nor can they confirm whether the returned result came from the model they requested.
• The most important “usage” figure in the bill is counted by the relay service itself.

The more layers a request passes through, the longer the chain becomes. As the chain grows, uncertainty increases if any single stage fails or behaves opaquely. Therefore, low cost only becomes meaningful when the request path is traceable and billing can be verified.

3. Where the Profit Comes From: Three Revenue Sources and a Cost Checklist

In general, relay services earn three kinds of money: the money charged for reducing access barriers, the money earned from quota management, and the money made possible by information asymmetry.

3.1 Access Barriers: Packaging the Trouble and Charging a Premium

For individual developers and small teams, directly using frontier model services often involves several practical barriers:

• Network: unstable connections, high latency, or unreliable access.
• Payment: difficulty using the payment methods accepted by the provider.
• Account: registration may require phone or identity verification, and may involve regional restrictions or risk-control bans.
• Terms: data processing terms, content usage authorization, and compliance requirements related to data transfer.

A relay service packages these issues behind a single base URL and key. Users can enter these two values into coding tools, command-line utilities, or custom scripts and start calling the model. Its core value is hiding complexity. A convenience premium is not inherently unreasonable. The key question is whether it is clearly disclosed: what services does the premium correspond to, and can it be explained with a clear bill?

3.2 Quota Management: Top-Ups, Pooling, and Mismatches

The first source comes from payment and in-platform pricing. Users usually top up in their local currency, while the backend may price usage in some kind of virtual credit. For example, a conversion such as “1 local currency unit = 1 credit” may appear. In this case, the so-called credit is often only an internal billing unit, not a bank-rate equivalent of a specific foreign currency. It may also use a fixed conversion ratio that is unrelated to the actual settlement rate or to exchange-rate fluctuations at the time of top-up.

The second source comes from resource pooling and supply-demand mismatch. A light user may top up 100 yuan, use only 10 yuan, and then stop logging in. The remaining quota stays dormant for a long time, while the relay service has already paid upstream resources based on usage. A heavy user may connect a command-line coding tool to a project and run it continuously overnight. For the relay service, such users may be loss-making. Common responses include rate limiting, queueing, or switching channels.

When these two situations combine, the effective unit price borne by individual users may not be lower than the official price. It may simply be averaged and spread across different users.

3.3 Information Opacity: Routing Becomes a Black Box

This is the most concerning of the three revenue sources. Claims such as “full-capability,” “no downgrade,” or “direct provider connection” are difficult for users to verify. In the past, some resale or aggregation channels have been reported to engage in the following practices:

• Using a weaker model while claiming it is a flagship model.
• Replacing a full-precision version with a quantized or compressed version.
• Quietly shortening the context window.
• Injecting additional instructions before user input to change response style.

Some of these practices can be noticed through response quality; others are much harder to detect. When opaque routing is used to “switch models,” the benefits go to the relay provider while the risks remain with the user.

3.4 Cost Structure: Whether a Low Price Is Reasonable Depends on Where the Money Goes

Cost item	Description	Factors affecting scale
Upstream model fees	Input and output token fees for requests forwarded upstream	Request volume, model selection, cache hit rate
Bandwidth and traffic	Especially long-lived connections for streaming output	Concurrency, output length per request
Compute and operations	Infrastructure and staffing for gateways, logs, monitoring, and related systems	System complexity, availability requirements
Risk control and compliance	Content safety, data protection, and related procedures	Business model, user scope
Payment channels and funds	Channel fees, reserve funds, exchange-rate fluctuations	Top-up volume, payment methods
Bad debt and loss	Refunds, banned accounts, unrecovered prepayments	Channel stability, internal management

To judge whether a price is reasonable, the core question is whether it can cover costs and leave a reasonable margin. If the quoted price is clearly below a reasonable cost range, the gap usually has to be filled in one of these ways: lowering upstream quality, betting that prepaid users will not consume their full quota, or reducing investment in security and compliance.

4. Where Tokens Can Quietly Be Lost: Billing Details That Affect Almost Every Caller

Even if the channel is fully transparent, the following details can significantly affect actual costs. Regardless of which type of service you use, it is worth checking each item carefully:

1. Multi-turn conversations accumulate historical messages. Each request carries all previous messages, so input tokens grow with the conversation. If n historical messages are sent on turn k, the number of messages counted as input across the conversation can reach n(n + 1) / 2. This growth is especially noticeable in coding scenarios with frequent tool calls and long results.
2. System prompts and tool definitions also consume tokens. Long system prompts or tool descriptions appear in every request. If the service provider injects additional prompts and those tokens are billed to the user, actual spending may exceed the official reference.
3. Discounts from cache hits may not be realized. Some model providers offer lower pricing for repeated prefix content, but this requires the request to be handled in a specific way and actually sent to a service that supports the capability. Whether users benefit depends on whether the relay service faithfully passes the request through.
4. Output length can easily exceed expectations. If a request does not limit output length, the model may generate long repetitive content, and output tokens are usually much more expensive than input tokens. Setting a reasonable output limit can often reduce the bill significantly.
5. Retries and fallbacks may cause duplicate billing. If an upstream times out and the system retries automatically, or if it switches from one channel to another, tokens already generated may still be billed. Users should confirm whether the bill includes this portion.

A more fundamental principle is that the authoritative source for billing exists only on the side that actually provides the model. Therefore, it is more reliable to pull usage data from the source and reconcile it line by line with gateway records than to simply trust one side’s bill.

5. How to Evaluate a Service: Five Questions

Instead of focusing only on “which service is cheaper,” it is more useful to ask whether the low price is sustainable and verifiable. The following five questions can help evaluate a service:

1. Can it provide per-request usage details? An ideal bill should include input and output token counts, cache-hit information, the model name or version identifier, and request time. Exporting this information is also preferable.
2. Does it clearly state the service level? Rate-limit thresholds, timeout and retry policies, failover behavior, and historical availability determine whether it is reliable in production.
3. How is data handled? Whether request and response content is retained, how long it is retained, who can access it, and whether it is used for model improvement should all be clearly stated.
4. Is the operating entity clear? Whether it can issue formal invoices, sign formal agreements, and provide technical support directly determines whether accountability is possible after an issue occurs.
5. Can users switch away? Whether the service is compatible with mainstream API specifications, whether historical data can be exported, and how much business code would need to change when migrating to a direct provider or another service.

For services that are vague on the first four questions and only emphasize price, they are not suitable for core business workloads. For routine verification, there is also a low-cost method: send the exact same question multiple times and compare whether the response style and level of detail remain stable. Large fluctuations often mean the request has been routed to different channels or models.

6. Building a Controllable Model Gateway with Common Cloud Capabilities

For teams with some engineering capability or large call volume, a more reliable approach is to build a unified model-access gateway on their own side. This allows the team to decide “whose hands the request passes through.”

• Unified entry: all business traffic goes through the company’s own gateway, making auditing, rate limiting, and billing easier.
• Multi-upstream routing: connect to multiple upstream services and dynamically choose based on price, latency, and availability, with automatic fallback if one upstream fails.
• Full logging: record the channel, usage, and latency for every request so reconciliation is possible.
• Budget and quota control: set monthly limits by project or user to avoid unexpected spending caused by abnormal calls.
• Observability: integrate monitoring and alerting to notify teams when error rates, latency, or balance anomalies occur.

This type of gateway does not need to be built from scratch. Mature cloud capabilities can be used to set it up quickly, such as container platforms, object storage, message queues, logging services, monitoring and alerting, key management, and API gateways. Cloud providers such as Baidu AI Cloud offer general-purpose capabilities in these categories. Specific specifications and pricing should be checked against the official documentation.

6.1 Example: Core Gateway Logic

The following Python pseudocode shows a minimal viable gateway skeleton:

import uuid, time
from dataclasses import dataclass
@dataclass
class Account:
    balance: float          # remaining quota
    models: set             # available models
    monthly_spent: float = 0.0
    monthly_cap: float = 100.0   # monthly budget cap
class UpstreamError(Exception):
    pass
def estimate_input_tokens(req) -> int:
    # Estimate with the same tokenizer as the upstream to keep estimates aligned with settlement
    return count_tokens(req.messages) + count_tokens(req.get("tools", []))
def handle_chat(req, account: Account):
    # 1. Permission and budget checks: whether the model is authorized and whether the monthly budget is exceeded
    if req.model not in account.models:
        raise PermissionError("model_not_allowed")
    if account.monthly_spent >= account.monthly_cap:
        raise PermissionError("monthly_budget_exceeded")
    # 2. Estimate and reserve quota based on input tokens and the output limit for a worst-case estimate
    est = estimate_input_tokens(req) * PRICE_IN \
        + req.get("max_tokens", DEFAULT_MAX_OUT) * PRICE_OUT
    if account.balance < est:
        raise PermissionError("insufficient_balance")
    account.balance -= est
    request_id = uuid.uuid4().hex
    # 3. Multi-upstream routing: try by priority and fail over automatically
    for upstream in route_for(req.model):
        started = time.monotonic()
        try:
            resp = upstream.chat(req, request_id=request_id)
            break
        except UpstreamError:
            continue
    else:
        account.balance += est          # release quota if all upstreams fail
        raise UpstreamError("all_upstreams_failed")
    # 4. Use usage returned by the official API as the settlement basis and refund the difference between the hold and actual usage
    usage = resp.usage
    actual = usage["prompt_tokens"] * PRICE_IN \
        + usage["completion_tokens"] * PRICE_OUT
    account.balance += est - actual
    account.monthly_spent += actual
    # 5. Write complete billing details for auditing, reconciliation, and alerts
    audit_log.write({
        "request_id": request_id,
        "model": resp.model,               # upstream model identifier actually used
        "channel": resp.channel,           # channel actually used
        "prompt_tokens": usage["prompt_tokens"],
        "completion_tokens": usage["completion_tokens"],
        "cached_tokens": usage.get("cached_tokens", 0),
        "cost": actual,
        "latency_ms": int((time.monotonic() - started) * 1000),
    })
    return resp

Several design choices in this skeleton are worth borrowing:

• First reserve quota based on the worst case, then settle according to the usage returned by the official API. This avoids excessive spending from concurrent requests while also avoiding overcharging due to overly conservative estimates.
• Model permission, monthly budget, and account balance are checked in sequence. If any check fails, the request does not enter the forwarding stage.
• If routing fails, the full reserved quota is released so users do not pay for failed requests.
• Each audit record includes the upstream model identifier and channel name actually used. This is precisely the information that is most easily hidden by relay services and most worth preserving in a self-hosted gateway.

In practical deployment, teams can choose the appropriate hosting approach based on scale and cost. Individual developers may only need a desktop client or a long-running server process. Team-level services are usually deployed on cloud servers or container platforms, with state stored in a managed database, secrets managed by a key management service, logs sent to a logging service for search and analysis, and monitoring alerts configured for error rates, latency, and balance changes. Message queues are suitable for offline batch processing tasks so that real-time conversations are not affected. Selection can be coordinated with cloud providers’ capabilities such as cloud servers, container platforms, managed databases, object storage, logging services, monitoring and alerting, and API gateways, with details checked against official documentation.

One caveat is that a self-hosted gateway does not automatically eliminate all risks. If multiple upstreams point to the same model provider, the actual dependency has not been diversified. Channel diversification only makes sense when it is based on truly different suppliers.

7. Enterprise Scenarios: Compliance Often Matters More Than Price

For enterprises, choosing a model service also requires considering data and compliance issues:

• Data transfer: sending requests containing business data, personal information, or important data to services outside the country may trigger requirements such as data export security assessments and personal information protection rules.
• Industry regulation: sectors such as finance, healthcare, and government often have additional data-processing requirements. The model-call chain and retention records need to be auditable.
• Content safety: when serving the public, generated content must meet relevant management requirements, and the calling side may need to support content moderation responsibilities.
• Trade secrets: once source code, customer lists, contract terms, and similar information enter an uncontrolled path, the consequences of leakage can be difficult to reverse.

Therefore, enterprises are generally better off prioritizing services with complete compliance qualifications, clear data-processing terms, support for dedicated network access, and audit logs, while using a self-hosted gateway for unified control. Development and testing stages can use lower-cost channels, but calls involving production data should be included in formal data classification, grading, and access-control systems.

8. Conclusion: Low Price Is Not the Problem; Opacity Is

Returning to the original question: where does the money in relay services come from? The answer is the premium charged for reducing access barriers, the float accumulated from quota pools, and the room created by opaque routing. The first two are essentially business models. As long as the rules are clear and the accounts are transparent, users can evaluate them on their own. The one that truly deserves caution is the third: when users cannot even confirm whose hands their request passed through, a low price becomes a transaction that cannot be audited.

Whether for individual developers or enterprise teams, three things are worth checking first:

1. Look at the request path: how many layers does it pass through, what does each layer do, and is it traceable?
2. Look at the bill: whose usage standard is used, and can it be reconciled line by line?
3. Look at fallback options: if something goes wrong, can users switch away, and can accountability be established?

Once these three questions are clear, a low price can be a worthwhile choice. If demand is large and controllability is important, it may be better to build a model gateway using common cloud capabilities, turning “whose hands the request passes through” from a marketing claim into a record that can be queried line by line.