Fake Model Detection in Third-Party LLM API Relays: Latency and Token Signals Behind "Adulteration"
Third-party LLM API relays and aggregation platforms are cheap and convenient. They may also quietly replace the high-priced model you paid for with a smaller or free one and bill you for the flagship anyway. This article explains how this kind of adulteration works, why it is hard to catch, and how to detect it. It compares content fingerprinting with metadata-based analysis, which is harder to evade, and shows how to turn detection into routine, automated checks. Readers will learn which latency, token-counting, streaming, and error-behavior signals point to swapped models, plus a practical checklist for choosing providers and avoiding vendor lock-in.
In recent years, many third-party API relay sites and aggregation platforms have emerged. They resell access to large language models at prices below the official ones, attracting both developers and casual users. Yet many of these services carry a hidden problem: adulteration.
By adulteration, we mean providers that do not actually call the high-priced model they claim to offer. Instead, they quietly swap in a cheaper or free smaller model and bill requests at the flagship model’s rate. Users believe they are using a top-tier model when they are actually getting a scaled-down version.
Below is a systematic walkthrough, covering how adulteration works, how to detect it, and how to guard against it.
1. What API Relay Adulteration Is, and Why It Persists
Relay sites are essentially resellers. They buy model access in bulk from upstream providers or obtain quotas through other channels, then resell it to downstream users at a markup. When upstream costs are high and profit margins are squeezed, some providers are tempted to substitute models. They serve requests with a cheaper one and pocket the difference.
Such behavior is hard to spot because, to ordinary users, text from different models “looks about the same.” Short answers and common-sense questions are nearly impossible to tell apart by eye. The gap only becomes visible on complex tasks that demand strong reasoning or long context.
What makes adulteration hard to prevent is information asymmetry. The provider controls the real backend routing, while users only see the output at the front end.
2. Detection Overview: Two Complementary Approaches
In practice, detection methods fall into two categories that work well together:
- Content fingerprint probing: Send crafted inputs, observe the “innate habits” in the model’s outputs, and infer which model is behind the endpoint.
- Request-side metadata analysis: Ignore what the model says and look only at process data from the request and response, such as latency, token counts, and streaming cadence.
The first approach is intuitive but relatively easy to evade. The second is harder to circumvent and tends to be more stable, so it is best to combine the two in practice.
3. Method 1: Content Fingerprint Probing
How It Works
Models differ in their training data, tokenizers, and alignment strategies. As a result, each model retains fairly stable output patterns. When faced with obscure, boundary, or leading inputs, a model may produce content that betrays its identity:
- fixed stock phrases
- disclaimer text in a specific format
- odd repetitions of certain URLs or snippets
- even internal markers
These “dirty tokens” work like fingerprints that reveal the model’s true identity.
Limitations
This method has clear boundaries:
- Fingerprint libraries go stale. Models are constantly updated, so existing fingerprints may stop working as soon as a version changes, and keeping them current takes effort.
- Same-family variants are hard to tell apart. Models in the same family with different sizes or versions often share much of their training corpus and alignment behavior. Small variants can be difficult to distinguish reliably from their content.
- Input-layer filtering defeats it. Savvy relay operators can sanitize or block “trap inputs” that tend to expose the real model at the entry point, so the distinguishing behavior never appears.
For these reasons, content fingerprints work best as auxiliary validation, not as the sole basis for a verdict.
4. Method 2: Request-Side Metadata Analysis (Recommended)
This method has a key advantage. It does not depend on what the model says but on process information from the request and response, which simple input filtering cannot easily hide.
1. Latency Stability Analysis
For identical or similar requests, the same model’s response time tends to be stable overall with local fluctuations. If the backend secretly routes requests among several models, their different inference speeds show up directly in the latency. Responses may come back instantly one moment and take tens of seconds the next, with high variance. This is a strong sign of dynamic model switching.
To run the test, send the same request 10–20 times in a row, record how long each call takes, and examine the distribution.
import time, statisticsdef probe_latency(client, prompt, rounds=15):samples = []for _ in range(rounds):t0 = time.perf_counter()client.chat(prompt) # send the same requestsamples.append(time.perf_counter() - t0)mean = statistics.mean(samples)stdev = statistics.pstdev(samples)print(f"mean {mean:.2f}s, std dev {stdev:.2f}s, "f"dispersion {stdev / mean:.2%}")return samples
How to read the results:
- Official direct connections usually have stable, predictable latency with low dispersion.
- A bimodal distribution (some calls very fast, others very slow) or clearly elevated dispersion is a warning sign that the backend may be switching models.
2. Token Count Consistency
Different models use different tokenizers. The same input can therefore be split into different numbers of tokens, and models also differ in how long their generated outputs tend to be.
To test this, use a fixed, fairly long prompt, such as one asking the model to summarize a few hundred words of text. Send it 5–10 times and compare the token consumption of each response. Most APIs report token counts in the response or usage fields.
- With the same real model, token consumption for the same input should stay relatively stable, with little fluctuation.
- If the counts jump around sharply with a large standard deviation, the backend is likely switching between models, since their tokenization and output-length patterns differ completely.
def probe_token_usage(client, prompt, rounds=8):usages = []for _ in range(rounds):resp = client.chat(prompt)# read token counts from the usage info in the response (field names depend on the API)usages.append(resp.usage.total_tokens)print("token usage per call:", usages)print("spread:", max(usages) - min(usages))
Fix randomness parameters, such as setting the temperature to 0, before comparing. This reduces interference from randomness in generation itself.
3. Other Corroborating Signals
Beyond the two checks above, these process-level signals are also worth adding to regular inspections:
- Streaming cadence: Real streaming output typically arrives at a natural word-by-word or chunk-by-chunk pace. If an entire response lands at once with no sense of streaming, the relay layer may be buffering the full output and then presenting it as a stream.
- Error codes and error formats: Each provider’s error structure and wording for content moderation triggers, overlong inputs, and out-of-range parameters follows a fairly consistent style, which can serve as an auxiliary identification point.
- Capability boundary probing: Try long contexts, specific multimodal features, or particular parameter combinations. If a claimed capability frequently fails or behaves abnormally, the service is likely not delivering what it advertises.
5. Engineering: Making Detection a Routine Check
For developers, a more robust approach is to turn these probes into an automated inspection script. Run it on a schedule (for example, daily) against the same set of fixed prompts to collect latency and token statistics. Write the results to a logging service and configure monitoring alerts. When dispersion or token-count variation crosses a threshold, the script triggers a notification automatically. This lets you detect adulteration as soon as it occurs rather than waiting for user complaints.
A simplified flow:
Scheduled job → send fixed probe requests → collect latency / tokens / error rate→ compute dispersion metrics → if threshold exceeded, alert (message queue + monitoring alerts)→ archive logs for retrospective comparison
6. A Prevention Checklist for Everyday Users
- Prefer official or first-tier channels. Use the model provider’s official API directly, or choose reputable cloud providers with established access to model capabilities (for example, official channels such as Baidu AI Cloud). This is the simplest and most reliable way to avoid third-party relay adulteration at the source.
- Be wary of prices far below the actual cost. Prices that are unreasonably low are often a strong sign of adulteration. Behind such discounts, providers may be swapping in cheaper models.
- Test the scenarios you actually care about. Compare providers on the complex tasks that matter to you, such as long-document reasoning, coding, and multi-turn conversations, rather than judging by simple Q&A alone.
- Watch latency and usage stability. Call the same request several times. If latency swings widely or usage figures vary a lot, be on alert.
- Keep evidence and stay portable. Turn your key calls into a reusable evaluation set. That makes it easy to switch to a backup provider at any time and reduces the risk of being locked in by a single channel.
7. Conclusion
API relay adulteration is a way to profit from information asymmetry. The way to counter it is to use quantifiable process metrics instead of vague subjective impressions. Content fingerprints can provide auxiliary evidence. Latency stability and token count consistency on the metadata side are the more reliable criteria.
For individual users, the best choice is official, reputable channels. For developers, detection should be automated and routine. Once you can continuously and quantitatively verify which model you are actually getting for your money, the room for adulteration shrinks dramatically.
