How to Detect Token Padding and Model Substitution in API Relay Services

1. Background: The Opaque Relay Layer

As demand for large-model API calls grows, a type of intermediary service between official APIs and end users has become more common. These services aggregate multiple upstream channels and expose them at lower unit prices and through a more unified access method. The proxy gateway itself is a neutral technical form: enterprise internal gateways, billing proxies, and unified routing layers all belong to the same category.

The problem arises precisely because users often cannot see the real upstream call chain. When users can only access models indirectly through an intermediary layer, two types of quality loss may occur:

• Token padding: The actual amount of computation used is far lower than the amount deducted on the bill, and the numbers shown to the user are inflated.
• Model watering / substitution: Requests are silently forwarded to a cheaper, weaker model, while still being billed as the higher-end model the user purchased.

This article does not discuss any specific provider. Instead, from a general engineering perspective, it provides a reproducible and measurable method for identifying and verifying these issues.

2. Technical Nature of the Two Problems

Before discussing detection methods, it is useful to clarify the causes so that the problems can be addressed more effectively.

Common token padding methods include:

• Exploiting differences between tokenizers of different models: billing according to a tokenizer that produces a higher token count, while actually forwarding the request to a lower-cost upstream.
• Injecting hidden content or padding redundant characters into responses to artificially increase the number of output tokens.

The motivation for model watering is cost arbitrage. The per-call cost of a high-end model and a lightweight model may differ by several times. If the intermediary secretly downgrades the model, it can reduce upstream spending while maintaining the original selling price. Because ordinary users usually lack a reliable comparison baseline, such substitution is often difficult to notice.

Once the underlying causes are clear, the detection approach becomes straightforward: bypass the intermediary’s self-reported claims and use independent, verifiable measurements to cross-check the service.

3. Verify Token Billing Consistency

The most direct starting point is to reconcile the numbers on the bill with the actual token counts. The key is to implement an independent token counting process rather than trusting the usage field returned by the server.

import os
import requests
ENDPOINT = os.environ["RELAY_ENDPOINT"]  # Relay service endpoint
API_KEY = os.environ["API_KEY"]          # API key
def call_and_inspect(prompt: str):
    resp = requests.post(
        f"{ENDPOINT}/v1/chat/completions",
        headers={"Authorization": f"Bearer {API_KEY}"},
        json={
            "model": "some-model-id",
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0,
        },
        timeout=30,
    ).json()
    usage = resp.get("usage", {})
    reported = usage.get("total_tokens")  # Consumption claimed by the server
    text = resp["choices"][0]["message"]["content"]
    return reported, text

After obtaining the returned text, use a public and commonly available tokenization algorithm, such as the official tokenizer for the corresponding model or a general BPE implementation, to locally recalculate the input and output token counts. Then compare the local result with the reported value. If the reported value is consistently and significantly higher than the local estimate, the billing measurement may be inflated.

When doing this, pay attention to the following points:

• Use the tokenizer that corresponds to the model you actually purchased, rather than applying the wrong tokenizer to the wrong model.
• A single deviation may be caused by encoding details. Multiple samples should be collected to observe statistical trends instead of drawing conclusions from a single request.
• Pay special attention to whether input token billing is abnormally high, since this is a common area for padding.

4. Probe Output Quality and Capability Fingerprints

Billing reconciliation can help detect “fake numbers,” but it cannot directly detect model substitution. The key to detecting model substitution is to establish a comparable quality baseline.

A recommended practice is to sample the official model or a trusted baseline once during the initial integration period using a “golden test set,” and record its output characteristics. Then periodically use the same set of questions to probe the relay service and compare the results. The test set should cover dimensions that can reveal differences between model tiers:

• Complex multi-step reasoning: Math or coding problems that require a rigorous chain of logic. Weaker models often fail midway through the reasoning process.
• Long-context recall: Construct “needle in a haystack” tasks by burying key information in the middle of a very long context containing tens of thousands of tokens, and check whether the real context window meets the expected requirement. If the service silently switches to a shorter-context model, such questions will usually fail.
• Instruction following: Multi-constraint formatted output tasks that test how well the model follows boundary conditions.
• Capability fingerprints: Different models have stable differences in refusal style, domain knowledge, knowledge cutoff dates, and vocabulary habits. Several “fingerprint probe” questions can be designed to track whether the response style drifts over time.

GOLDEN_SET = [
    {"q": "Hide a key in a long text, then ask what it is", "expect": "correct recall"},
    {"q": "A word problem requiring five-step reasoning", "expect": "complete reasoning chain"},
]
def probe_quality():
    for case in GOLDEN_SET:
        reported, text = call_and_inspect(case["q"])
        # Add human or automated scoring here, and record whether text satisfies expect
        yield case["q"], text

5. Analyze Latency and Throughput Characteristics

Models of different sizes have relatively stable physical characteristics in terms of response latency and generation speed. Lightweight models usually have faster first-byte response and generate more tokens per second. Higher-end models usually behave in the opposite way. Although a relay layer introduces additional network overhead, these relative characteristics still have reference value.

It is recommended to measure two metrics under fixed network conditions and fixed prompts:

• Time to First Token (TTFT)
• Generation throughput, measured in tokens per second

Samples should be collected separately under long, medium, and short input lengths. If you purchased a higher-end model but consistently observe unusually fast and low-latency response characteristics that do not match the expected behavior of that model, further confirmation should be done using the quality probing methods described earlier.

6. Cross-Check Statistical Distribution and Determinism

This section adds another layer of cross-validation. Set the temperature parameter to 0 to request deterministic output, and send the same prompt multiple times:

• If the same input still produces high-variance or divergent results under temperature=0, the upstream may be randomly routing requests to multiple different models.
• Comparing the “deterministic output” with the stable output of a baseline model can help determine whether the actual upstream model is consistent.

Combined with reproducible experiments using fixed seeds, this method can further help identify whether cross-model dynamic scheduling is occurring.

7. Practical Recommendations

These methods should be engineered into a routine inspection mechanism:

1. Build a baseline library: During initial integration, sample trusted sources and solidify the golden test set as well as the feature fingerprints of each model.
2. Perform periodic reconciliation: Turn token billing checks, quality probes, and latency measurements into recurring tasks, and record trend curves over time.
3. Set alert thresholds: Define reasonable fluctuation ranges for each metric. When a metric exceeds the threshold, trigger an alert instead of waiting for user complaints.
4. Maintain multi-provider redundancy: For critical workloads, keep multiple switchable channels to avoid being locked in by a single intermediary.
5. Use contract and SLA constraints: Include quality metrics in service terms and use verifiable data as the basis for accountability.

Overall, detecting token padding and model substitution is essentially about using independent measurements to counter information opacity. Rather than trusting the intermediary’s self-reported claims, developers and teams should judge the real service quality through multi-dimensional cross-verification, including tokenization reconciliation, quality baselines, latency characteristics, and statistical distributions. By turning this detection capability into a routine quality safeguard, teams can access models at lower cost while maintaining both reliability and cost control.