Detecting Token Inflation in LLM Relay Services: A Systematic Auditing Method

When using an LLM relay service, callers typically can see only the request entry point and the final response. They cannot directly confirm which model received the request, whether the provider modified the prompt, duplicated context, or even artificially increased token counts in billing statistics. The term “token inflation” generally refers to token measurements in the service chain that do not match the user’s actual input, or to cost amplification through appended hidden content, repeated context, fabricated usage, or similar mechanisms.

It is important to note that token inflation is not necessarily malicious. Some relay layers may automatically add system prompts, tool descriptions, historical conversation, format constraints, or safety policies. All of these can genuinely increase input tokens. The key to professional judgment is not to draw conclusions from a single high usage reading, but to distinguish “explainable context growth” from “abnormal growth that cannot be derived from the request content.”

1. Establish a Token Measurement Baseline

The first step in identifying anomalies is to build an independent baseline rather than simply trusting the usage fields returned by the relay service.

Prepare a set of fixed test requests covering the following cases:

1. Empty content or very short text;
2. Ordinary text with gradually increasing length;
3. Mixed-language, numeric, punctuation, and special-character text;
4. The same content sent repeatedly;
5. Requests that change only spaces, line breaks, and message roles.

For each request, record the following information:

• The actual number of characters and bytes sent;
• The number of messages and their role distribution;
• System instructions, user content, and historical context in the request;
• Input tokens, output tokens, and total tokens returned by the service;
• Request latency, status code, and response length;
• Whether multi-turn conversation, tool calling, or streaming output is enabled.

Then use a public tokenization rule that matches the target model, or implement an approximate local counter, to calculate the theoretical input size. Because different models tokenize text differently, local results may not exactly match the server side. However, under the same model and encoding rules, the growth trend is usually informative.

For example, construct text of length (n) and let (n) increase by a fixed step. If the actual input tokens increase smoothly with text size, the measurement logic is broadly normal. If the user content adds only a few characters while server-side tokens suddenly multiply, further investigation into hidden context or billing definitions is needed.

2. Observe Anomalous Reactions Through Low-Frequency Inputs

The “dirty-token method” mentioned in the original approach can be abstracted as low-frequency tokenizer-unit probing. Different models use different vocabularies and tokenization boundaries. Certain rare character combinations, uncommon symbol sequences, or mixed-encoding content may cause different model reactions.

However, such tests should not rely on large amounts of unverified gibberish, and a single refusal or anomalous output should not be treated as confirmation of model identity. A more reliable approach is to design controlled inputs:

• Rare but valid character combinations;
• Short fragments from different languages;
• Combinations of numbers, punctuation, and whitespace;
• Strings of equal length but different character distributions;
• Requests with the same semantics but different surface forms.

Repeat each input category multiple times and observe the following indicators:

• Whether the task is understood normally;
• Whether irrelevant content appears;
• Whether refusals occur frequently;
• Whether garbled text, repetition loops, or format失控 appear;
• Whether first-token latency and overall response time change significantly;
• Whether response style, structure, and error patterns remain stable.

The value of this method lies in forming “reaction signatures,” not in finding a single magic string. A reliable judgment should come from differences that persist across multiple input groups. For example, if one model shows a stable combination of characteristics in rare-character handling, numeric reasoning, and format following, while another model’s response pattern is clearly different, this can serve as supporting evidence for routing identification.

3. Conduct Knowledge-Boundary Probes

Models differ in training-data scope, knowledge update time, and knowledge organization. Knowledge-boundary tests can therefore help assess the actual routing.

Use public, low-risk, verifiable questions where possible, and avoid personal privacy or sensitive information. Three categories of questions can be designed:

1. Time-Boundary Questions

Ask about public facts before and after a specific point in time, and compare whether the model can distinguish the information scope. Such tests cannot rely simply on “knows” or “does not know,” because the model may obtain answers through reasoning, context, or external tools.

2. Domain-Specific Detail Questions

Select foundational and boundary concepts from multiple domains, and observe whether the answers show a stable terminology system, definition style, and reasoning habits. The focus is not on judging whether an answer is good or bad, but on identifying whether its expressive structure remains consistent over time.

3. Restricted-Information Questions

Design questions whose answers can be verified through public materials but do not belong to common Q&A templates. Ask the model to state uncertainty, provide reasoning grounds, and check whether it remains consistent across multiple rounds.

Knowledge boundaries can only serve as supporting evidence. A relay service may add retrieval, caching, or custom prompts, and models may hallucinate. Knowledge probes must therefore be analyzed together with token measurement, response style, and stability data.

4. Check the Stability of Repeated Requests

If the relay layer dynamically modifies content before or after requests, or switches models based on load, identical requests may show noticeable variation. Stability checks should focus on four areas.

First, whether input tokens are stable. For exactly the same request, with historical conversation and tool capabilities disabled, input tokens across multiple calls should not usually fluctuate widely without a pattern. If they do, confirm whether dynamic system prompts, implicit context, or measurement errors exist.

Second, whether output length is stable. Randomness can cause answer differences, but if model configuration remains consistent, output length usually should not fluctuate extremely without an obvious reason. Pay particular attention to large repeated passages, meaningless expansion, and additional content unrelated to the question.

Third, whether latency is stable. Different routes, models, and loads produce different latency. Record time to first byte, time to first token, and full response time, and observe whether anomalous inputs trigger additional processing chains.

Fourth, whether errors are stable. Genuine model characteristics often appear as repeatable error types, such as consistent format failure, weaker understanding of a specific language, or continuous refusal on a certain task type. If errors are completely random, they may be more related to network, rate limiting, or service load.

5. Build Behavioral Fingerprints Instead of Relying on a Single Phenomenon

A behavioral fingerprint is a composite description of model response characteristics. It may include:

• Instruction-following order;
• Degree of compliance with format constraints;
• Information retention in long text;
• Handling of numbers and symbols;
• Formatting habits for code or structured text;
• Refusal boundaries and explanation style;
• Clarification strategy for ambiguous questions;
• Output-length control capability;
• Context-memory performance in multi-turn conversations.

To improve accuracy, define quantifiable metrics for each test request. For example, format tasks can measure field completeness; summarization tasks can measure coverage of key facts; repeated tests can calculate output similarity; long-context tasks can check recall of specified information.

Then combine multiple metrics into a test matrix instead of relying on subjective judgments such as “looks like a certain model.” The matrix should include at least simple tasks, complex tasks, long-context tasks, and anomalous-input tasks. Only when multiple dimensions point to the same conclusion is it appropriate to consider the model-identity evidence strong.

6. How to Identify True Token Inflation

To judge token inflation, the following conditions should generally be met at the same time:

1. The user-visible request content has not changed significantly;
2. A persistent deviation exists between the local baseline and server-side measurement;
3. The deviation cannot be explained by system prompts, historical messages, tool definitions, or safety policies;
4. The deviation recurs across multiple requests and multiple length levels;
5. The returned usage fields do not match response length or context size;
6. The anomalous growth shows a fixed pattern across different request types.

A difference value can be used for analysis:

[
D = T{service} - T{estimated}
]

Here, (T{service}) is the token count reported by the service, and (T{estimated}) is the independent estimate. Do not look only at a single difference; observe how the difference changes with input length. If (D) is roughly constant, it may indicate a fixed system prompt. If (D) increases linearly with the number of history turns, conversation context may be repeatedly carried. If (D) suddenly expands under specific request types, there may be tool descriptions, hidden templates, or routing-policy changes.

7. Common Misjudgments and Improvement Suggestions

First, different tokenizers cause normal errors. Character count, byte count, and token count are not the same concept and cannot be converted with a fixed ratio.

Second, streaming responses may use different statistical definitions. Some services aggregate statistics after generation is complete, while others count in segments during streaming. Display differences may exist between the two.

Third, automatically added system content may be a reasonable configuration. Only when the provider cannot explain its source and the deviation is long-term, repeated, and unexplainable should token inflation be investigated further.

Finally, model-identity identification is not absolutely reliable. Model updates, quantization methods, sampling parameters, caching strategies, and safety layers can all change behavior. Test reports should use cautious language such as “high-probability judgment” or “signs of routing changes,” rather than definitive conclusions unsupported by evidence.

8. Recommended Audit Workflow

A complete audit can be performed in the following order:

1. Fix model parameters, session state, and request templates;
2. Archive input content locally and calculate independent token estimates;
3. Run length-gradient, character-gradient, and multi-turn context tests;
4. Record server-side usage, latency, errors, and response bodies;
5. Conduct knowledge-boundary and behavioral-fingerprint tests;
6. Repeat all tests across multiple rounds to exclude randomness;
7. Analyze differences, trends, and anomaly points;
8. Ask the provider to explain hidden context and billing definitions;
9. Save request summaries, timestamps, and response evidence;
10. Before confirming a problem, first rule out tokenizer differences and configuration changes.

With this approach, developers can not only identify suspected token inflation but also determine whether the anomaly comes from hidden prompts, context duplication, routing switches, inconsistent statistical definitions, or actual measurement problems on the service side. Compared with simply sending a few special strings, an audit method based on baselines, repeated experiments, and multidimensional behavioral evidence is more robust and better suited to enterprise procurement, cost accounting, and long-term operations.