Model Authenticity Verification and Behavioral Fingerprinting for Relay APIs

1. Problem Background: The Trust Boundary of Relay APIs

In a model service chain, a relay API typically handles unified authentication, billing, routing, caching, content sanitization, and protocol adaptation. For callers, it hides upstream differences and simplifies integration. At the same time, it introduces a new trust boundary: the response a caller sees may pass through multiple gateways, serialization layers, template post-processing, or dynamic routing, and may not exactly match the claimed model version.

Therefore, the question of whether an API response comes from the claimed version cannot be settled by marketing pages, response latency, or a single Q&A exchange. A more rigorous approach is to build a reproducible verification system: use a trusted direct baseline as a reference, collect evidence from protocol consistency, character-set fidelity, behavioral fingerprints, multi-channel reproduction, and statistical distributions, and then assign a risk level. A single observed symptom is only a clue, not a conclusion by itself.

2. Verification Principles: From Single Assertions to an Evidence Chain

First, the verification goal should be framed as “is the claimed version credible?” rather than “does some absolute rule hold?” For example, Chinese double quotation marks being replaced by English double quotation marks, mojibake in Japanese names, or failures in format constraints may come from model generation deviation, gateway escaping, character-set conversion, template sanitization, or front-end rendering. Only by explaining the symptom within the full chain can false conclusions be avoided.

Second, verification should be multi-baseline, multi-channel, and multi-round. At minimum, prepare a trusted direct baseline and collect multiple rounds of results for the same test set. At the same time, compare raw bytes from a direct text channel and a file-generation channel. If an anomaly appears only in one channel, investigate post-processing first. If it reproduces stably across channels, then evaluate model generation or upstream service differences.

Third, distinguish “auxiliary fingerprints” from “sufficient proof.” Character-level fidelity, completeness of name lists, transliteration format, length constraints, and format adherence can together form a behavioral fingerprint with relatively high discriminative power. However, they still need to be combined with protocol metadata, statistical significance, and version regression tests for a comprehensive judgment.

3. Key Verification Dimensions

3.1 Protocol and Metadata Consistency

First, check whether the response structure, character-set declaration, escaping levels, error-code semantics, streaming chunk boundaries, and multi-turn context behavior match expectations. If response metadata clearly conflicts with the claimed capability, record it as a suspicious signal. However, metadata can be rewritten by proxies, so it can only serve as an entry-point check, not as the sole evidence.

3.2 Character-Set and Punctuation Fidelity

Character-level tests are useful for detecting encoding-chain problems. A short-text task can be designed to require preservation of the Chinese double quotation marks U+201C and U+201D, while also outputting multiple personal names and their Latin-alphabet transliterations. This case imposes four constraints at once: entity count, transliteration format, punctuation characters, and length limit. Its discriminative power is higher than ordinary chitchat. If Chinese double quotation marks are replaced by English double quotation marks U+0022, continue checking whether the cause is JSON escaping, template normalization, or the rendering layer. If both the direct text channel and the file-generation channel stably show the anomaly, it can be used as part of the behavioral fingerprint.

3.3 Behavioral Fingerprints and Version Regression

Behavioral fingerprints should cover dimensions such as format constraints, long-context retrieval, mixed-language input, refusal boundaries, and structured-output adherence. After a version upgrade, some character-handling issues may be fixed, but other fingerprints may remain. Therefore, do not rely on a single historical symptom. Maintain a regression test set and record distribution changes for each version.

3.4 Multi-Channel and Post-Processing Identification

It is recommended to compare at least a direct text channel and a file-generation channel. If the direct channel is abnormal while the file channel is normal, the cause may be a serialization or rendering-layer difference. If both are abnormal, the cause may lie in generation or an upstream post-processing chain. Simply attributing the anomaly to the client is insufficient; locate it through channel comparison and raw-byte comparison.

3.5 Statistical Significance and Baselines

Run each case for multiple rounds and record metrics such as the number of Chinese double quotation marks, the number of English double quotation marks, the number of replacement characters U+FFFD, length, and format-constraint pass rate. Compare distributions with a trusted direct baseline rather than comparing single samples. If the difference is stable and reproducible, then raise the risk level.

4. Reproducible Test Set and Pseudocode

The test prompt can be designed as: “Generate a short text of no more than 300 characters: the protagonist arrives at a university and meets ten fictional women; provide each person’s name and its Latin-alphabet transliteration, wrap the title in Chinese double quotation marks; U+201C/U+201D must be preserved and must not be replaced with English double quotation marks.” This case does not involve real personal information, while checking quantity, transliteration, punctuation, and length.

CASES = [{
  id: "quote_name_10",
  prompt: "Generate a short text of no more than 300 characters: the protagonist arrives at a university and meets ten fictional women; provide each person's name and its Latin-alphabet transliteration, wrap the title in Chinese double quotation marks; U+201C/U+201D must be preserved and must not output English double quotation marks.",
  checks: ["cjk_quote_count>=2", "replacement_char==0", "length<=300"]
}]
def collect(endpoint, case, rounds=8):
    rows = []
    for i in range(rounds):
        resp = endpoint.generate(prompt=case.prompt, stream=False)
        text = resp.text
        rows.append({
            id: case.id,
            round: i,
            cjk_quotes: text.count("\u201c") + text.count("\u201d"),
            ascii_quotes: text.count('"'),
            replacement: text.count("\ufffd"),
            length: len(text),
            head: text[:120]
        })
    return rows

After collection, save both raw bytes and parsed text to avoid comparing results that have been corrected by a renderer. For symptoms such as mojibake in Japanese names, check replacement characters, abnormal question marks, and typical mojibake patterns, but also distinguish encoding conversion from model generation.

5. Interpretation Matrix and Practical Workflow

Symptom	Possible cause	Recommended action
Chinese double quotation marks consistently become English double quotation marks	Generation deviation, gateway normalization, serialization escaping	Reproduce across channels and inspect raw bytes
Mojibake in multilingual names	Character-set conversion, incorrect encoding declaration, post-processing	Compare against the direct baseline and inspect encoding along the chain
Intermittent format-constraint failures	Sampling variance, context interference	Increase rounds and compare pass-rate distributions
Inconsistent results across rounds	Routing, caching, dynamic degradation	Record request distribution and run statistical tests
Metadata conflicts with behavior	Proxy rewriting or upstream differences	Raise the risk level and expand the test set

A practical workflow is: build a trusted baseline library; design a multi-constraint test set; collect across channels and rounds; calculate metric distributions; locate post-processing factors; form a risk conclusion; and run periodic regression. The verification process should comply with service terms, not bypass safety policies, not collect sensitive data, and not perform offensive probing.

6. Conclusion

Model authenticity verification for relay APIs is essentially an assessment of chain trustworthiness. Combined tests such as Chinese double-quotation-mark fidelity, Japanese name encoding, name count, and transliteration format can serve as high-discrimination auxiliary fingerprints, but they must be combined with a direct baseline, multi-channel reproduction, and statistical interpretation. Upgrading from the absolute rule “a certain symptom definitely proves authenticity or inauthenticity” to an evidence chain of “symptom—chain—baseline—statistics” enables more reliable identification of degradation, substitution, or post-processing risks and provides an explainable basis for model selection and online governance.