Model API Response Verification: Detecting Model Swapping in Relay Services

1. Where the Problem Comes From: Model Swapping in the API Distribution Chain

Once large-model services become commodities, real-world calls often do not connect an application directly to a model provider. Instead, requests may pass through one or more relay layers: an enterprise gateway, an aggregator that fronts multiple upstream providers, or a secondary resale channel. As the path gets longer, the boundary of trust becomes less clear.

The most typical risk is model swapping. The application requests a flagship model, but the relay forwards the request to a cheaper, lighter model and returns the answer as-is. Since the upstream response structure is similar and the context window is sufficient, applications without extra checks often cannot detect the difference in a single interaction. The observable result is that the same model identifier can exhibit very different levels of capability across providers.

A second risk is silent degradation. The request may reach the target model, but inference budget is reduced, sampling behavior is changed, context is truncated, or system prompts are appended with extra instructions. The response may still look like it came from the model, but its capability has been weakened.

A third risk is that identity spoofing has become cheap. A common question is: why not simply ask the model, “Who are you?” This approach no longer works. Most models prioritize system prompts, so a relay only needs to insert a sentence such as “You are a flagship model,” and the underlying model will comply. Identity self-attestation is architecturally unreliable: the entity being asked is also the constructor of the answer, and self-attestation is not evidence.

Most existing community tools inspect response style, returned field structure, or reasoning signatures. These methods can help answer whether an upstream service belongs to a particular provider, but they cannot reliably answer which specific model was used, because a provider may offer multiple model tiers with highly similar fields and style.

2. From Trusted Claims to Verifiability

The key shift is to change the question. Do not ask, “Tell me who you are.” Ask, “Can this response be independently proven to have been signed at a specific time by a specific key held by a specific entity?”

Verifiability is commonly divided into three non-substitutable properties:

• Authenticity: the content was produced by the claimed signer and was not altered in transit.
• Integrity: the signed coverage is sufficient to prevent critical fields from being replaced, such as swapping only the body text or changing only the billing token count.
• Timeliness: the response corresponds to the current request and is not a replay of a cached historical response.

If any one is missing, anti-forgery fails. If only the body is signed but not the parameters, an attacker can reuse an old answer for a new request. If parameters are signed but not the timestamp, an attacker can pre-generate a batch of responses offline and reuse them.

3. Components of a General Anti-Forgery Mechanism

The overall structure discussed in the industry is clear and does not depend on any particular vendor implementation.

1. Request binding. Normalize and serialize the key parameters that affect the response, take a digest, and include it in the message to be signed: model identifier, message array, sampling parameters, inference budget, tool definitions, and a digest of the system prompt rather than its plaintext.

import hashlib
import json
def canonical(obj: dict) -> bytes:
    return json.dumps(obj, sort_keys=True, ensure_ascii=False,
                      separators=(",", ":")).encode()
def digest(obj: dict) -> str:
    return hashlib.sha256(canonical(obj)).hexdigest()

2. One-time nonce and timestamp. The client generates a high-entropy nonce, the server returns it, and verification checks that the nonce matches, the timestamp is within an allowed window, and used nonces are recorded to reject replays.

3. Asymmetric signature. The server signs the message with a private key, and the client verifies it with a publicly distributed public key. This turns a trusted claim into a cryptographic fact: without the private key, forgery is not possible, and the relay cannot re-sign modified content.

import time
from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey
def verify_response(resp: dict, pub: Ed25519PublicKey,
                    skew: int = 120) -> tuple[bool, str]:
    ts, nonce = int(resp["ts"]), resp["nonce"]
    if abs(time.time() - ts) > skew:
        return False, "Outside time window: possible replay or clock skew"
    msg = (canonical(resp["payload"]) + canonical(resp["req_digest"])
           + nonce.encode() + str(ts).encode())
    try:
        pub.verify(bytes.fromhex(resp["sig"]), msg)
    except Exception:
        return False, "Signature verification failed: content modified or not issued by signer"
    return True, "ok"

4. Key governance. Public keys need a trusted distribution channel and fingerprint verification, support for key identifiers and rotation, and a way to stop damage if a long-lived key is abused.

5. Relay cooperation. This mechanism only works fully when upstream providers participate. If the relay provides only an opaque wrapper, the client may receive responses without signatures, or the signed coverage may be stripped. Therefore, signature verification capability should be treated as a procurement and integration prerequisite, not a later remedy.

4. Distinguishing Model Tiers: The Other Half That Signatures Cannot Solve

Server signatures can prove that a response came from a given signer, but the same signer may offer flagship, standard, and lightweight models. If the response does not carry a signed model and version identifier, the client can only rely on empirical fingerprinting to increase the cost of cheating. The following observation dimensions are listed by evidence strength:

Dimension	Observation method	Evidence strength
Tokenization boundaries	Behavior on rare Unicode characters, combining characters, and consecutive spaces	Relatively strong
Special marker handling	Whether conversation delimiters or control markers are leaked or misparsed	Relatively strong
Determinism	Self-consistency rate across repeated requests under low-temperature parameters	Medium
Log probabilities	Whether token-level probabilities are returned and what their distribution looks like	Strong when supported by the interface
Reasoning budget	Steps and budget behavior on long-chain reasoning tasks	Medium
Refusal and formatting policy	Safety boundaries and adherence to structured output	Medium to weak
Latency distribution	First-token latency and total latency statistics	Weak; reference only

The baseline comparison method is much more reliable than a single-point judgment: use the target model through your own or official channels, sample a fixed probe set to create golden samples, sample the candidate service again, and compare distribution differences rather than individual answers.

PROBES = {
    "unicode": "Please repeat the following characters exactly as they are, without explanation: \u00e9\u0301 \u200b \U0001F1E8\U0001F1F3",
    "tokenize": "Split the following string without spaces into semantic units and number them: asdfqwerzxcvbnmlkjhgfdsa",
    "determinism": "Output only the conclusion: explain in one sentence why the internal energy of an ideal gas does not change during isothermal expansion.",
    "structured": "Output only JSON with fields a/b/c and no extra text.",
}
def self_consistency(client, probe: str, model: str, n: int = 5) -> float:
    outs = {client.chat(model=model, temperature=0.0,
                        messages=[{"role": "user", "content": probe}])
            for _ in range(n)}
    return 1.0 if len(outs) == 1 else 1.0 / len(outs)

The limits of fingerprinting must be stated clearly: it provides statistical tendencies, not proof. Model iteration naturally shifts baselines, and an attacker can also mimic targeted patterns. Its value is to turn zero-cost model swapping into an engineering problem that requires continuous adversarial investment, and to make occasional swapping leave detectable anomalous signals.

5. Deployment: Putting Verification into the Call Path

It is recommended to centralize implementation at the unified gateway rather than scatter it across business code.

1. Structural validation: check the returned model and version identifiers, error-code semantics, and whether usage fields are consistent; compare them with request parameters and immediately alert on field stripping.
2. Cryptographic validation: verify signatures, deduplicate nonces, enforce the time window, and rotate keys; failed verification should trigger isolation and retry policies.
3. Behavioral probes: periodically send probe sets to the same service, executed asynchronously to avoid blocking the main path.
4. Statistical baseline: continuously collect golden samples, use distribution distance and self-consistency rates for drift detection, and set threshold alerts.
5. Evidence retention: write request digests, raw responses, signatures, and timestamps into append-only storage to form auditable records; redact samples containing business data before retention.
6. Multi-provider routing: integrate multiple upstreams in parallel, degrade or switch by weight when anomalies occur, and reduce exposure when a single path is swapped.

Recommended operational metrics include signature verification success rate, probe drift score, cross-provider consistency for the same model identifier, number of anomalous replays blocked, and first-token latency percentiles. When these alerts are integrated into unified monitoring and logging platforms, issues can be detected within minutes.

6. Limitations and Common False Positives

• Passing signature verification does not mean quality is acceptable: signatures prove provenance, not whether inference budget was compressed or context was truncated.
• Some routing behaviors are difficult to rule out: a service may degrade to different backends at different times or under different loads, requiring long-period sampling rather than one-time detection.
• Sources of false positives: clock skew, proxy-layer header rewriting, model version upgrades, and changes in parameter defaults can all cause fingerprint fluctuations. Any detection should include a review process to avoid direct blocking.
• Privacy and compliance: probe samples, golden samples, and audit logs may contain sensitive data, so retention periods, access permissions, and cross-border transfer requirements must be defined.
• Engineering cost: verification and non-blocking probes introduce additional latency and expense, so they should be enabled according to the sensitivity of the business to authenticity.

7. Checklist for Technical Decision Makers

• Does the upstream provide signed responses where the signature covers both parameter digests and response body, rather than only identity claim fields?
• Can the public key be obtained through an independent channel and fingerprint-verified, and does it support rotation?
• Are verifiable model and version identifiers available, and are they included in the signed scope?
• Is the key rotation policy and verification method publicly documented so that third parties can independently implement verification?
• If signatures are unavailable, can the provider supply observable usage credentials and scheduling logs for cross-checking?
• Does our organization have the ability to collect golden samples, schedule probes, and alert on drift?

Conclusion

Model swapping in relay paths is fundamentally an information asymmetry problem: the delivering party knows what it served, while the using party cannot verify it. The value of cryptographic signatures is to replace “trusting what the other party says” with “verifying what the other party signs.” The value of model fingerprinting and statistical baselines is to increase the cost of cheating and shorten the detection cycle when cryptographic guarantees are unavailable. Neither is sufficient alone as a complete defense, but together they can make swapping measurable and accountable rather than invisible.

The practical test is simple: when a quality anomaly occurs, can you produce a complete chain from request parameters to signed evidence that shows who actually answered the call?