LLM Response Anti-Fraud: Designing and Implementing Verifiable Provenance Mechanisms

1. The Problem: Response Authenticity Has Traditionally Relied on Trust

Developers who use model API relay services, proxies, or reseller endpoints often encounter a familiar issue: the documentation and billing catalog may list a flagship model, but the quality of the responses can vary significantly. A model may perform well on complex reasoning tasks and then fail on simple questions. This variability can come from sampling randomness, context length, workload conditions, or other factors, but it may also have another cause.

1.1 Typical Problems in Relay Paths

• Model substitution: The caller pays for a flagship model, but the gateway forwards the request to a lower-tier model from the same vendor, or even to a model from another vendor, and returns the result as if it came from the original model.
• Implicit downgrade: For cost control or rate limiting, a service may reduce the maximum output length, limit the reasoning budget, or silently switch to a lighter model on the backend. If the provider does not disclose this, users may have no practical way to detect it.
• Impersonation: A distilled or imitation model may be presented as the original model.
• Content tampering: A response may be modified, truncated, or augmented with extra content while passing through an intermediary.

1.2 Why Existing Detection Methods Are Insufficient

A common approach is to ask the model directly: “Who are you?” However, an attacker can simply add a sentence such as “You are a certain flagship model” in the system prompt, and most models will answer accordingly. Identity self-description therefore loses its evidentiary value.

Community detection methods for relay services usually rely on indirect inference:

• Comparing response style, common phrasing, and refusal patterns;
• Checking the model name field in the response JSON;
• Verifying signatures attached to extended thinking content to determine whether the content was produced by the upstream service.

These methods share a common limitation: they try to infer the model identity from the response itself. Style can be imitated, fields can be forged, and thinking-content signatures can only prove that the response came from a service of that vendor; they cannot distinguish which specific model tier produced it.

Model providers face a similar dilemma. When users complain that “the model has become less capable,” the provider may struggle to produce persuasive counterevidence even if it did not change the model. Ultimately, without verifiable evidence, the authenticity of a response depends on trust.

2. Core Idea: Turn Trust into Verifiable Evidence

The essence of LLM response anti-fraud is response provenance. A verifiable provenance mechanism provides evidence that can be checked cryptographically and proves three things:

• Identity: Which service, model, and weight version produced the response;
• Integrity: Whether the response was modified between generation and delivery to the caller;
• Point in time: When the response was generated and which request it corresponds to.

The design is based on three principles:

• The signer only signs; it does not route: Signatures can only be generated by the model service provider that actually runs the model. Relay or gateway services that only forward requests do not have access to the signing key.
• The verifier trusts cryptographic evidence, not self-description: Model names and version numbers should be included in the signed payload rather than inferred from a statement inside the response.
• Evidence must be retainable and auditable: Verification results can be written to audit logs for later accountability and dispute resolution.

This approach draws on established ideas from software supply chains and web certificate systems: generate a digest for an artifact, sign the digest, distribute the signature together with the artifact, and let anyone verify it independently using a public key.

The relevant cryptographic tools are straightforward:

What needs to be proven	Tool	Role
Whether the content was modified	Cryptographic hash function	Generate a fixed-length digest for a response of arbitrary length; changing even one bit produces a completely different digest
Who generated it	Asymmetric digital signature	Sign with a private key and verify with a public key; the public key can be distributed openly
Which request it corresponds to	Request fingerprint + nonce	Include request characteristics and a one-time random value in the signature to prevent replay
Whether it is the current version	Version information in the signed payload	Bind model identity and version, and support key rotation and revocation

2.1 What the Signed Payload Should Contain

The signed payload, meaning the structured metadata that is signed, should at least include:

• A unique response ID and parent request ID, for correlation with tracing systems;
• Service identifier, model identifier, and version number;
• A normalized request fingerprint, such as a digest of the model name, message content, sampling parameters, and optionally the system prompt;
• A digest of the complete output content;
• A generation timestamp and a caller-provided nonce;
• A key ID. A provider may use multiple signing keys in parallel, for example different tiers or different batches. With a key ID, the caller knows which public key to use for verification, and keys can be revoked precisely after upgrades or key exposure.

One detail deserves emphasis: the signature covers metadata and the content digest, not the full response text. Therefore, the signature size is independent of the response length. Signing services such as cryptographic modules or hardware security modules can process many thousands of signatures per second, with little impact on API throughput.

2.2 How Hash Functions Contribute

A hash function acts like a digital fingerprint for content. Any input of arbitrary length produces a fixed-length digest. If even one bit changes, the avalanche effect makes the digest completely different. Hash functions are also one-way: the original content cannot be reconstructed from the digest.

The signature covers metadata that is only tens or hundreds of bytes, while content integrity is enforced by the content digest. Together, they provide complete protection. This leads to an important conclusion: an attacker can preserve the provider’s signature, but cannot replace the content with their own, because the digest will no longer match.

2.3 Hash-Chain Commitment for Streaming Output

Most mainstream LLM APIs use streaming output. If the signature is placed only in the final data chunk, two problems arise:

• Users may not receive the signature if they close the page midway;
• A complete response may be long, and verifying it only after the entire response is generated may not be practical.

A better approach borrows the hash-chain idea from blockchain technology. For each output increment, the service can attach a cumulative digest of all content emitted so far:

h0 = H("")                     // H is a cryptographic hash function
hi = H(h(i-1) || delta_i)      // delta_i is the i-th output increment

When the client receives each chunk, it only needs to perform one hash computation to confirm that the received content has not been modified. When the entire response is complete, the service signs the final cumulative digest together with the metadata. This enables both mid-stream anti-tampering checks and final verification.

3. Overall Architecture: Generation, Distribution, and Verification

[Caller Application]
   │ ① Request with nonce             ▲ ④ Response + provenance credential
   ▼                                  │
[Relay Gateway / Proxy] ──Forward──► [Model Service Provider (signing private key)]
                                     │ Run model → generate response
                                     │ Compute digest → sign → attach credential
                                     ▼
                                [Audit Log / Transparency Log Service]

• Model service provider: Runs the model in a controlled environment and holds the signing private key. The private key is best stored in a hardware security module, cryptographic module, or trusted execution environment. Only a “sign this digest” interface should be exposed; the private key itself should not be exposed.
• Relay gateway or proxy: Only forwards requests and responses and does not hold any signing private key. It may attach its own signature to claim that “this response passed through our service,” but it cannot forge the upstream signature.
• Caller application: Verifies credentials according to preset policies, such as “refuse to display if there is no valid signature” or “alert if the version does not match.”
• Audit and monitoring: Records verification results, sets failure-rate thresholds in log services and monitoring platforms, and sends alerts for anomalies.

4. Key Design: From Forgeable Signatures to Trusted Signatures

4.1 The Signature Must Cover the Complete Input Used to Generate the Response

This is the most easily overlooked vulnerability. If a signature covers only the final output and does not bind it to the input, an attacker can use a very short prompt to “fish” a complete response from the real service, then place that signature and response into the response for an arbitrary question. This is signature replay: using a correct signature with incorrect content.

Therefore, the request fingerprint must cover all inputs actually used by the model service provider for inference:

• Message content;
• Sampling parameters, such as temperature, maximum output length, and reasoning budget;
• Optionally, a digest of the system prompt.

4.2 Nonce: Replay Prevention and Response Freshness

Even if the signature covers the input, an attacker may cache an old signed response pair and return it when a user asks the same question. The defense is for the caller to generate a cryptographic nonce for each request and require that nonce to appear in the signed payload:

import hashlib
import json
import secrets
def fingerprint(request: dict, nonce: str) -> str:
    # Normalize key request fields into JSON and compute a SHA-256 digest
    material = {
        "model": request["model"],
        "messages": request["messages"],
        "sampling": request.get("sampling", {}),
        "system_hash": sha256_hex(request.get("system", "")),
    }
    canonical = json.dumps(material, sort_keys=True,
                           ensure_ascii=False, separators=(",", ":"))
    return "req_" + hashlib.sha256((canonical + "|" + nonce).encode()).hexdigest()

If a gateway removes the nonce or replaces it with its own nonce, the signature no longer covers the original request. The caller can detect this by comparing the locally computed request fingerprint with the request fingerprint in the credential.

4.3 When Relays Add Their Own Signatures: Check Coverage, Not the Name

Some gateways issue signed credentials themselves. The format may look similar to the upstream format, but the signature may come from the gateway’s own key. In this case, do not rely on the credential name. Instead, inspect what the signature actually covers:

• If the gateway claims that the response was “generated by a certain model,” the signature must come from the key of that model service provider;
• If the signature only proves that the gateway forwarded the content, then it is a relay attestation and cannot prove the model identity.

5. Example Implementation: Credential Structure and Client Verification

The following example uses Python-like pseudocode to illustrate a general design. In production, audited standard cryptographic libraries should be used.

# ---------- Server side (model service provider) ----------
def issue_proof(request_fp: str, content: str, model_id: str,
                model_version: str, key_id: str, signing_key) -> dict:
    """Issue an anti-fraud credential for a response.
    The signing key exists only inside a controlled signing environment.
    """
    payload = {
        "rid": new_response_id(),          # Unique response ID
        "req_fp": request_fp,              # Request fingerprint covered by the signature, including nonce
        "model": model_id,                 # Read from the currently loaded model in the signing environment
        "version": model_version,          # Read from the weight version, not from request parameters
        "content_sha256": sha256_hex(content),
        "ts": now_unix_seconds(),
        "kid": key_id,                     # Key ID used by the caller to select the verification public key
    }
    return {
        "v": 1,
        "alg": "ES256-K",                  # Example: elliptic-curve signature algorithm
        "payload": canonical_json(payload),
        "sig": sign(signing_key, canonical_json(payload)),
    }
# ---------- Client side (caller application) ----------
def verify_proof(resp: dict, proof: dict, local_req_fp: str,
                 trust_store, max_skew: int = 300) -> dict:
    """Verify an anti-fraud credential and return the verification result.
    The trust store contains trusted public keys and supports key rotation.
    """
    if proof is None:
        return {"ok": False, "reason": "NO_PROOF"}
    payload = canonical_json(proof["payload"])
    # 1) Verify the signature: find the public key by key ID and reject revoked keys
    pub = trust_store.get(proof["payload"]["kid"])
    if pub is None or pub.revoked:
        return {"ok": False, "reason": "KEY_NOT_TRUSTED"}
    if not verify(pub, payload, proof["sig"]):
        return {"ok": False, "reason": "BAD_SIGNATURE"}
    # 2) Content integrity: the digest must match the signed content
    if sha256_hex(resp["content"]) != proof["payload"]["content_sha256"]:
        return {"ok": False, "reason": "CONTENT_TAMPERED"}
    # 3) Replay protection: the request fingerprint must match the locally computed value, including the current nonce
    if proof["payload"]["req_fp"] != local_req_fp:
        return {"ok": False, "reason": "REQ_MISMATCH_OR_REPLAY"}
    # 4) Freshness: the timestamp must be within the allowed skew
    if abs(now_unix_seconds() - proof["payload"]["ts"]) > max_skew:
        return {"ok": False, "reason": "STALE_PROOF"}
    return {"ok": True, "model": proof["payload"]["model"],
            "version": proof["payload"]["version"]}

These are illustrative snippets and do not correspond to any specific vendor API or parameter set. A production implementation should also consider:

• A degradation policy when verification fails, such as refusing to display the response or only raising an alert;
• How the trust store is obtained and updated, for example by periodically synchronizing public keys through signed model metadata files;
• Key rotation and revocation mechanisms to quickly contain losses after version upgrades or key exposure.

6. Capability Boundaries: What It Can and Cannot Prevent

Many discussions describe this kind of mechanism as making it impossible for relay services to impersonate models. However, a calm analysis shows that its capabilities have clear boundaries.

6.1 Cases It Can Effectively Prevent

• Content tampering: If the content is modified, the digest will not match and verification will fail immediately;
• Forged signatures: If the signer only has forwarding permissions but claims to be the model service provider, verification will fail;
• Cross-service substitution: If another vendor or a distilled imitation attempts to impersonate the original model, a whitelist of signing keys can reject it directly;
• Replay of old responses: Mismatches between the nonce and request fingerprint allow replayed responses to be identified.

6.2 Cases It Cannot Prevent Alone

• Same-vendor downgrade substitution: If the service provider silently routes a flagship request to a cheaper model from the same vendor and signs it with the flagship key, the signature is still “valid.” This requires the provider to act honestly, or requires binding keys to specific model versions.
• Key leakage: If a signing key is leaked, an attacker can issue seemingly valid credentials. This is why signing keys should be placed in hardware and support rotation and revocation.
• Client-side verification abandonment: If the application does not verify credentials, or silently accepts unsigned responses, the mechanism becomes ineffective. Verification should be the default behavior in the client.
• Sampling randomness: The same model can behave very differently under different sampling parameters. A valid signature alone cannot prove that the provider has not secretly changed sampling parameters. If the request fingerprint covers sampling parameters, such changes can be detected, but this does not reveal whether the model itself is “cheaper.”

6.3 Additional Practical Considerations

• The protocol requires cooperation from both upstream and downstream: Both the caller and the model service provider must implement the agreed mechanism. If the upstream does not issue credentials, the system can only fall back to indirect methods such as style comparison and fingerprint identification.
• Forging signatures is not cost-effective for counterfeiters: To package an imitation model with a real credential, the attacker needs a real model signature. If every request must actually pass through the original model, the imitation loses its cost advantage. To issue a fake credential, the attacker must obtain the private key.

Therefore, a more accurate positioning is as follows: the mechanism raises the cost of impersonation from “modifying a line in the system prompt” to “attacking the key management system,” while providing an auditable evidence chain for all participants.

7. Evidence and Dispute Resolution: From Single Verification to Long-Term Audit

A single verification proves that one response is intact. When users suspect that “the model has been replaced” or “downgraded,” historical evidence is also needed.

One practical approach is to treat signed responses retained by users as anchors. The service provider can periodically publish signed statements containing its system configuration and capability version information, such as model version, context length, and default sampling parameters, without including any user data. Users can then verify both the response credential and the corresponding configuration statement for that time. This shows that the response was indeed generated by the declared model and configuration, and that the signing key was valid at that moment.

Token usage details returned by reasoning models provide another useful signal. These details may include input tokens, output tokens, and reasoning tokens, and can be included in the signed payload. If a user finds that the reasoning-token usage is abnormally low and has valid historical credentials, they have objective evidence to appeal to the service provider.

For paid APIs, billing is an important basis for usage accounting. Once the usage accounting chain is trustworthy, mismatches between actual usage and the declared model can be discovered. For example, a response may be billed as coming from a flagship model, while the signed metadata points to a lighter version.

From an engineering perspective, credentials and historical responses can be stored centrally in object storage, verification results can be recorded in log services, and anomaly indicators can be tracked through monitoring and alerting. In the event of a dispute, these materials can reconstruct the full timeline. In Baidu AI Cloud’s practice, for example, keys can be hosted in a key management service and audit records can be stored in object storage. These are general cloud engineering practices and do not represent any specific product capability.

8. Implications for Service Providers: Disclosure Beyond Signatures

• Same-vendor downgrade is the core trust issue: Signatures can prove that content has not been modified, but proving that the model has not been replaced requires the model service provider to establish disclosable and auditable version and configuration management.
• Credentials should be treated as part of the product capability: Credential formats, public key distribution channels, key rotation cycles, and revocation processes should be included in API documentation and change announcements rather than hidden inside implementation details.
• Version and configuration changes must leave traces: Each upgrade should generate a new signed configuration statement, and historical versions should be retained so users can trace back to “which model was used at that time.”

9. Engineering Implementation and Evolution Roadmap

9.1 Engineering Essentials

• Private key protection: Private keys should exist only in hardware security modules, cryptographic modules, or trusted execution environments. The signing service itself should also be authenticated to prevent misuse as a general-purpose signing endpoint.
• Deterministic serialization: Before signing, the payload must be normalized, including field ordering, unified numeric formatting, and consistent Unicode encoding. Otherwise, the content may be unchanged, but the serialization result changes, causing verification to fail.
• Atomic return: Credentials should be returned together with the response content and stored together. This avoids broken chains where a response exists without its credential.
• Key lifecycle management: The full lifecycle of generation, distribution, rotation, revocation, and archiving should be planned, ensuring that both old and new public keys can verify signatures correctly during rotation.

9.2 Staged Rollout

1. Pilot phase: Issue credentials in internal or small-traffic scenarios, collect the distribution of verification failure reasons, and confirm protocol details.
2. Default-on phase: Client SDKs verify credentials by default and monitor metrics such as unsigned responses, verification failure rates, and version drift.
3. Evidence retention: Package credentials and responses into object storage to build a long-term audit chain.
4. Ecosystem coordination: Encourage upstream and downstream participants, such as relay services, gateways, and evaluation platforms, to unify credential formats and trust models.

10. Conclusion

Disputes about relay services swapping models or providers “downgrading” capabilities ultimately come from information asymmetry: users cannot verify what they actually purchased. Adding a verifiable “anti-fraud code” to LLM responses shifts the judgment of authenticity from trust to evidence. This direction is correct and engineering-feasible.

At the same time, it is important to remain clear-eyed. Signatures can solve whether content has been modified and whether it was signed by the legitimate key holder, but they cannot solve whether that key holder is honest. A truly trustworthy AI service requires cryptographic mechanisms, provider self-discipline, industry norms, and user oversight. For developers and enterprise users, rather than debating whether a response “looks like” a flagship model, it is better to integrate credential verification into the default flow of clients and gateways as early as possible, so that every call leaves verifiable evidence.