Verifying AI Responses: Cryptographic Proofs for Detecting Model Substitution

I. The problem: why should you trust that an AI response came from the claimed model?

Under a model-as-a-service model, a single request usually passes through a long chain:

Client → authentication and billing gateway → traffic routing → inference backend, possibly with fallback, caching, or rewriting policies → model weights → streaming response

Whenever the user and the service provider are not the same party, there is room for a mismatch between what was requested and what was actually served. Common distortions include the following.

1.1 Model substitution: high-tier requests routed to cheaper models

A single API name may expose multiple model tiers. A typical case is a request billed as a high-performance tier being forwarded by the gateway to a smaller and cheaper model from the same family, or to a comparable model from another vendor.

1.2 Prompt disguise: self-reported identity proves nothing

A common first reaction is to ask the model, “Who are you?” But if the system prompt says “You are powered by model X,” almost any model can repeat that answer. Identity questions only test whether a model can role-play; they do not establish what model is actually running.

1.3 Silent degradation: capabilities can shrink without changing weights

Even if the weights are unchanged, capability can be reduced quietly:

• Lowering numerical precision or compressing the model.
• Reducing context length or truncating the reasoning chain.
• Changing default sampling parameters or serving stale cached answers.
• Leaving temporary degraded configurations in place after maintenance.

These problems are common in incident reviews, but from the user’s side they often appear simply as “why is it so much worse today?”

1.4 Distilled impersonation: behavior looks similar, but it is not the original model

One concern is that a model’s behavior can be imitated by training a mimic model on outputs from a stronger model. The mimic model may learn the target’s style and even its refusal boundaries. Such models can produce polished, similar-sounding answers, but they often struggle with complex reasoning, long contexts, and numerical computation. Compared with direct model substitution, distillation-based imitation is harder to detect with conventional content-level checks.

1.5 The limits of existing detection methods

The community commonly uses three types of checks:

• Response style comparison: Build baseline output vectors and apply similarity or statistical tests. Style can be deliberately imitated, and statistical results are only probabilistic.
• Response field validation: Check metadata such as the returned model identifier or usage statistics. The problem is that these fields are filled in by the service provider itself, making them self-attested.
• Signature and fingerprint validation: Verify signatures over inference traces or model-identity fingerprints. This can confirm that the upstream provider is a certain model vendor, but it still cannot tell which exact model tier was called.

More fundamentally, the service provider’s own statements cannot be independently verified by the user. When quality drops, it may be because the provider changed the model, or because the model was updated. Either way, the user has no independently verifiable evidence.

Most existing checks therefore stay at the level of “looking at content, looking at fields, or looking at signatures.” The first can be imitated; the latter two are filled in by the party being verified and therefore still rely on trust. The core question becomes:

Can the service provider produce a cryptographic proof that anyone can verify, but that the provider itself cannot retroactively alter or forge?

II. Core idea: turning trust into commitments and proofs

2.1 Cryptographic commitments

In cryptography, a commitment lets one party lock a value now and reveal it later. It must satisfy two properties:

• Binding: Once issued, the commitment cannot be changed to a different value without being detected.
• Hiding: Before revelation, others cannot infer the committed value from the commitment itself.

The simplest implementation is a cryptographic hash such as SHA-256. With a domain-separated encoding rule, content is converted into a fixed byte string and hashed to produce a binding value. Because SHA-256 is collision-resistant, finding two different contents with the same hash is computationally infeasible.

Hash commitments have a classic pitfall. If the content is easily enumerable, such as “what day is today,” an attacker can try candidates one by one and reverse the commitment. Adding a sufficiently long random value, or salt, restores hiding.

2.2 Three levels of verification strength

Applying commitments to AI response verification produces three progressively stronger levels:

Level	Who makes the commitment	What it proves	What it does not prevent
L1: Post-hoc audit	The user	The response was not locally altered	Provider-side model substitution
L2: Transparency log	The service provider	The response has not been replaced since generation, and replacement cannot be hidden	Use of the wrong model at generation time
L3: Verifiable proof	The service provider or model vendor	The response was generated by the declared model with the declared parameters	Quality differences of the upstream model itself

The higher the level, the stronger the proof, but the more cooperation it requires from the model service provider.

2.3 Why verifying large-model responses is especially hard

For ordinary computation, such as multiplying large numbers, a verifier can simply recompute the result. Text generation from large models creates three additional challenges:

1. Asymmetric recomputation cost: If the verifier reruns the model, compute cost is comparable to generation, defeating the purpose of cheap verification.
2. Stochastic output: When sampling temperature is greater than zero, the same input can produce different answers each time.
3. Exact reproduction is difficult: Even with greedy decoding, floating-point accumulation order can differ across hardware and concurrency configurations, causing some tokens to diverge.

These challenges are exactly what tools such as verifiable computation and zero-knowledge proofs are designed to address.

III. How an anti-forgery proof is generated

A complete AI response verification mechanism usually involves four roles:

• Generator: The model service provider or model supplier. It commits to the response and provides proofs.
• Verifier: The end user or a third-party auditor.
• Public ledger: A transparency log service that appends commitments in time order.
• Clock and signing service: Provides trusted timestamps and proves that commitments were actually issued by the service provider.

The overall workflow can be broken into several stages.

3.1 Step 1: commitment

After generating a response, the generator collects the key elements of the inference into an “inference record” and computes a commitment. The record should include at least:

• Input: The original prompt, system prompt, and conversation history.
• Parameters: Sampling temperature, nucleus-sampling parameter, maximum generation length, random seed if supported, and retrieval or tool-call results.
• Model: Model identifier, version, and weight digest.
• Output: All tokens with their sequence indices.
• Signature: A provider private-key signature over the above content, plus a timestamp.

Illustrative code:

import hashlib, hmac, json, os
def _sha(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()
def make_record(prompt: str, model_id: str, weight_digest: str,
                tokens: list[str], temperature: float, seed: int | None) -> dict:
    """Build an inference record from the key elements of one inference call."""
    return {
        "input": {
            "prompt": prompt,
            "tokens": [ord(c) for c in prompt],
        },
        "params": {
            "temperature": round(float(temperature), 6),  # Normalize the numeric value to avoid different representations of the same value.
            "seed": seed,
        },
        "model": {"id": model_id, "weight_digest": weight_digest},
        # Assign an index to each token to prevent output order from being rearranged.
        "output": [{"i": i, "t": t} for i, t in enumerate(tokens)],
    }
def merkle_root(leaves: list[str]) -> str:
    """Merkle tree: repeatedly hash pairs of nodes until only the root remains."""
    level = [_sha(l.encode()) for l in leaves]
    while len(level) > 1:
        if len(level) % 2:               # If the number of nodes is odd, duplicate the last node so every node can be paired.
            level.append(level[-1])
        level = [_sha(a.encode() + b.encode()) for a, b in zip(level[::2], level[1::2])]
    return level[0]
def build_commitment(record: dict, prompt: str, signing_key: bytes) -> dict:
    prompt_leaf = _sha(prompt.encode())  # Keep the prompt private; commit only to its hash.
    params_leaf = _sha(json.dumps(record["params"], sort_keys=True).encode())
    output_root = merkle_root([str(item["t"]) for item in record["output"]])
    head = json.dumps({"input": [prompt_leaf], "model": record["model"]},
                      sort_keys=True, ensure_ascii=False).encode()
    head_leaf = _sha(head)
    root = merkle_root([head_leaf, params_leaf, output_root])
    signature = hmac.new(signing_key, root.encode(), hashlib.sha256).hexdigest()
    return {"root": root, "output_root": output_root, "params_hash": params_leaf,
            "head_hash": head_leaf, "signature": signature}

A Merkle tree is a common foundation for blockchains and transparency logs. It uses a root hash to summarize a large amount of data and can produce a short inclusion proof for any individual item.

If the raw text were hashed as one block, changing any token would change the overall hash, but it would be impossible to identify exactly what changed. With a Merkle tree, the provider can prove that “token i is t” without exposing other content. Inputs that may contain sensitive information, such as prompts, can also be committed by hash without revealing the original text.

An illustrative commitment message is shown below. Field names are illustrative only; field case and naming style must be consistent across the entire pipeline.

{
  "request_id": "7f3c-4e2a",
  "commitment": {
    "root": "9b1d…e7",
    "output_root": "c4a0…21",
    "params_hash": "5e6f…8d",
    "head_hash": "a7b2…3c",
    "signature": "3d9e…f1"
  },
  "model": { "id": "chat-large-2025q3", "weight_digest": "sha256:77c1…b0" }
}

Appending the commitment to a public ledger is also essential. If the provider only generates commitments for its own responses, users cannot tell whether a commitment was created on the spot. Only after commitments are appended to an ordered public ledger does retroactive replacement become impossible to hide.

3.2 Step 2: commitment proof

When a user verifies a response, the provider generates a Merkle inclusion proof for each token in the response. The verifier only needs to recompute along the hash path to confirm that the token exists under the original root. The computation cost grows logarithmically with response length.

The diagram below shows the hashing direction.

def merkle_proof(leaves: list[str], index: int) -> list[tuple[str, bool]]:
    """Return the path as (sibling hash, whether the sibling is on the right)."""
    level = [_sha(l.encode()) for l in leaves]
    proof = []
    while len(level) > 1:
        if len(level) % 2:
            level.append(level[-1])
        sibling = index ^ 1                          # Position of the sibling paired with the current node.
        proof.append((level[sibling], bool(sibling & 1)))
        level = [_sha(a.encode() + b.encode()) for a, b in zip(level[::2], level[1::2])]
        index >>= 1
    return proof
def verify_merkle(leaf: str, proof: list[tuple[str, bool]], root: str) -> bool:
    h = _sha(leaf.encode())                            # Diagram check: sibling is on the right.
    for sibling, sibling_on_right in proof:            # Diagram check: append sibling to the right.
        if sibling_on_right:
            h = _sha(h.encode() + sibling.encode())
        else:
            h = _sha(sibling.encode() + h.encode())
    return h == root

This commitment proof prevents “post-generation swapping.” Once a response has been generated and recorded, no one can silently replace it with another version without detection. However, it assumes that the correct model was already used during generation.

3.3 Step 3: zero-knowledge proof

A zero-knowledge proof is a cryptographic protocol that lets a prover convince a verifier that a statement is true without revealing information beyond the statement itself. It has three basic properties:

• Completeness: If the statement is true, an honest prover can always convince the verifier.
• Soundness: If the statement is false, a cheating prover can pass verification only with negligible probability.
• Zero knowledge: The verifier learns nothing beyond the truth of the statement.

A classic illustrative example is graph three-coloring. A map must be colored so that adjacent regions have different colors. The prover claims to know a valid coloring. Each round, the prover covers the map and reveals only two adjacent regions, allowing the verifier to check whether their colors differ. If the prover does not know a valid coloring, random guessing is likely to fail eventually. After enough rounds, the verifier becomes highly confident.

In real deployments, the statement is usually more specific, for example:

There exists a set of model parameters W and an inference computation such that the hash of W equals the publicly declared weight digest, and, given the input and sampling parameters, this computation produces exactly the output committed in the public commitment.

With a zero-knowledge proof, the verifier does not need to rerun the computation or see the model or prompt text.

A lighter approach is probabilistic checking: randomly sample several positions in the response and require the prover to show that the computation at those positions is correct. This is a classic idea from verifiable computation research. Under suitable assumptions, the probability that a cheater passes random spot checks decreases exponentially with the number of checks.

3.4 Why zkSNARK

zkSNARK is one commonly used family of zero-knowledge proof systems. Its name reflects four characteristics:

• Succinct: Proof size and verification time are small and grow only weakly with the size of the verified computation. Responses may be tens or hundreds of thousands of bytes, and generation may take seconds, but such proofs are often only hundreds of bytes to a few kilobytes, with verification in the millisecond to tens-of-milliseconds range. These are illustrative magnitudes and depend on the scheme and implementation.
• Non-interactive: The prover can produce one proof without repeated interaction with the verifier, which fits streaming Q&A scenarios.
• Completeness, soundness, and zero knowledge: These are the basic properties of zero-knowledge proofs.
• Argument of knowledge: Security relies on computational hardness assumptions. A computationally unbounded attacker could theoretically forge a proof, just as collision-resistant hash security depends on computational hardness.

3.5 End-to-end flow

Combining the three parts, a verifiable Q&A interaction works as follows:

1. When generating the response, the service provider computes a commitment to the inference record and appends it to the public ledger.
2. At the same time, it generates a zero-knowledge proof stating that “this output was computed by a model with weight digest H using the publicly declared parameters.”
3. During verification, the user first uses the signature and timestamp to confirm that the commitment belongs to the service provider, then uses the inclusion proof to confirm that the response has not been replaced, and finally uses the verification algorithm to check the zero-knowledge proof.

If all three steps pass, the user has strong reason to believe that the response came from the declared model and was not modified after generation. The process does not require trusting any single intermediary.

In practice, implementations often rely on unified proof frameworks that encapsulate low-level components such as arithmetic circuits, constraint systems, and polynomial commitments, such as KZG commitments over pairing-friendly curves. Selection criteria should pay particular attention to support for incremental proofs and hardware acceleration, avoiding schemes that are unsuitable for large-model scale.

3.6 Practical approximations: random recomputation and random challenges

Generating a complete zero-knowledge proof is still expensive. The industry therefore uses several lighter compromises:

• Random recomputation spot checks: The generator first uses a secure random number generator to decide which positions should be recomputed and commits to those positions. The provider recomputes only the specified positions and returns intermediate values with proofs. The verifier can then check them at a cost far below full generation.
• Random challenges: After a response arrives, the verifier randomly selects several positions and asks the provider to supply computation proofs for those positions.

Randomness is critical. If spot-check positions can be predicted, the provider can compute carefully only the positions that will be checked. For this reason, the protocol must specify a shared random source, and both sides must use cryptographically secure randomness rather than independently inventing values.

IV. Four major obstacles to real-world deployment

4.1 Nondeterminism: the same input may produce different results

Large-model inference is fundamentally floating-point computation. When matrix multiplication uses tiling and parallel reduction, the order of addition affects rounding errors. When concurrent requests are merged into the same batch, batch composition may also affect individual numerical results. These tiny differences can be amplified across dozens of layers and eventually flip a token choice.

Common engineering mitigations include:

• Using a fixed batch size or disabling dynamic batching.
• Fixing tensor-parallel partitioning.
• Preferring greedy decoding or fixing the random seed during sampling.
• Committing to the final token sequence rather than every intermediate floating-point value.

Even with these measures, bit-for-bit consistency across hardware platforms and inference frameworks is still difficult. Protocol design should therefore clearly state whether it targets probabilistic consistency or deterministic consistency.

4.2 Computational overhead: the cost of circuitization

To describe a model’s forward pass inside a zero-knowledge proof, operators such as matrix multiplication, nonlinear activation, and attention softmax must be converted into constraints in an arithmetic circuit. The larger the model and the longer the sequence, the larger the constraint system. Proof generation time is often orders of magnitude slower than native inference.

If the proof statement itself includes the full input, its size can grow linearly with prompt length. Common engineering optimizations include:

• Using hash commitments instead of raw data.
• Precompiling constant parameters, such as model weights, into the circuit.
• Batch-proving the token-by-token decoding process.

In short, proving that every step is correct is theoretically feasible, but expensive in engineering practice.

4.3 Trust assumptions: proofs still have premises

zkSNARK does not eliminate trust. It moves trust to clearer places:

• Trusted setup: Many zkSNARK schemes require a one-time public reference string. The parameters are safe as long as each participant destroys its own toxic waste. The system remains secure if at least one participant is honest. But if all participants collude, fake proofs may be possible.
• Cryptographic assumptions: If collision-resistant hashes or elliptic-curve pairing assumptions are broken, the proof system fails.
• Signatures and clocks: Commitment signatures depend on key security; timestamps depend on clock services. If a signing key leaks, an attacker can backdate arbitrary commitments.
• Circuit correctness: A proof can show that a circuit was executed, but not that the circuit equals the intended model. Therefore, the model weight digest must come from a trusted source, for example a digest signed and published by the model supplier.

4.4 Operational costs: keys, clocks, and ledgers

Providing a proof for every request adds compute, bandwidth, and storage costs:

• The signing service must be highly available, and keys should preferably be stored in hardware security modules or key management services.
• The transparency ledger needs reliable write confirmation and efficient query support.
• Proof caching, batch generation, and proof sampling per request are common cost-control methods.

V. From principles to practice: role-based verification checklists

5.1 End users and developers: self-checks without cryptography

Cryptographic verification requires provider cooperation, so ordinary users may not have access to it immediately. In the short term, they can still build a “baseline profile” using engineering methods.

import asyncio, hashlib, time
import matplotlib.pyplot as plt   # Used only for local plotting.
def digest(text: str) -> str:
    """Normalize whitespace before hashing so minor spacing differences do not dominate."""
    return hashlib.sha256(" ".join(text.split()).encode()).hexdigest()
async def probe(client, task: str, rounds: int = 20):
    t0 = time.perf_counter()
    # Example call; if the service supports reproducible parameters such as a random seed, set them.
    resp = await client.generate(prompt=f"After completing the task, output only the answer: {task}",
                                 temperature=0.0)
    elapsed = time.perf_counter() - t0
    # Use model identifiers, usage statistics, and other metadata in the response if available.
    # Exact response structure depends on the API.
    meta = resp.get("meta", {})
    return {
        "model": meta.get("model"),
        "usage": meta.get("usage"),
        "latency_ms": elapsed * 1000,
        "digest": digest(resp["text"]),
        "len": len(resp["text"]),
    }
async def benchmark(client, tasks: list[str], rounds: int = 20):
    rows = [r for task in tasks for r in await asyncio.gather(
        *[probe(client, task) for _ in range(rounds)])]
    by_model = {}
    for r in rows:
        by_model.setdefault(r["model"], []).append(r)
    for model, items in by_model.items():
        n = len(items)
        unique = len({i["digest"] for i in items})
        lat = sorted(i["latency_ms"] for i in items)
        lens = [i["len"] for i in items]
        print(f"[{model}] samples={n} | unique answers={unique} | median latency="
              f"{lat[n // 2]:.0f}ms | P95={lat[int(n * 0.95) - 1]:.0f}ms"
              f" | average length={sum(lens) / n:.1f}")
        # Plot length and latency curves by sampling date to detect abrupt step changes.
        plt.plot(lens, label=f"{model} length")
    plt.legend()
    plt.show()

The script above is only illustrative. For real deployments, it is better to follow this checklist:

• Metadata verification: Check whether the model identifier, usage statistics, and request ID in the response match expectations, and whether the billed tier differs from the returned model identifier. These fields are filled in by the service provider and should be treated as clues, not conclusions.
• Fixed sampling parameters: Set temperature to zero or fix the random seed so that answers to the same question are as stable as possible.
• Consistency spot checks: Send the same request multiple times under zero-temperature settings and measure the repetition rate of response hashes. For the same model and version, the repetition rate should be relatively high. If every response is different, either sampling parameters are not effective or the backend is dynamically switching.
• Statistical profiling: Record response length, time to first token, and per-token generation speed over time. A sudden step change, such as answers becoming much shorter or generation suddenly becoming faster, may indicate a backend change.
• Capability benchmarks: Prepare reasoning, math, and coding tasks with standard answers. Score them regularly and map the results to the model versions published by the provider. If scores drop significantly, request a written explanation from the service provider.
• Evidence retention: Store prompts, complete responses, metadata, and timestamps, or at least hashes and signatures, for future comparison.
• Contract-level confirmation: In procurement or service agreements, specify model versions, downgrade notification obligations, advance notice periods for changes, and how violations are determined.

These methods cannot prove exactly which model is behind the service, but they can raise the cost of substitution and reveal obvious anomalies.

5.2 Model service providers: making verifiability a product capability

For model service providers, verifiability is both a constraint and a differentiating capability. A phased implementation is recommended:

1. Reproducible requests: Support fixed random seeds, fixed batching strategies, or a deterministic mode so that the same request produces consistent results under the same version as much as possible.
2. Metadata transparency: Return model identifier, version, weight digest, sampling-parameter echo, and request ID in responses, and provide an independently queryable “look up by ID” interface.
3. Commitments and transparency logs: Generate commitments for each request and append them to a transparency ledger. Provide an inclusion-proof query interface so any third party can verify that a response has not been replaced.
4. Zero-knowledge proof pilots: Start with short outputs, small models, and low-concurrency scenarios, such as classification, extraction, or fixed-length structured output. Random spot-checking can also be used, generating full proofs only for a subset of requests.
5. Audit and monitoring: Introduce monitoring metrics such as model version changes, latency step changes, and response-length distribution drift. Track commitment generation and proof generation failure rates in centralized logging.
6. Compliance and documentation: Clearly document the coverage of proofs, including which stages are covered and which trust assumptions are used. Avoid overpromising claims such as “fully provable end to end.”

VI. Boundaries and conclusion

Returning to the original question: verifying an AI response is essentially about turning “trust the provider” into “verify a cryptographic proof.” This technical route can address whether a response was replaced and which model generated it. It does not solve whether the model is intelligent enough. Model quality still requires an evaluation system. Commitment mechanisms can detect substitution, but they cannot replace quality assessment.

For end users, the most practical approach today is to build a self-check system based on statistical profiling, metadata verification, and evidence retention. For service providers, the goal is to make reproducibility, traceability, and verifiability standard capabilities. As proof costs decline with specialized hardware and algorithmic optimization, “every response is verifiable” may eventually become infrastructure, much as HTTPS became fundamental to web browsing.

It is still too early to say when anti-forgery proofs will become common in mainstream large-model services. But it is clear that replacing trust with proof is an important step toward maturing large-model services.