LLM Output Authenticity with Zero-Knowledge Proofs: A Practical Guide

1. The Problem: Who Generated the Answer?

When developers use a large language model API, what they receive is usually just a JSON response: the generated text, token usage, and a finish reason. The response does not directly prove which model produced the text, which model version was used, or what sampling parameters were applied. The caller cannot verify these details on its own.

The presence of middle layers makes the trust chain even longer. A request may pass through gateways, aggregation platforms, proxy services, or other intermediaries before it reaches the model. Each layer has an opportunity to change the model that is actually used.

In practice, this creates three common problems:

• Downgrading without disclosure. A user may pay for a flagship model but receive a response generated by a smaller and cheaper model. The quality gap can be obvious on coding, reasoning, and other demanding tasks. Casual users, however, may find it difficult to tell the difference: a fluent-looking answer may have come from a model with far fewer parameters.
• Silent degradation. A provider may switch models during peak traffic, shorten output length, or reduce inference compute without saying so. When users feel that the model has become “less intelligent,” it can be hard to determine whether the cause is a model update, a load-management strategy, or a mistaken impression.
• Impersonation. A model trained to imitate the output style of a target model may look similar on the surface, but it can still differ significantly in factual accuracy and reasoning depth.

The core issue is that model identity is not cryptographically bound to the response.

2. Why Existing Detection Methods Are Not Enough

Common detection approaches generally fall into three categories:

1. Identity interrogation.
   This approach asks the model directly, “Who are you?” It is easy to bypass: an intermediary can add a system prompt such as “You are a flagship model,” and most models will answer accordingly. A model’s self-description comes from the prompt, not from its underlying weights, so such answers have little evidentiary value.

2. Output feature comparison.
   This approach analyzes stylistic features, high-frequency words, tokenization habits, refusal patterns, and other statistical signals. Different models do exhibit differences because their training data and alignment strategies differ. However, statistical features can only support a probabilistic judgment. Prompts, temperature settings, fine-tuning, quantization, and hardware precision can all change the output distribution, making false positives and false negatives possible.

3. Response-field and reasoning-summary validation.
   This approach checks model identifier fields in the response, or verifies signed reasoning summaries and similar metadata. These methods may show that “the response came from the API of a certain provider,” but they do not answer the more important question: behind the same endpoint, was this particular request served by the flagship version or a lightweight version? Was the underlying model replaced?

These methods share a common weakness: they can infer, but they cannot prove. Their conclusions depend on the detector’s experience and on whether the intermediary cooperates, so they can be challenged at any time. Model providers are also put in a passive position. When users complain about degraded quality, even if the weights and inference configuration have not changed, the provider may still struggle to produce evidence that a third party can independently verify.

3. Turning Trust into Verification with Zero-Knowledge Proofs

A zero-knowledge proof is a cryptographic protocol in which a prover can convince a verifier that a statement is true without revealing the secret itself. Applied to response authentication, the statement to be proved can be stated precisely as:

Response R is the output obtained by performing a forward inference pass on input X using model M, whose weights are committed by C, under inference parameters P.

In schemes such as zkSNARKs, succinct non-interactive arguments of knowledge, the statement is translated into a set of arithmetic constraints. Each step of the inference becomes an equation that must be satisfied. The prover finds an assignment that satisfies all constraints and generates a proof from it. The verifier only needs to check the proof to confirm that “such an inference existed and its result matches the claimed response.”

A good proof system usually has three properties, which map directly to this use case:

• Completeness: A genuine inference can always produce a proof that passes verification.
• Soundness: A party that did not actually perform the computation can almost never forge a passing proof.
• Zero knowledge: The verifier learns nothing beyond whether the statement is true.

Zero knowledge is especially important here. Model weights are a core asset of the provider and cannot be made public just to support verification. Zero-knowledge proofs allow a provider to prove that “the response was computed from these weights” without disclosing the weights. This reduces the provider’s concern while aligning with its commercial interests.

A common misunderstanding should also be clarified: zero-knowledge proofs protect the model secret from being disclosed to the verifier, not the user data from the model provider. User input is still processed by the model during inference. Data protection must be addressed through transport encryption, retention policies, and compliance controls. The two concerns should not be confused.

Proof-Carrying Inference is an active research direction. The idea is to attach a cryptographic proof of the inference process to the model output so that a verifier can confirm the response’s provenance. Representative work includes zkFormer, which designs dedicated proof schemes for Transformer inference, and zkLLM, which chains multiple smaller models into a pipeline and generates a succinct proof for the pipeline. Their common goal is to turn “who generated the response” from a question of trust into a mathematical question.

4. Three Steps: Commit, Prove, Verify

Step 1: Commit — Pin Down Identity

A commitment is a cryptographic construction. The committer first places a value into a sealed envelope and gives the envelope to another party. Later, the committer can neither open it arbitrarily nor deny its contents.

In the anti-impersonation scenario, three types of information need to be committed:

• Model fingerprint: A cryptographic hash of the model weights. A complete weight file usually consists of multiple shard files. Each shard can be hashed separately and then aggregated into a root hash. This works similarly to a Git commit ID: any tiny change to the weights causes the fingerprint to change completely.
• Inference parameters: Sampling temperature, maximum output length, a digest of the system prompt, and related settings.
• Input and output digests: Hash values of the user request and the model response.

The key principle is: what is published is the commitment, not the raw data. Outsiders see fingerprints and hashes, while the weights remain inside the provider. The Python pseudocode below illustrates how a fingerprint can be computed:

import hashlib
def compute_model_fingerprint(weights_iterator) -> str:
    """
    Compute the model fingerprint (pseudocode).
    weights_iterator: yields weight shards as byte streams in a fixed order,
    avoiding loading all parameters into memory at once.
    Returns: the public model fingerprint, i.e., the root hash.
    """
    h = hashlib.sha256()
    for shard in weights_iterator:
        # Hash each shard separately before feeding it into the root hash.
        # This prevents shard reordering or substitution.
        h.update(hashlib.sha256(shard).digest())
    return h.hexdigest()

Step 2: Prove — Produce a Credential for the Computation

After committing, the provider must prove that “the response was indeed computed from these weights,” rather than being generated by a different model, produced arbitrarily, or assembled from cached fragments. The overall process is:

1. Formalize the inference process into a set of arithmetic constraints.
2. The prover uses the intermediate tensors from the actual execution as private witness input to the solver.
3. The proving algorithm generates a proof, π.

Before the constraint system is compiled into a circuit, it is usually expressed in an intermediate representation, such as R1CS or Plonkish, which is often compared to assembly for circuit languages. Specialized compilation schemes already exist for Transformer-style structures. For example, matrix multiplication can be broken down into dot products, and nonlinear activation functions can be expressed efficiently with lookup tables. Quantization can also map floating-point operations to integer operations over a finite field.

The most expensive part of proving is usually the polynomial commitments and FFT computations required to generate the proof. Several optimization directions have been explored, such as using lookup tables to accelerate hash functions, reducing the number of hash invocations, and replacing some polynomial commitments with the Sumcheck protocol.

Overall, these optimizations are still evolving. Compared with mature inference engines, the gap remains large. This is the most important constraint when discussing engineering deployment later.

Step 3: Verify — A Publicly Checkable Anti-Counterfeit Code

Verification requires only three things:

• The proof π.
• Public inputs, including the model fingerprint commitment, inference parameter digest, request digest, and response digest.
• A public verification key.

Any third party can verify independently without accessing the model weights or trusting the provider.

zkSNARKs have particular advantages here:

• Proofs are usually only a few hundred bytes to a few kilobytes.
• Verification time is constant, typically on the order of milliseconds.
• Verification based on elliptic-curve pairing is lightweight enough to run in a browser or even on smart devices.
• Some schemes can be verified with a single on-chain transaction, making public recording on a blockchain feasible.

However, many pairing-based schemes require a trusted setup. For circuit-specific schemes, a structured reference string must be generated. During that process, secret parameters must be destroyed before anyone can retain them; otherwise, they could be used to forge proofs. Universal circuits with a one-time reusable setup can mitigate this issue. For scenarios requiring public verifiability, this is an important factor when choosing a scheme.

5. Receipt Structure and Replay Protection

A proof material is essentially an “anti-counterfeit code.” It can be returned together with the response by the provider, or generated asynchronously. An example receipt structure is as follows:

{
  "version": "1.0",
  "model_fingerprint": "a3f1c9…e7",
  "params_commitment": "b52d…9f",
  "request_digest": "c81e…04",
  "response_digest": "d07a…b3",
  "nonce": "8f3c-…",
  "timestamp": 1767225600,
  "proof": "<base64-encoded zero-knowledge proof>",
  "public_inputs": "<encoded public inputs>",
  "verification_key_id": "vk-2026-07"
}

One of the most overlooked issues is replay attacks. Suppose an attacker once obtained a receipt with a valid proof generated by a flagship model. Later, the service switches to a lightweight model and returns the cached text together with the old receipt. Because the old receipt itself is valid, checking only whether the proof passes will still succeed.

There are two layers of protection:

1. Digest binding. Before verification, the client recomputes the hash of the response and compares it with the response_digest field in the receipt. If even one byte of the text is changed, the digest will not match. The JSON above uses ellipses to abbreviate hashes; in real use, full values should be retained.
2. One-time nonce. For each request, the client generates a random nonce, sends it with the request to the model, and includes it in the proof as a public input. This prevents cached content from being reused for new requests.

A timestamp should also be included in the public inputs and combined with a validity-window check to reduce the time window in which a receipt can be exploited.

The client-side SDK verification logic can be approximated as follows:

def verify_receipt(receipt: dict, vk: bytes, seen_nonces: set) -> bool:
    # 1. Verify the answer content: recompute the digest to detect replacement.
    if sha256(receipt["answer"]) != receipt["response_digest"]:
        return False
    # 2. Check validity window: reject receipts outside the allowed time range.
    if not within_valid_window(receipt["timestamp"]):
        return False
    # 3. Replay protection: accept each nonce only once.
    if receipt["nonce"] in seen_nonces:
        return False
    seen_nonces.add(receipt["nonce"])
    # 4. Verify the proof. Public inputs cover the model fingerprint,
    #    parameter commitment, request digest, response digest, and nonce.
    public_inputs = encode([
        receipt["model_fingerprint"],
        receipt["params_commitment"],
        receipt["request_digest"],
        receipt["response_digest"],
        receipt["nonce"],
    ])
    # Verification does not access any model weights.
    return zk_verify(vk, public_inputs, receipt["proof"])

There is also a common design flaw: if the public inputs include only input and output digests but not the model fingerprint, a relay service could wrap an answer generated by a cheap model inside a genuine, valid proof and pass it off as an answer from an expensive model. To close this gap, the model fingerprint must be included in the public inputs and strictly constrained by the proof circuit.

For relay providers, the simplest and safest approach is to pass through the upstream receipt unchanged: forward the request and response as they are, deliver the proof materials to the end user, and avoid holding any keys. If the provider replaces the model on its own, the fingerprint mismatch will be exposed immediately.

6. Deployment Challenges: How Far Are We From Stamping Every Response?

1. Performance Overhead

This is the biggest obstacle. A single large-model inference involves billions or even more multiplication and addition operations. Zero-knowledge proving requires each operation to be rewritten into constraints over a finite field and then proven, which can cost thousands of times more than the inference itself, or even more.

Current public research mostly focuses on small or medium-sized models, or on only part of a model. A complete proof for every response from a production-scale large model is still far away.

Possible compromises include:

• Asynchronous receipts: Return the answer normally first, then generate the proof asynchronously and deliver it through a receipt endpoint.
• Sampling-based verification: Generate proofs only for some requests in each session.
• Layered proving: Prove only critical paths or critical layers.

2. Trust and Governance

• The circuit itself may contain hidden backdoors. It should be publicly audited by multiple parties, ideally compiled in a transparent and trusted environment to ensure that the circuit matches the model description.
• The constraint system must correspond strictly to the model version. Changing the quantization scheme, modifying an inference kernel, or even switching the implementation of the inference engine may invalidate the constraints and require recompiling the circuit.
• The fingerprint system needs governance, including public registration, periodic recomputation, and version rotation. Otherwise, the provider may still use old, valid receipts to respond to questions about a new version.

3. Privacy Boundaries

• During inference, the model can still access user input. Zero-knowledge proofs do not change this.
• Whether a request digest can be used to reconstruct the original input depends on the input length and predictability. Highly sensitive scenarios should assess the risk of hash reversal and consider adding salt where necessary.
• Receipts contain request and response digests and are therefore call records. Their retention period and access permissions should follow data-security rules.

7. Capability Boundaries: What Can and Cannot Be Proven?

Question	Can zero-knowledge proofs answer it?
Was the response computed by a model corresponding to a given fingerprint?	Yes. This is the core capability.
Were the inference parameters the same as claimed?	Yes, provided the parameters are included in the public inputs.
Was the response tampered with in transit?	Yes. Digest comparison can detect it.
Was a cached response reused?	Yes, when combined with a nonce and timestamp.
Was the model “smarter,” or was the answer high quality?	No. It proves provenance, not quality.
Was the model trained through distillation?	No. Training-data provenance is outside the proof’s scope.
Did the provider retain user data?	No. Privacy compliance controls are still required.

In other words, zero-knowledge proofs address provenance authenticity, not quality ranking. Even a flagship model can produce incorrect answers. But once provenance is verifiable, quality evaluation can be anchored more reliably by combining it with industry benchmarks, manual spot checks, and A/B comparisons.

8. Complementary Technologies: Remote Attestation and Hardware-Based Trusted Execution

In addition to mathematical proofs, another path establishes trust at the hardware level: remote attestation. Confidential computing based on a trusted execution environment runs the model inside a hardware-protected enclave. A chip vendor embeds a root key in the processor at manufacturing time. The platform issues an attestation based on that root key. After a remote user verifies the attestation, it can confirm that “the expected program is indeed running without tampering.”

The two routes have different focuses:

• Remote attestation proves “what program ran in what environment.” It can also provide runtime confidentiality for user input, but its root of trust lies with the hardware vendor, and side-channel attacks remain a recurring concern.
• Zero-knowledge proofs prove “what constraints the computation satisfies.” Their root of trust lies in mathematical assumptions and circuit correctness. They offer stronger public verifiability, but provide limited protection for user data and carry large performance overhead.

In real systems, the two can be combined: TEEs protect data and the execution environment, while zero-knowledge proofs provide publicly verifiable computational receipts to all stakeholders.

9. How Different Roles Can Prepare

• Model providers: Establish fingerprint registration for different versions, include version information in API response fields, reserve interfaces for generating and distributing proof materials, and assess the impact of inference-engine upgrades on the constraint system in advance.
• Intermediary providers: Stick to the pass-through principle. Do not replace upstream identity fields, do not modify response content, and deliver upstream receipts unchanged to end users.
• Enterprise buyers: Introduce verifiability requirements into procurement terms, and specify the verification method and sampling ratio used during acceptance.
• Application developers: Reserve receipt-verification hooks in the client SDK, generate nonces on the request side, record and compare model fingerprints, and set up alerts for responses that fail verification. Verification and receipt retention can rely on common cloud capabilities such as object storage, log services, and cloud monitoring. For example, receipts can be archived in object storage with lifecycle policies, and verification results can be collected centrally for auditing and alerting.

Conclusion

Return to the original question: how can you know whether a response was generated by the model you paid for? Zero-knowledge proofs provide a theoretically grounded path that turns “trust or not” into “verify or not.” They cannot immediately expose every relay service that swaps models: the circuit size of large-model inference still exceeds the capabilities of current proof systems, and establishing the root of trust requires participation from multiple parties. But they point in the right direction. In future model services, identity information should not be just a string that anyone can fill in. It should be a cryptographic credential that anyone can independently verify. For developers and enterprises, two actions can start now: manage model versions carefully in system design, and write verifiability into procurement standards.