Can You Verify Which LLM Generated an Answer? The Principles and Limits of Zero-Knowledge Proofs

Developers who call large language model APIs often face a basic question: did the response actually come from the model they paid to use? A service provider could route the request to a cheaper model and return the result unchanged. The user sees only a block of text and has no visibility into what happened behind the API.

Why Asking the Model Directly Is Not Reliable

The most obvious approach is to ask the model, “Which version are you?” This provides almost no meaningful evidence. A service provider can place an identity instruction in the system prompt, and the underlying model will usually follow it. The model’s self-description reflects the prompt it received, not the provenance of the computation.

Limitations of Existing Detection Methods

Common detection approaches generally fall into three categories: style comparison, response-field inspection, and reasoning-trace analysis.

Style comparison depends on human judgment and can produce false conclusions when the sample size is small. Inspecting response fields only shows what the server returned, and those fields can be constructed or modified. Reasoning-trace analysis may indicate that an answer came from a particular type of service cluster, but it is difficult to associate the result with one exact model version.

When users suspect that a model has become weaker, these methods usually cannot provide a conclusion that others can independently reproduce and verify.

The Basic Idea Behind a Zero-Knowledge Proof Solution

A zero-knowledge proof, such as a zkSNARK, is a cryptographic protocol that allows a prover to demonstrate that a computational relationship is valid without revealing the private inputs used in the computation.

Applied to large language models, the basic idea is as follows:

1. The model provider commits to the model weights.
2. For each inference request, the provider generates a proof.
3. The proof demonstrates that, under the weights represented by the commitment, the specified input and sampling parameters produced the claimed output.
4. A third party uses the public verification key and commitment value to check the proof.

The following table compares three common approaches:

Approach	What it proves or guarantees	Main trust assumptions	Cost and latency	Practical boundary
zkSNARK proof	The output was computed from the committed weights according to the specified computation	The commitment registry and verification key are trustworthy	Proof generation is expensive and grows with model size	Suitable when independent third-party verification is required
Trusted Execution Environment (TEE)	Inference ran inside protected hardware	The hardware vendor and remote-attestation mechanism are trusted	Usually lower than proof generation, but depends on specific hardware	Hardware vulnerabilities and side-channel risks remain
Style fingerprints and statistical detection	The output distribution resembles historical samples	Historical samples are representative	Relatively low cost	Useful for screening, but not conclusive evidence

What Happens in the Three Stages

Step 1: Commit Before Inference

The commitment is created before inference and applies to the model itself, not to an individual response. The model provider must publish the commitment value for the model weights in advance and register the model architecture version, circuit version, and tokenizer version.

The commitment is typically a cryptographic digest of the weights. It should not reveal the weights or allow outsiders to reconstruct them.

When a request arrives, the service records the input, decoding parameters such as temperature and top_p, and the random seed. The commitment does not contain the output that has not yet been generated. That output does not exist at this stage.

Step 2: Generate the Response and the Proof

The model generates an output token sequence using the recorded parameters. After generation, the proving system creates a zero-knowledge proof for the statement that the committed weights, input, sampling parameters, and output sequence satisfy the required computational relationship.

The output becomes part of the binding only after it has been generated.

If sampling is random, the service must preserve the random seed. Without it, a third party cannot reproduce or verify the relevant computational relationship.

Step 3: Independent Verification

The verifier receives the response, proof, commitment value, and verification key, and then runs the verification program. The result is either valid or invalid.

A valid result means that, under the registered weight commitment and circuit description, the output satisfies the computational relationship for the specified input and parameters.

It does not, by itself, prove that the service provider never changed models. That conclusion depends on the commitment registration process and the source of the verification key being trustworthy.

Requirements Before Production Deployment

The conceptual workflow is straightforward, but a production implementation must address several additional issues:

1. Registration of weight commitments: Who publishes the commitment value? What prevents the service provider from replacing it after registration? The rules must be explicit.
2. Circuit representation of inference: The model’s forward pass must be converted into a form that the proving system can process. This usually requires instrumentation or rewriting parts of the computation graph.
3. Verification keys and trusted setup: Some proof systems require a one-time trusted setup. The participants and the key-destruction process must be publicly documented.
4. Deterministic sampling: Temperature, random seeds, and floating-point operation order can all affect reproducibility. A common specification is required.
5. Version consistency: The tokenizer, circuit version, and model weight version must match during verification.
6. Proof storage: Historical proofs must be retained to support offline review and later auditing.

Any pseudocode used to illustrate this workflow would show only the relationship between the calls and would omit all of these details. It should not be treated as production-ready code.

Benefits and Appropriate Use Cases

The main value of this approach is that verification no longer depends solely on the service provider’s statement. A third party can run the verification process independently, and historical proofs can be checked offline. Even if the service is later discontinued, the result can still be reviewed as long as the proof and commitment have been preserved.

The model weights can remain private. A verifier does not need access to the weights, and the model provider does not need to publish them to provide evidence.

The scope of the solution must also be clear. The provider has to integrate the proving process and actively issue proofs. If the provider refuses to do so, an unproven response cannot receive a verification conclusion through this mechanism.

Main Limitations

First, the proof covers only inference that passes through the proving pipeline. If the provider invokes another model outside that pipeline or uses an unregistered commitment, the mechanism cannot detect it.

Second, the proof establishes whether a computational relationship holds. It says nothing about model quality. Whether the answer is correct, biased, or weaker than before requires separate evaluation methods.

Third, cost and latency remain practical constraints. Proof-generation time and hardware overhead increase with model size. The actual figures depend on the model, circuit implementation, and hardware environment, so they must be measured in the target deployment rather than inferred from general assumptions.

Fourth, the trust chain depends on commitment registration and any trusted setup. If the registration process is missing or the verification key comes from an unreliable source, the evidentiary value of the proof is significantly reduced.

Implications for Different Stakeholders

For model providers, publishing commitments and supplying proofs can provide verifiable material when users question which model was used. Once a commitment is public, replacing the weights requires a new registration, which also affects operational procedures.

For end users, critical applications can require the provider by contract to issue proofs and retain the response together with its proof. These records can then be checked if a dispute arises.

For regulators and auditors, proofs can serve as one type of audit evidence. However, common standards are needed; otherwise, proofs from different providers will be difficult to compare.

Selection Guidance

If the goal is only to detect obvious model substitution, lower-cost statistical methods and periodic sampling can be used for initial screening. Proof-based verification is more appropriate when the value of the application justifies the additional engineering and computing cost.

Before selecting a technology, define what must be proven, who is trusted, and who will make the final decision in a dispute. The technical choice should follow those requirements.

Conclusion

Zero-knowledge proofs provide a way to obtain verifiable evidence about which model computation produced an answer. However, the evidence depends on a clearly defined set of assumptions, including model commitments, verification keys, inference circuits, sampling behavior, and registration procedures.

Understanding these assumptions is essential for judging the strength of the evidence and for using the technology appropriately in system design.