Zero-Knowledge Proofs for LLM Output Authenticity: What They Can and Cannot Prove

Why LLM Output Authenticity Matters

After large language models are integrated into business systems, the request path is rarely a direct connection between a user and an official inference endpoint. Requests often pass through multiple gateways, aggregation APIs, and internal routing layers.

This improves integration efficiency, but it also creates a trust gap. A caller may be unable to determine whether the returned text actually came from the model version they purchased, whether it was replaced with a cheaper model, or whether a smaller model produced through secondary distillation was presented as the original model.

Traditional approaches are not sufficient.

One approach relies on a model’s self-declaration, such as asking, “Which model are you?” This can be changed easily through a system prompt. An intermediary only needs to add an instruction to produce a false answer.

Another approach compares writing style by sending a fixed set of prompts and checking whether the answers appear consistent. However, many models have similar styles. Different model sizes, quantization levels, or models from the same provider may share recognizable characteristics, making style-based comparisons unreliable.

A third approach relies on response headers, such as a returned model name. But this field is written by the intermediary. It only shows what the intermediary is willing to claim; it does not prove that the computation actually ran on that model.

Zero-knowledge proofs provide a cryptographic path for addressing this problem.

They allow a prover to demonstrate that an output satisfies an agreed computational relationship without revealing internal state, model weights, or the full inference process. The word “agreed” is essential: a proof can cover only the computational relationship encoded in its constraints.

The narrower the scope of the proof, the less it can establish. The broader the scope, the more expensive proof generation becomes.

The Three-Step Verification Process

A typical design consists of three stages:

1. Commitment
2. Proof generation
3. Verification

1. Commitment

After receiving an input, the inference service encodes the model identifier, model-weight hash, prompt digest, and output text into a record. It then computes a commitment for that record.

A commitment needs two properties:

• Binding: The prover cannot later associate the same commitment with a different output.
• Hiding: The commitment itself should not reveal the input or output content.

A hash commitment with a random salt can generally provide both properties, relying on the collision resistance and preimage resistance of the hash function.

A Pedersen commitment provides binding under discrete-logarithm assumptions and perfect hiding.

Regardless of the commitment scheme, encoding rules must be fixed. This includes field order, byte order, delimiters, and version numbers. Otherwise, the same content may produce different digests, causing verification failures.

2. Proof Generation

The proof circuit describes the consistency constraints between the commitment and the output. In practice, there are two common trade-offs.

Full Inference Proofs

A full inference proof places every step of the model’s forward pass inside the circuit.

This covers the complete computation process. However, for current large language models, proof generation is extremely expensive. It is generally practical only for small models or offline audits.

Partial Constraints

A partial-constraint approach proves consistency for selected public values, such as:

• Output hash
• Truncation length
• Temperature
• Stop conditions
• Random seed

The actual computation is delegated to other mechanisms.

This approach has manageable cost, but it can only prove that these public parameters are consistent with the claimed output. It cannot prove that the output was actually computed using the specified model weights.

If an intermediary controls the inference environment, it can generate text outside the constrained computation and still attach a syntactically valid proof.

3. Verification

The verifier needs the commitment, public parameters, proof string, output text, and verification key.

Verification does not require access to model weights or another model invocation. It is generally much faster than rerunning inference, but actual performance depends on the proof system, circuit size, and hardware environment. Measurements should be performed in the target environment rather than relying directly on promotional figures.

Several types of cryptographic material must be distinguished:

• Proving key: Held by the prover and used to generate proofs.
• Verification key: Publicly distributed and used to validate proofs.
• Witness: Private data known by the prover during computation.

A verifier uses the verification key but cannot read the witness.

What Zero-Knowledge Proofs Can Detect

The ability to detect fraud depends on the trust assumptions behind the model registry and proof environment.

Model Substitution

A gateway may route a premium request to a lower-tier model and return the result unchanged.

If the commitment binds a model-version hash and that hash comes from a trusted registry maintained by the model owner, a substitution causes verification to fail.

However, this depends on the prover being unable to keep using the original registered fingerprint after substitution. If the prover controls both the model and the proving environment, it may create commitments consistent with the registered fingerprint for an arbitrary model. In that case, substitution resistance fails.

Silent Downgrades

A service may switch quantization levels under the same model name or replace the model with a distilled smaller model.

If the version fingerprint changes, the commitment will no longer match the registered value, allowing verification to detect the change. This again requires the registered fingerprint to come from a trusted source.

Distillation-Based Impersonation

An external model may imitate a product’s style and claim to be that product.

Without weights corresponding to the registered fingerprint, it cannot generate a commitment that passes verification under that fingerprint. This conclusion also depends on the trustworthiness of both the registry and the proof mechanism.

What Proofs Cannot Prove

A proof only establishes that an output is consistent with the claimed computational relationship.

It cannot independently prove that a model is sufficiently capable, that an answer is correct, or that the model meets a particular quality bar. Capability assessment still requires evaluation datasets.

Authenticity mechanisms address identity and integrity. They do not solve model-quality ranking.

Define Trust Assumptions Before Deployment

Before implementation, the trust assumptions should be documented explicitly. Common approaches include the following, which can be used separately or combined:

1. The model owner publishes and signs registered model-weight hashes.
2. The inference environment runs inside a trusted execution environment, with remote attestation provided by hardware.
3. The proof circuit covers full inference, or covers sampled constraints from critical layers combined with randomized audits.

Each approach has limitations. The appropriate choice depends on the threat model and cost budget.

Recommended Engineering Architecture

A layered design is recommended.

Integration Layer

Attach commitment and proof fields to the response body while maintaining compatibility with existing chat interfaces. Older clients can ignore these fields.

Gateway Layer

The gateway can perform full verification or sampled verification.

High-value requests should be verified in full. Batch workloads can use sampling to balance latency and cost.

Storage Layer

Store commitments, proofs, and request metadata in object storage or a logging service to support later investigation and accountability.

Streaming Responses

For streaming output, each chunk can receive an incremental commitment, followed by a final aggregate proof at the end of the response. This helps avoid excessive first-token latency.

Align Circuit Constraints With Business Semantics

Circuit constraints must match the semantics of the actual application.

Temperature, top-p, maximum output length, stop sequences, and tool-call results should all be included as public inputs. Otherwise, an attacker may alter these parameters while keeping the text hash unchanged.

Deterministic decoding is easier to prove because the output depends only on the input and model weights.

For sampling-based decoding, the random seed must be included in the commitment, and the seed source must be verifiable. Otherwise, the prover may select a favorable seed after the fact.

Tool Calls and Multimodal Inputs

Tool calls require additional handling.

If a tool result enters the public input only as a hash, the caller still cannot determine whether an intermediary altered that result. The tool service should sign its returned value. Verification should check the signature, and the signature digest should be included in the commitment.

Multimodal scenarios require the same discipline. Image or audio content hashes must be bound to the proof so that the text cannot pass verification while the attached media is replaced.

Replay Protection

Replay protection must cover the proof itself.

A proof string can be copied and reused for another request. The commitment should therefore include:

• Request identifier
• Timestamp
• Caller identity
• Expiration period

The verifier should confirm that the timestamp falls within an allowed window and reject repeated request identifiers.

Clocks between the verifier and prover must be synchronized; otherwise, expiration checks may produce inconsistent results.

Performance and Side-Channel Considerations

Performance and security are parallel constraints.

Proof generation is usually more expensive than a single forward inference pass. It may be completed asynchronously or delegated to edge nodes.

Verification must remain lightweight enough to run in browsers or mobile clients.

Circuit implementations should use execution paths that are independent of private data whenever possible, reducing the risk that attackers infer internal structure from proof-generation timing.

Key Management

Key management must distinguish between proving keys and verification keys.

Both should be stored through a key management service. When proving keys are rotated, updated verification keys must be distributed accordingly, and compatibility rules for historical proofs should be clearly defined.