Zero-Knowledge Proofs for LLM Output Verification: How Proof-of-Inference Works and How to Deploy It

1. The core problem: a swap that happens in the dark

In a typical LLM deployment, one or more gateways sit between the client and the actual inference service. The gateway handles authentication, rate limiting, billing, and protocol translation, and in the process gains complete control over both requests and responses. The user sees “I called a flagship model.” The gateway sees “I can route this request to any model I can reach.”

That is an information asymmetry: the price is set by one side, the model actually used is decided by the other, and nothing enforces consistency between them. Several failure patterns follow:

• Model substitution: forward a flagship-model request to a smaller, cheaper model, but bill at the flagship rate.
• Silent downgrade: switch to more aggressive quantization, a shorter context window, a cheaper decoding strategy, or quietly roll back to older weights.
• Distillation impersonation: serve weights distilled or fine-tuned from other models while marketing them as a flagship release.

The verification methods commonly used in the community are mostly heuristics:

Method	What it actually proves	Where it breaks
Ask the model “what are you?”	Almost nothing	A single identity claim in the system prompt is enough to fake it
Inspect response fields and metadata	The response conforms to some protocol format	Fields can be forged completely
Compare style, verbal tics, refusal boundaries	Statistical similarity	Prompt alignment, post-processing rewrites, sampling noise
Verify a signature on the inference trace	The response came from some legitimate upstream	Cannot distinguish the flagship from a smaller model in the same family
Fixed test suite with regression scores	Relative quality drift over a period	Requires labels and baselines; lagging and expensive

The trust gap is even more awkward because it runs in both directions. Users suspect providers of substitution, and providers accused of silently degrading their models have no cryptographic way to clear themselves. Both sides can offer only statistical evidence and subjective experience, with no reproducible basis for arbitration.

2. The core idea: replace trust with a verifiable mathematical statement

The goal is not to make an answer “look more genuine.” It is to restate the claim as a statement that can be independently verified:

There exists a set of weights w whose cryptographic commitment equals the value C I published in advance; running the forward computation with those weights under the declared inference configuration on this input produces exactly this response.

This statement becomes verifiable thanks to a few basic properties of zero-knowledge proofs (here, the common succinct non-interactive argument of knowledge construction, typically called a zkSNARK):

1. Completeness: if the server actually executed honestly, it can always produce a proof that verifies.
2. Soundness: if it substituted the model, it cannot produce a passing proof without breaking the underlying cryptographic assumption.
3. Zero-knowledge: verification does not reveal the weights, so the model asset is not exposed.

One boundary needs to be stated clearly: what is proven is the provenance and consistency of the generation process, not the quality of the answer. It answers “was this computed with the weights I paid for,” not “is this answer correct.” Leaving that distinction blurred invites over-interpretation.

3. The three-phase flow: commit, challenge, prove

Step 1: Commitment — freeze the model identity into a short digest

When a provider releases a model version, it first publishes a cryptographic commitment over everything that influences the output:

# Illustrative: the commitment input should cover every factor that affects the output
commitment_input = {
    "weights_hash": sha256_tree(weights_tensors),   # Merkle digest over the weight tensors
    "config": {
        "dtype": "bf16",             # numeric precision, blocks silent quantization
        "quantization": None,        # quantization scheme
        "context_window": 200000,    # context limit
        "top_p": 1.0, "temperature": 0.0,
    },
    "model_version": "2026-02-release",
    "inference_graph_hash": sha256(computation_graph),
}
C = commit(commitment_input)    # commitment value: short, public, not invertible to the input
publish_to_transparency_log(C)  # append to a public, auditable append-only log

The commitment scheme must satisfy two properties at once: binding (the provider cannot later reinterpret the commitment as a different set of weights) and hiding (it reveals nothing about the weights). Once that holds, the model identity stops being a vendor assertion and becomes a public record anyone can check. Every later version change must produce a new commitment, and historical versions cannot be quietly edited.

Step 2: Challenge — make proofs impossible to pre-generate

Proving only that “some output corresponds to some weights” is not enough. An attacker could pre-generate proofs offline and serve them on demand. So the proof has to bind the unpredictable elements of the current session into the public inputs:

challenge = {
    "nonce": client_random_bytes(32),   # generated by the client; unpredictable to the server
    "request_hash": sha256(canonical_json(messages)),
    "session_id": session.id,
    "timestamp": now_utc(),
}

Because the client supplies the nonce and folds the request content and timestamp into the challenge, the server cannot prepare proofs for requests that may arrive in the future, nor reuse the proof of one real inference for a different request.

Step 3: Prove and verify

The server expands the forward computation into a set of arithmetic constraints (a circuit) and produces a proof. The verifier needs only the public verification parameters, the commitment C, and the public inputs (the challenge and the output hash):

# Server side
proof = prove(
    witness = {"weights": w, "activations": trace},   # private; never leaves the network
    public  = {"commitment": C, "challenge": challenge,
               "output_hash": sha256(response)},
)
# Client / third-party auditor
assert verify(proof, public_inputs)   # cost is orders of magnitude below proving cost

The biggest engineering payoff is that verification is cheap: proving is a one-off heavyweight job, while verification can run frequently on the client, at an auditor, or in a CI pipeline. That is also what makes “third-party escrow of evidence plus after-the-fact arbitration” a viable business model.

4. Threat model: what it stops, and what it does not

The boundaries have to be listed honestly, or the mechanism will be misused.

What it can effectively detect:

• Model substitution: different weights yield different commitments, and the proof fails.
• Silent downgrade: if quantization, context length, and sampling parameters are in the commitment, changing any of them violates the constraints.
• Distillation impersonation: distilled weights do not match the official commitment, so the claim contradicts the proof.
• Version rollback: older commitments are already timestamped in the transparency log and can be checked against the timeline.
• Response tampering: the output hash is a public input, so a single edited character in a relay fails verification.
• Pre-generated and replayed proofs: the challenge binds the client nonce and the request content.

What it does not cover:

• Trust in the commitment itself: if the published weight hash does not correspond to the model the vendor advertises, the proof will faithfully prove only that “this is the thing I committed to.” The root of trust still has to rest on the release process, reproducible builds, and multi-party witnessing.
• Prompt layer and pre/post-processing: system prompt replacement, content-filter rewrites, formatting truncation, and log redaction all happen outside model inference and outside the scope of the proof. The prompt template should be included in the commitment, and the pre/post-processing pipeline audited separately.
• Retrieval and tool calls: retrieved documents and external tool returns are part of the input and are proven to be used faithfully, but their accuracy and freshness are not proven.
• Sampling randomness: if non-zero temperature is allowed, an explicit and reproducible source of randomness must be agreed on, otherwise the same input has no uniquely determined output.
• Availability and compliance: denial of service, content safety, and the legality of data sources are all separate issues.

5. Five hard problems in real deployments

1. Performance and proving overhead

Proving costs far more than inference, so proving every response is unrealistic in most settings. The workable path is tiered sampling: full proofs for high-value requests (core production paths, paid audit scenarios), low-rate sampling for ordinary traffic, plus lightweight client-side consistency checks for routine monitoring. The sampling rate itself should be published, so that “we appear to verify” does not quietly mean “we barely verify.”

2. Deterministic inference

This is the easiest place to get burned. Non-associativity of floating-point arithmetic, batch composition order, differences between operator implementations, and concurrent reduction strategies can all make the same weights on the same input produce different bit-level results. For proofs to hold, the inference stack must fix the precision and quantization scheme, fix the operator implementations, fix how batching and sequences are organized, and use an explicit random source. That conflicts directly with dynamic batching in the name of throughput, so a “verifiable mode” usually needs its own inference path, trading some performance for provability.

3. Commitment version management and revocation

Commitments are not one-off artifacts. Model iteration, security patches, and emergency rollbacks all need a lifecycle: publication and effective dates for new commitments, freezing and revoking of old ones, append writes and consistency checks against the transparency log, and a queryable map of which commitment was valid for which time window. Without this, proofs create confusion instead of removing it.

4. Distribution of verification parameters and the root of trust

Verification keys must be obtainable by independent third parties and verifiable in origin, otherwise the attack surface simply moves from forging responses to swapping verification keys. The usual answer is a publicly auditable parameter-generation process with multi-party witnessing, plus independent mirrors and digest distribution.

5. Observability

A proof only has value if it can be traced. Log the proof together with the request ID, session ID, and commitment version: store evidence files in object storage with versioning and access auditing enabled, send pipeline logs to a log service for search and retention, and add monitoring and alerting for verification failure rate, sampling coverage, and commitment version distribution. On Baidu AI Cloud, for example, a typical combination of object storage, log service, and monitoring and alerting gets this evidence chain running without any additional cryptographic infrastructure.

6. Practical advice for enterprises and developers

Procurement and contracts: turn verifiability from a trust statement into a contractual requirement, such as public model-version commitments, advance notice of version changes, statements or proofs that can be verified independently, and defined remedies when verification fails. This is more binding than any style test.

Self-hosting and private deployment: implement a verification gateway at the inference layer. It generates challenges for inbound requests, attaches proofs and commitment references to outbound responses, and leaves verification to the client SDK. Start with a narrow pilot (one business line, one model) so deterministic-inference compatibility problems surface in a small scope.

Monitoring: even before adopting full cryptographic proofs, build a baseline regression system with a fixed test suite, fixed scoring criteria, before/after comparison around version changes, and alerts on abnormal movement. It cannot prove that the weights are identical, but it locates the time window of a sudden quality change quickly and complements strong proofs.

Expectations: do not treat verifiable inference as a compliance cure. Legitimacy of data sources, labeling of generated content, content safety review, and cross-border data flows are separate issues, each needing its own mechanism.

7. Conclusion

The “real vs. fake” debate over LLM outputs is fundamentally a missing trust infrastructure. Leaving answer correctness to prompt wording and provenance to verbal assurance means guessing. What cryptographic proofs contribute is not a smarter answer but a shift from “I said so” to “you can check it yourself.”

They will not eliminate model substitution overnight, because the performance cost, the deterministic-inference constraints, and the root-of-trust work are real barriers. But they change one crucial premise: from “please believe me” to “please verify me.” For developers who pay for inference, enterprises that need to audit, and model providers who keep getting accused of silent degradation, that is infrastructure work worth investing in.