Verifying LLM Identity via API: A Behavioral Fingerprinting Approach

1. Background and Scope

Many teams consume large language models through third-party APIs. The model version promised in the contract may differ from what the backend actually executes. Some providers claim to serve a top-tier closed-source model while actually routing requests to a smaller open-source model or a fine-tuned variant. Others quietly switch to cheaper versions during peak hours.

API consumers have no access to weight files, so hash comparison and structural metadata are unavailable. Judgment must rely solely on model behavior—the relationship between inputs and outputs. This approach has three clear boundaries:

• The result is evidence of consistency, not absolute proof. Minor version upgrades, quantized deployments, or system prompt changes by the same vendor can all alter output characteristics.
• Fine-tuned variants often behave similarly to the base model on many probes and may be judged as a match.
• Conclusions depend on the probe set. Changing the question bank can change the verdict.

For this reason, reports should state “high likelihood of consistency with the baseline” or “deviation from baseline, manual review required,” rather than asserting a specific model has been confirmed.

2. Three Signal Types and Their Applicability

Signal	API requirement	Cost	Main false-positive risk	Applicable scenarios
Output entropy and probability distribution	Endpoint returns logprobs or top_logprobs	Low, computed locally	Temperature, top_p, prompts, and length changes all alter entropy	APIs exposing token probabilities
Semantic embedding position	Only response text needed	Medium, requires an embedding model	Stylistically similar models are hard to distinguish; prompt template changes have large impact	Most black-box APIs
Watermark detection	Provider publishes the watermark scheme and cooperates with detection	High, depends on cooperation	Unusable when the scheme is undisclosed; rewriting and translation weaken watermarks	Scenarios with signed cooperation agreements

Output entropy. Aligned models typically produce more concentrated outputs with lower entropy. However, this rule only holds when conditions are identical. Fix temperature, top_p, system prompts, and maximum output length, and normalize for response length. Otherwise, raising the temperature by one notch may shift entropy more than the difference between models.

Semantic embedding position. Convert responses to fixed probes into vectors and compare their positions in the embedding space. It depends only on text, so it applies broadly. But it is sensitive to prompt templates—use exactly the same template for every test run.

Watermark detection. Watermark verification requires provider cooperation and a disclosable watermark scheme. If the provider does not publish the detection method, the verifier cannot construct effective trigger probes. It works best as supplementary verification between cooperating parties, not as independent evidence against an external provider.

3. Probe Set Design

Probes are the foundation of verification. The following parameters are recommended:

• Task types: reasoning, common sense, math, code, Chinese writing, and instruction following—six categories in total.
• Volume: 40 questions per category, roughly 240 questions overall. Use stratified sampling by category for each session, drawing 15 questions per category, about 90 questions per run.
• Repetitions: send each probe 3 times and average the results to reduce sampling variance.
• Confidentiality: keep the question bank private, store questions and reference answers separately, and log access. Rotate the bank periodically to reduce the risk of targeted optimization.
• Verdict records: for each session, save the timestamp, the claimed model name, probe IDs, response text hashes, entropy statistics, embedding vectors, similarity scores, and conclusions.

4. Feature Extraction Code

The following code demonstrates the full pipeline: fixing sampling parameters, requesting logprobs, computing entropy statistics, extracting embedding vectors, and calculating similarity. The API URL and key are passed via environment variables rather than hardcoded. Python 3.10 or higher is required.

import math
import os
import numpy as np
import requests
from sentence_transformers import SentenceTransformer
API_URL = os.environ["PROBE_API_URL"]   # OpenAI-compatible chat endpoint
API_KEY = os.environ["PROBE_API_KEY"]
MODEL = os.environ["PROBE_MODEL"]       # The model name claimed by the provider
TIMEOUT = 60
embedder = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")  # Fixed once chosen
def ask(prompt: str) -> dict:
    """Send one probe with fixed sampling parameters and request logprobs."""
    payload = {
        "model": MODEL,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
        "top_p": 1,
        "max_tokens": 256,
        "logprobs": True,
        "top_logprobs": 5,
    }
    headers = {"Authorization": f"Bearer {API_KEY}"}
    resp = requests.post(API_URL, json=payload, headers=headers, timeout=TIMEOUT)
    resp.raise_for_status()
    return resp.json()["choices"][0]
def entropy_features(choice: dict) -> dict | None:
    """Compute mean entropy, top-1 token probability, and related statistics. Returns None if logprobs are unavailable."""
    steps = (choice.get("logprobs") or {}).get("content")
    if not steps:
        return None
    entropies, top1, chosen = [], [], []
    for step in steps:
        tops = step.get("top_logprobs") or []
        if not tops:
            continue
        probs = np.array([math.exp(t["logprob"]) for t in tops])
        probs = probs / probs.sum()  # Normalized only within top-k, so entropy is approximate
        entropies.append(float(-(probs * np.log(probs + 1e-12)).sum()))
        top1.append(float(probs.max()))
        chosen.append(math.exp(step["logprob"]))
    if not entropies:
        return None
    return {
        "mean_entropy": float(np.mean(entropies)),
        "mean_top1_prob": float(np.mean(top1)),
        "mean_chosen_prob": float(np.mean(chosen)),
        "n_tokens": len(entropies),
    }
def run_session(probes: list[str], repeats: int = 3) -> dict:
    """Run one session over all probes and return the feature vector and entropy statistics."""
    texts, feats = [], []
    for prompt in probes:
        for _ in range(repeats):
            choice = ask(prompt)
            if choice.get("finish_reason") != "stop":
                continue  # Discard responses truncated by length or ended abnormally
            texts.append(choice["message"]["content"])
            f = entropy_features(choice)
            if f is not None:
                feats.append(f)
    vector = embedder.encode(texts, normalize_embeddings=True).mean(axis=0)
    entropy = None
    if feats:
        entropy = {k: float(np.mean([f[k] for f in feats]))
                   for k in ("mean_entropy", "mean_top1_prob", "mean_chosen_prob")}
    return {"vector": vector, "entropy": entropy, "n_texts": len(texts)}
def cosine(a: np.ndarray, b: np.ndarray) -> float:
    """Compute cosine similarity between two vectors."""
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

The cosine function compares the current session’s embedding vector against a stored baseline vector. Combined with the entropy statistics, it provides the evidence needed to flag sessions for manual review when outputs drift from the baseline.