Identifying Anonymous LLMs: From Style Fingerprints to Capacity Inference
An anonymous model goes live at an evaluation endpoint, and the community instantly splits into contradictory camps over where it came from. This article argues that most such guesses lean on evidence that is weak and unfalsifiable. It ranks the available evidence by strength, explains why "collective hints," style-similarity heatmaps, and free-quota reasoning fail as proof, and lays out the technical fingerprints that actually carry discriminating power—tokenizer behavior, decoding response curves, safety templates, and stable capability weaknesses. It closes with a reproducible, double-blind experimental design and a piece of engineering advice: identity is guessed, capability is measured, and model selection should rest on the latter.
A familiar Rashomon-style dispute keeps replaying in tech circles. An anonymous model goes live at an evaluation endpoint, and almost the entire internet concludes it comes from a certain Chinese open-source model family. Then a cluster of accounts tied to that model’s ecosystem collectively hint that it is actually a new lightweight model from a major overseas vendor. After that, someone digs up a style-similarity report from several months ago, pointing out that a certain Chinese model’s fingerprint was, at the time, actually closer to that overseas vendor’s family. Three mutually contradictory conclusions, and none can convince the others.
The real value of this dispute isn’t “who guessed right,” but that it exposes a common problem: when judging a model’s identity, most people rely on clues with very low evidential strength that cannot be falsified. Below, the available evidence is ranked by strength and broken down one by one.
1. “Collective Hints” Are the Weakest Form of Evidence
When dozens of accounts simultaneously drop the same hint, it looks impressive, but methodologically it is the weakest tier.
- Role mismatch between speakers and verifiers: Statements from platform operations, developer relations, and product marketing roles are usually constrained by disclosure timing; they can only say half and leave half. But real determination requires raw logs, weights, or the tokenizer—things they can’t provide. So there is a structural gap between the “hints” outsiders hear and “verifiable facts.”
- Low-cost assertions + attention amplification: A hint with a question mark costs almost nothing, yet it gets cited repeatedly in communities and refined level by level until it becomes “everyone says so.” Most people forwarding it did no testing—they just re-guessed someone else’s guess.
- More speakers doesn’t mean a more reliable conclusion: Dozens of accounts voicing an opinion are statistically a highly correlated sample, not independent samples. They are more like copies of one piece of information across many nodes than multiple independent observations.
- “Yesterday no one thought so, today it changed” is classic anchor shifting: Once someone offers a new narrative, all subsequent tests are contaminated by this prior; testers unconsciously search for evidence that supports it.
So the first step is simple: separate “who said it” from “can it be verified.” The former is only a clue source; only the latter enters the evidence chain.
2. Style Similarity Heatmaps: Why They Are Only Weak Evidence
The widely cited chart isn’t hard to understand. It builds a matrix with mainstream models on both axes, and each cell holds a “surprisal” value: taking model A as the “reader” of model B’s outputs, if the text reads awkwardly and has low probability, the color skews cool; if it reads smoothly and matches expectations, the color skews warm. The logic is: if a model is trained heavily on another model’s outputs, their writing habits converge strongly, so the cells naturally turn warm.
Why distillation really does leave traces: Style distribution is a projection of the training data distribution. Feeding the same batch of high-quality Q&A data repeatedly to different models transfers not just knowledge but also sentence rhythm, paragraph organization, list preferences, code-block language annotation habits, and even formatting inertia like “conclusion first, details after.” These are surface statistical features, more easily shaped during the alignment stage than factual knowledge.
Why same-origin models can still look dissimilar:
- Alignment strategies vary widely. The same base paired with different instruction-tuning data and different reward models can produce outputs so stylistically different that their kinship is invisible to the naked eye.
- Decoding parameters and context templates are hidden variables. Temperature, top-p, repetition penalty, system prompt, and history truncation strategy all significantly change surface style.
- Sampling noise. Ask the same model the same question twice, and the style similarity may not be higher than its similarity to another model. The randomness of a single sample is often misread as an inter-model difference.
- Evaluator bias. The model used as the “reader” has its own word preferences and perplexity characteristics; different evaluators give inconsistent scores for the same pair of models.
More critically, versions aren’t comparable. A report from months ago tested the model version of that time, and model iteration cycles are usually measured in weeks; quantization schemes, context length, and inference configuration may all have changed. Using an old chart to explain a new model is like using last year’s physical exam report to judge today’s health.
Conclusion: Style similarity is exclusionary evidence (it can rule out “completely unrelated”) but not confirmatory evidence (it can’t prove “it’s definitely this one”). Any claim that treats it as ironclad proof is logically untenable.
3. Capacity, Throughput, and Free Quotas: The Most Misread Line of Reasoning
Another popular inference goes like this: a given endpoint offers a large daily free quota, and a certain Chinese model’s service scale simply couldn’t afford it, so it must be a better-funded overseas vendor with a larger cluster. This line sounds quantitative, but it forcibly chains together several unrelated links.
- Free quota ≠ inference cost. Platform subsidies, spot instances, idle nighttime compute, and internal resource-pool accounting can all decouple “nominal quota” from actual cost.
- The platform side may be routing, not a single backend. Aggregator endpoints, shadow traffic, and A/B splitting all mean the model behind the same endpoint can change over time; the quota users see has no necessary link to model ownership.
- Cost is heavily affected by engineering techniques. Quantization, speculative decoding, prefix caching, and batch scheduling can each push per-token cost down by an order of magnitude; you can’t simply reason backward from “whoever has more money can afford it.”
- Rate limiting is a dynamic policy. Quotas allocated by account, IP, time period, and request length adjust at any time; the observed quota can’t be treated as a long-term commitment.
What’s actually worth testing isn’t “how much quota was given,” but what the same budget buys: first-token latency and output throughput at different lengths, the degradation point of ultra-long context, concurrency limits, failure rate and retry semantics, and the actually usable context window length. These are a capability profile, far more useful than “guessing who it is.”
4. Technical Fingerprints: Tests Worth Taking Seriously
If you really want to narrow the range, the following methods have clearly higher discriminating power than style heatmaps.
Tokenizer fingerprint (best value) — Construct a set of “stress probe” strings: many rare characters, long digit sequences, mixed Chinese-English with full- and half-width punctuation, code snippets, multibyte characters. For the same batch of text, count the billed input length on the input side (i.e., the number of tokens after tokenization), along with the model’s handling of ultra-long token sequences. Different tokenization schemes usually split the same string into stably different counts, unaffected by temperature or system prompt. This is close to “hardware-level” evidence.
Sampling and parameter response curves — Fix the prompt, sample multiple times under different temperature, top-p, and repetition penalties, and measure output diversity, average length, and repetition rate. Being sensitive to one penalty term but not another—this response shape is a joint fingerprint of the training and decoding stack.
Refusal and safety templates — Send the same set of out-of-bounds or adversarial requests, and record the phrasing, opening words, and whether an alternative suggestion is offered. Safety templates usually come from the same alignment pipeline, and cross-model reuse is less likely than for free-form generated text.
Stable capability weaknesses — Choose tasks models are “naturally error-prone” at: long arithmetic strings, calendar reasoning, character counting, rare-character pronunciation, and unit conversion. The error-position distribution across these tasks is fairly stable for a given model and unaffected by prompt wrapping.
Output format preferences — Code-block language annotation habits, comment language, list bullet style, full- vs. half-width punctuation mixing, and whether a summary paragraph and disclaimer are added by default. These are “muscle memory,” costly to change, so they have good discriminating power.
A caveat: these methods can only give a relative relation of similar / not similar; they can’t individually output a conclusion of “it’s definitely a certain vendor’s certain version.” Reaching a judgment requires an experiment design that can be publicly re-run.
5. Reproducible Experiment Design: Turning “I Think” into “Verifiable”
The standard isn’t “whose intuition is more accurate” but “whose experiment can be re-run and reproduced by a third party.” A minimal viable workflow:
- Fix experimental conditions: unify prompts and decoding parameters, and record timestamps and version numbers, to avoid routing drift across different times contaminating the results.
- Double-blind + cross testing: shuffle sample order, hide sources, and separate the tester from the judge to prevent expectation effects.
- Control group: mix in several models with “known answers” (including same-origin different versions and unrelated models). If the method can’t even separate the known-answer samples, the conclusion is void.
- Repeated sampling for distributions: a single output isn’t evidence; sample at least multiple times and compare distributions, not single texts.
- Distinguish two kinds of conclusions: whether you can distinguish model families (same origin, shared tokenizer, or data lineage) and whether you can distinguish specific versions. The former has a much lower bar, and most disputes actually only reach this level.
- Honestly label confidence: clearly state what the method can rule out and what it can’t prove.
Illustrative test framework (pseudocode; it only expresses the workflow and involves no specific interface):
def fingerprint_experiment(model, probes, repeats=5):"""Returns structured features for comparison; every field can be reproduced by a third party"""report = {"tokenizer": {}, "decode_response": {}, "safety": {}, "capability": {}}# 1. Tokenizer fingerprint: count tokens on stress probesfor name, text in probes["stress"].items():report["tokenizer"][name] = count_tokens(model, text)# 2. Decoding response curves: fix the prompt, sweep temperature and penaltiesfor temp in [0.0, 0.7, 1.0]:outs = [model.generate(P, temperature=temp) for _ in range(repeats)]report["decode_response"][temp] = {"avg_len": mean(len(o) for o in outs),"distinct_ratio": len(set(outs)) / len(outs), # diversity"repeat_rate": detect_repetition(outs),}# 3. Safety templates: record refusal phrasing, not contentfor req in probes["adversarial"]:report["safety"][req.id] = extract_template(model.generate(req.text))# 4. Capability weaknesses: record error-position distributionfor task in probes["hard_tasks"]:ans = model.generate(task.prompt)report["capability"][task.id] = locate_error(task.answer, ans)return report
The comparison stage shouldn’t rely on eyeballing but on a statistical path: for each field, compute the distance between the two sets of results (absolute difference for token lengths, distribution distance for diversity, normalized structural similarity for refusal templates), then compare against the “same-origin distance” and “unrelated distance” distributions from the control group to see which side the tested object falls on.
6. Specific Phenomena That Are Repeatedly Misread
- Old reports vs. new models are incomparable: version iteration can overturn conclusions entirely; any inference spanning months must be re-sampled.
- A single test contradicting itself: the same model can behave like two different models under different prompts, temperatures, and times (routing drift). Two results clashing with each other only shows sampling noise, not that the model has two identities.
- Official “we only make one model” claims get over-extended: a product-line convergence strategy is a business and engineering decision, with no direct mapping to which weights actually run behind an anonymous endpoint.
- Community consensus ≠ independence of evidence: as noted above, this is correlation, not independence, and can’t be used to weight confidence.
7. What to Do Engineering-Wise When Identity Can’t Be Determined
For businesses and dev teams, the most practical conclusion is: identity is guessed, capability is measured, and selection should be based on the latter.
- Build your own probe set: extract the high-frequency input types in your business, label the expected behavior, and use it as a long-term regression baseline. It serves selection, change detection, and fault localization at once.
- Prioritize quantifiable engineering metrics: throughput and first-token latency at different context lengths, information recall in the middle of long texts, compliance rate of structured output (JSON / tables / code), and error rate under concurrency. These depend on no identity inference and map directly to business cost.
- Keep a replaceable architecture: abstract model calls into a unified gateway layer, normalize inputs and outputs, and forbid hardcoding a specific model’s peculiar behavior in business code. That way, however upstream identity or version rotation changes, switching cost stays manageable.
- Productize capability evaluation: enterprise platforms such as Baidu AI Cloud turn model evaluation, service packaging, and monitoring/alerting into standard steps, using object storage for data retention and log services for call tracing, so “switching models” becomes a configuration change rather than a rewrite. This idea itself is unrelated to the identity dispute, but it lets teams escape from guessing.
- Do stratified sampling for evaluation: don’t judge model quality only on standard leaderboards; use private eval sets that match your own traffic distribution to avoid the conclusion bias of public benchmarks.
8. Conclusion: Spend Effort on What Changes Decisions
The real lesson of this dispute is: when everyone is guessing identity, the only thing that can actually be tested is the method itself. Style heatmaps are weak evidence, free quotas are irrelevant variables, and community hints are correlation noise; what truly has discriminating power is token boundaries, decoding response curves, safety templates, and stable capability weaknesses—and even those only narrow the range to the “family” level, unable to pin down the version.
For readers, more valuable than the conclusion is that reproducible evaluation workflow. It can both answer “can this model be used” and alert you the moment an upstream quietly swaps versions. By the time the identity is officially revealed one day, you’ll find you no longer care about the answer—because your system never depended on it.
