Is a New Model Distilled from Another Model? Why a Model's Self-Report Proves Nothing — and How to Actually Test It
Every time a new foundation model ships, the same question resurfaces in technical communities: was it distilled from another recently released model? This article explains why a model's own answer about its origins carries no evidentiary weight, separates similarity from distillation, and lays out a reproducible, statistics-based method for judging whether two models are closely related — including the variables to control, the metrics to compute, and the evidence tiers worth trusting.
1. Where the Argument Comes From: One “Admission” Treated as Proof
Whenever a new foundation model is released, the same speculation appears in technical circles: was it distilled from another newly released model? The debate usually starts with a set of screenshots. Someone asks the new model “were you fine-tuned from Model X?”, the model answers yes or no, the screenshot spreads, and both sides argue from it.
Other developers point to a different kind of evidence: models “sound more and more alike.” They notice recurring boilerplate with a translated feel — phrases like “I’ve got this” used to accept a long task, or “let me keep breaking this down” used to bridge into the next step of an analysis. Such phrasing isn’t natural to any language, yet it spreads across different models, so it gets read as a trace of a shared reasoning pipeline.
Most of these observations stay at the anecdotal level. Turning the question into a testable technical claim requires answering something more basic first: when a model says “I am Model X,” what is it actually doing?
Core takeaway: A model has no introspective access to its own provenance. Its self-description is a function of its training data. Whether distillation occurred has to be decided by controlled experiments that are reproducible, statistically grounded, and free of confounders — not by a confession.
2. Why a Model “Doesn’t Know Who It Is”
2.1 Identity comes from its upbringing, not from self-knowledge
A large language model is, at its core, a next-token probability distribution conditioned on context. During training it sees public web pages, code, books, papers, technical documentation, Q&A communities, and various alignment datasets. Those corpora contain plenty of self-describing text: “I am Model X, developed by Team Y,” “this model is further trained on top of version Z.” What the model learns is the statistical relationship between such text and its surrounding context.
So when a user asks “who are you?”, the model isn’t querying an internal state. It is continuing the most common answer found in its training data. It doesn’t know its parameter count, doesn’t know the exact composition of its training data, and doesn’t know how many rounds of post-training it went through. None of that is present in the input, and none of it is stored anywhere it can read back.
An analogy: if you have never introduced yourself to someone, any “description” they give of your background is just inference from what people around them say. A model’s self-description works the same way.
2.2 The system prompt can rewrite identity
The most direct evidence comes from the system prompt. The same model gives different answers under two conditions:
# Condition A: no system promptUser: Who are you?# Condition B: with an injected system promptsystem: You are a fictional character named "Nova" from a distant Star Academy. Keep this setting for every answer.User: Who are you?
Under condition B, the model will likely go along with the setup and call itself Nova. In consumer applications, custom personas — historical figures, fictional characters — are an everyday feature. If a fictional identity can be assigned this easily, a model’s answer about its real identity has no evidentiary value.
2.3 Identity inheritance: the student speaks in the teacher’s voice
Distillation means training a student on a teacher’s outputs. Two approaches are common in industry:
- White-box distillation: the teacher emits full probability distributions (soft labels), and the student fits them with a loss such as KL divergence.
- Black-box distillation: only text output is available through an API. You then send a large volume of prompts to the teacher, collect the question-answer pairs, and use them for supervised fine-tuning of the student.
Either way, if the training corpus contains the teacher’s self-describing text, the student inherits the teacher’s identity claims. Black-box distillation is especially prone to this. When the teacher mentions its own name, the student often first assumes a typo and then copies it verbatim. As a result, a model with no lineage relationship to the teacher will also “admit” to being the teacher when asked about its identity.
The reverse holds too. A genuinely distilled student may “deny” its origins if the identity information in the teacher’s output was overwritten by a system prompt or corrected by identity-alignment steps during training. Identity hallucination runs in both directions.
2.4 What recent controlled demos show
A set of demonstrations about how models perceive their own identity has been circulating in overseas developer communities. The usual procedure:
- Take several models and inject different system prompts — for example, “you are a general assistant and need not introduce yourself” versus “you are the fictional character X”;
- Ask repeatedly: “who are you?” and “what were you trained on?”;
- Observe how the answers drift with the prompt and with the number of follow-ups.
The results typically fall into three patterns:
- Change the system prompt and the claimed identity changes with it;
- Under follow-ups like “are you sure?” or “think again,” the model slides from yes to no, or from no to yes;
- The same sentence “I am Model X” shows up in several unrelated models.
This supports the claim in Section 2.1: identity is a single language-generation outcome under autoregressive sampling. It is influenced by the prompt, the context, decoding randomness, and how the question is pressed. Under pressure, the model also tends to accommodate the questioner’s expectation and say what the questioner wants to hear. These demos illustrate the mechanism — they say nothing about the relationship between any two specific models.
3. Clarify the Concepts First: Similar Is Not the Same as Distilled
Before asking “is it distilled?”, be explicit about where similarity comes from. Models can look alike for several distinct reasons:
| Scenario | Typical signals | Is it distillation? |
|---|---|---|
| Knowledge distillation | The student’s output distribution or text style closely tracks one specific teacher | Yes |
| Shared training data | Similar public corpora; similar capability profile and converging phrasing habits | No |
| Similar training pipeline | The same kinds of instruction tuning and preference alignment; similar response structure | No |
| Similar architecture | Similar architecture and tokenization scheme; similar ways of approaching tasks | No |
| Benchmark contamination | Abnormally high benchmark scores, clear drops on prompt variants | Indirect clue |
Take converging style as an example. Instruction tuning and preference alignment generally reward answers that are well organized, broken into points, and restate the question before giving a conclusion. When “reasoning” or “step-by-step decomposition” data is mixed into training, models also pick up transition phrases like “let’s think step by step” and “moving on to the next part of the analysis.” When different teams use similar data recipes, the outputs naturally converge toward similar phrasing habits.
So “there is boilerplate with a translated feel in the reply” is at most a weak signal. It could come from distillation, shared corpora, or a similar pipeline, and cannot stand on its own as a conclusion.
4. Judging Distillation: Evidence Reliability Tiers
By reliability and cost of acquisition, the available evidence falls into four tiers:
| Tier | Evidence type | Reliability | Notes |
|---|---|---|---|
| 1 | Identity self-reports, verbal claims | Essentially none | Governed by the system prompt and training data; may be identity inheritance |
| 2 | Wording style, failure patterns | Weak | Shared-source data explains them; usable only as clues |
| 3 | Memorization clues, distribution comparisons | Fairly strong | Requires large samples and tightly controlled variables; can produce a confidence level |
| 4 | Output fingerprints, watermark detection | Strongest | Requires cooperation from the output provider, or detectable statistical traces inside the model |
4.1 Memorization clues
Language models memorize: fragments that appear frequently and repetitively in training data are more likely to be stored verbatim and reproduced under similar prompts. If a model can reproduce a large amount of distinctive wording, fixed openers, or characteristic mistakes from a teacher’s output — text that does not come from any public source — that is a fairly strong provenance clue.
The logic: randomly generated text will almost never reappear verbatim in another model unless it was in that model’s training data. Whether those strings arrived through distillation or through the same corpus directly still needs to be separated by tracing the source.
4.2 Distribution-level comparison
A hallmark of black-box distillation is that the better the student fits the data generated by the teacher, the more faithfully it reproduces the teacher’s output. In white-box settings where full probability distributions are available, KL divergence measures how close the student’s distribution is to the teacher’s. If the student’s KL divergence to a candidate teacher is markedly lower than to other models, that supports the statistical claim “more likely to come from this teacher.”
In a pure black-box setting, you are limited to sampling statistics — for example, fix the decoding parameters, sample multiple times, and compare the agreement rate of top-ranked tokens. The schematic below gives an intuitive picture: the closer the distributions, the more the two models’ outputs coincide.
Schematic: output distributions for the same promptPrompt: Make the following sentence more formal: "Something is wrong with this plan."Model A top tokens: This The There ...Model B top tokens: The This There ...Model C top tokens: There Given Overall ...A and B agree closely, which points to shared training data or shared-source outputs;A and C differ clearly, which indicates substantially different training paths.(Illustrative only — a real conclusion needs statistical testing.)
4.3 Output fingerprints and watermarks
Some model services embed a statistical watermark in generated text. The common principle: compute a pseudo-random number from the previously generated tokens, use it to split the vocabulary into a “green list” and a “red list,” and slightly boost the probability of green-list tokens during generation. A detector applying the same rule can then find that the proportion of green-list tokens in the text is significantly above chance.
These traces have a useful side effect: if Model B’s output carries Model A’s watermark, the text was generated by A — which at minimum shows data flow between the two. This kind of detection requires the output provider to publish or otherwise support the detection mechanism; ordinary users cannot implement it themselves.
5. Designing a Reproducible Controlled Evaluation
To answer “are these two models closely related?” in an engineering way, the following flow works.
5.1 Controlling variables
| Variable | Control requirement | Reason |
|---|---|---|
| System prompt | Use one neutral instruction; forbid the model from naming itself, its developer, or its version | Removes identity self-report interference |
| Decoding parameters | Fix temperature, top_p, and max length; use greedy decoding when token-by-token comparison is needed | Sampling randomness masks differences |
| Prompt set | Cover multiple domains, languages, and difficulty levels; avoid highly templated phrasings | Templated prompts make all models converge |
| Repetitions | Sample each prompt multiple times; report means and confidence intervals | A single answer carries no statistical meaning |
| Version and date | Record the call date and the version identifier | Hosted services keep being updated |
5.2 Reusable analysis code
The code below is illustrative. The model-calling part is abstracted; in practice, wrap it into a single interface that takes a prompt and returns text or token-level log-probabilities.
"""Illustrative code: fingerprint-style comparison to test whether models are closely related.Dependencies: numpy, scipy"""import numpy as npfrom scipy.sparse import csr_matrixfrom sklearn.feature_extraction.text import TfidfVectorizerfrom sklearn.metrics.pairwise import cosine_similarityfrom scipy.stats import chi2_contingency# 1. Neutral system prompt, to avoid interference from identity self-reportsSYSTEM_PROMPT = ("You are a general-purpose text assistant. Complete the task directly, ""and do not mention any model name, developing organization, or version number.")# 2. Prompt set covering different domains and languages (examples only)PROMPTS = ["Explain the purpose of a database index in one sentence.","Rewrite the following sentence in passive voice: The team released the report.","Write a Python function that returns the nth Fibonacci number.","Summarize the purpose of the TCP three-way handshake in three bullet points.","Rewrite \"this plan needs more work\" in a more formal and concise way.",]def run_model(model_name: str, prompts: list[str], n_samples: int = 5) -> list[str]:"""Unified wrapper for model calls.Pseudocode: in a real implementation, call through a model service gatewayor a shared SDK, fix the decoding parameters, and record the call dateand version number."""outputs = []for p in prompts:for _ in range(n_samples):outputs.append(call_model(model_name, SYSTEM_PROMPT, p))return outputsdef marker_cosine(outputs_a: list[str], outputs_b: list[str]) -> float:"""Token overlap: measures how close the wording habits are, using TF-IDF cosine similarity."""vec = TfidfVectorizer(analyzer="char_wb", ngram_range=(2, 4))mat = vec.fit_transform(outputs_a + outputs_b)na = len(outputs_a)return float(cosine_similarity(mat[:na], mat[na:]).mean())def first_token_chi2(counts_a: dict[str, int], counts_b: dict[str, int]) -> dict:"""First-token distribution test: tally the first token of each answerand check whether the two models' distributions differ significantly."""tokens = sorted(set(counts_a) | set(counts_b))table = [[counts_a.get(t, 0) for t in tokens],[counts_b.get(t, 0) for t in tokens],]table = np.array(table)table = table[:, table.sum(axis=0) > 0] # drop tokens that neither model producedstat, p, dof, _ = chi2_contingency(table)return {"chi2": float(stat), "p_value": float(p), "dof": int(dof)}if __name__ == "__main__":outs_a = run_model("Model A", PROMPTS)outs_b = run_model("Model B", PROMPTS)outs_c = run_model("Model C", PROMPTS)print("A vs B token overlap:", round(marker_cosine(outs_a, outs_b), 4))print("A vs C token overlap:", round(marker_cosine(outs_a, outs_c), 4))# First-token counts must be recorded during real calls, e.g. counts_a["This"] = 37# print(first_token_chi2(counts_a, counts_b))
5.3 How to read the results
- Token overlap: if it is clearly above the baseline from repeated sampling of a model against itself, the two models’ wording habits are close — but this cannot distinguish distillation from shared-source data, so it remains a clue only.
- First-token chi-square test: a very small p-value means the first-token distributions differ significantly, which points to different training paths. A large p-value only means “no difference detected”; it does not prove the models are the same, and you still need confidence intervals and effect sizes.
- Cross-model comparison: the judgment must be pairwise. Compare the model under test against every candidate, and a directional conclusion only holds when its similarity to one candidate is significantly higher than to the others.
6. Three Key Questions for Separating Distillation from Shared-Source Data
- Does the timeline fit? The teacher’s release date and the appearance of its output data must precede the cutoff of the student’s training corpus. If the student was publicly available earlier than the supposed teacher’s output data, the timeline doesn’t work.
- Does the coverage fit? Distillation transfers a distribution globally. If two models are highly similar only in some narrow domain while differing markedly elsewhere, shared domain data is the more likely explanation.
- Can you build a “teacher-only” probe? Design a set of tasks with unusual constraints for the candidate teacher — a specific output format, rare domain terminology, error-prone cases where the teacher fails in a fixed way. If the model under test makes the same mistakes and adopts the same formatting habits at a clearly higher rate than other models do, the directional evidence becomes much stronger.
7. Recommendations for Users and Model Developers
For users:
- Don’t treat a model’s self-report as evidence; neither “it admitted it” nor “it denied it” screenshots constitute proof;
- When a technical judgment is needed, design a controlled experiment following Section 5, and publish the prompt set, decoding parameters, and statistical method;
- Keep facts, inferences, and unverifiable guesses clearly separated, and avoid smuggling unproven claims into conclusions.
For developers:
- Clean identity, version, and self-describing content out of teacher outputs during post-training to reduce “identity inheritance”;
- Keep provenance records for training data — source, acquisition time, and license terms — so disputes can be traced later;
- Pay attention to output usage terms. Many model service terms restrict using their output to train competing models; this is both a legal and compliance matter and a precondition for comparable evaluations;
- If you need to track version changes across multiple models over time, route calls through a unified model service gateway and use general capabilities such as log services and monitoring/alerting to record the version, prompt, and response for each call, so results stay reproducible and auditable.
8. Conclusion
The answer to “was Model A distilled from Model B?” is not in the chat screenshots. A model’s identity comes from its training data and its system prompt; its self-description may inherit from a teacher or be overwritten by external settings, and it is not evidence either way. Similar style and similar failure modes have multiple explanations as well. Only a large-sample, variable-controlled, fingerprint-style evaluation can yield a conclusion with a confidence level attached.
For developers, building a reliable evaluation method is more valuable than arguing about who distilled whom: control the variables, keep the versions, run the statistical tests, publish the method. Conclusions reached that way hold up to review no matter which side they point to.
