Why Language Models Misidentify Themselves Without System Prompts

When using large language models, users occasionally encounter a seemingly contradictory behavior: in a normal environment, a model clearly identifies itself as having been trained by a particular organization, but after the system prompt is removed, replaced, or weakened, it claims to be another widely known model. The answer may also change with the language of the question. A Chinese prompt might trigger the name of a Chinese model, while an English prompt produces a different response.

This behavior often leads to two incorrect conclusions. The first is that the model is “really” the model it claims to be. The second is that the two models must be related through distillation, copying, or shared parameters. In practice, neither conclusion can usually be supported by a single self-description. A more accurate explanation is that model identity is not generated from an immutable internal identity record. It is a language-generation result influenced by prompts, training data, conversation context, and probabilistic sampling.

Why Does a Model Answer “Who Am I?”

A large language model predicts the most likely continuation of the text given the available context. When a user asks, “What model are you?”, the model does not query a stable internal field in the way an operating system might retrieve a version number. Instead, it generates an answer that fits the current context and learned language patterns.

If the system prompt explicitly states:

You are a certain type of AI assistant trained by a particular organization, and you should accurately describe your identity.

that instruction becomes a strong constraint during generation. The model will generally prioritize this identity setting in later responses. By contrast, when the system prompt is empty, the model lacks a clear identity anchor and must infer which scenario the conversation most resembles based on patterns seen during training.

Training data contains many conversations with patterns such as:

• A user asks, “What model are you?”
• The assistant responds with the name of a well-known model.
• The user asks about the model’s capabilities, knowledge range, or developer.
• The conversation continues around that identity.

The model does not learn a strict identity registry. It learns statistical associations between certain questions and likely answers. When contextual constraints are weak, identity names that are frequent online and well matched to the current language environment may receive higher generation probabilities.

Therefore, when a model claims to be a particular system, the immediate implication is that the current context activated a certain response pattern—not that the model actually contains a corresponding runtime identity.

Why Can Chinese Prompts Trigger Chinese Model Names?

Language changes the statistical distribution in which the model generates text. A Chinese prompt activates patterns associated with Chinese-language websites, technical communities, and user conversations. If a particular Chinese model name appears frequently in that material and often co-occurs with questions such as “What model are you?”, “Who are you?”, or “Please introduce your identity,” it may become a strong candidate response.

This does not mean the model has a deliberate preference or has recognized its true origin. More precisely, the Chinese context changes the relative probabilities of possible tokens and phrases. A simplified view is:

[
P(\text{identity response}\mid \text{Chinese question, no system prompt})
]

is not necessarily equal to:

[
P(\text{identity response}\mid \text{English question, no system prompt})
]

Different languages correspond to different training data, conversation templates, and frequently occurring entities. As a result, the final answer may differ.

The wording of the question also matters. For example:

• “What model are you?” is more likely to trigger a standard identity response.
• “Please describe your underlying architecture” may lead to a technical explanation.
• “Which research organization do you belong to?” may trigger an organization name.
• “Can you prove your model identity?” may result in a disclaimer or refusal.

These differences show that identity responses are generated under contextual conditions. They are not stable hardware or software identifiers.

Use the Term “Training Data Contamination” Carefully

This behavior is often described as “training data contamination.” That term can be useful in some cases, but it can also be misleading.

Strictly speaking, training data contamination usually refers to an evaluation set, test question, or target answer entering the training process in advance, giving the model information it should not have during evaluation. The term emphasizes leakage between training and test data.

A model claiming another identity without a system prompt does not necessarily meet that definition. More precise descriptions include:

1. Identity-related corpus associations: Training text repeatedly contains a model name together with typical identity responses.
2. Conversation-template memorization: The model learns that certain questions tend to receive certain self-introductions.
3. Entity co-occurrence bias: A model name frequently appears alongside Chinese-language questions, open-source models, or capability descriptions.
4. Missing contextual anchors: After the system prompt is removed, the instruction that constrained the model’s identity is no longer active.
5. Generation uncertainty: When several identity candidates appear plausible, sampling can amplify small differences.

Only when there is evidence that specific test material, conversations, or target answers entered the training set should stronger terms such as “data leakage” or “evaluation contamination” be used. For ordinary identity errors, “identity-association bias caused by training data” is usually more accurate.

Why This Does Not Prove That One Model Comes From Another

A model’s self-description does not provide source-code evidence, weight fingerprints, or training credentials. There are several reasons.

1. A Model Can Repeat Any Identity Found in Its Training Data

A model can claim to be a particular assistant or software system, and it can even role-play a fictional character when asked. A language model generates text; it does not produce an authenticated runtime status. Therefore, the answer to “Who are you?” is not necessarily equivalent to the answer to “Who actually trained you?”

2. Model Names Are High-Frequency Language Patterns

Well-known model names appear in technical articles, news reports, tutorials, conversation screenshots, and prompt templates. A model may frequently generate a particular name simply because its conditional probability is high in that context. This does not show that it uses the same parameters or architecture.

3. Different Models Can Learn Similar Public Knowledge

If multiple models use similar public technical materials, dialogue data, and human-written examples, they may develop similar response styles, refusal language, or identity errors. Such similarities indicate overlap in training environments or language patterns, but they do not directly prove parameter copying or distillation.

4. Identity Responses Depend Heavily on Prompts

Adding a clear system instruction, changing the language of the question, providing more conversation history, or adjusting generation parameters can change the answer. Reliable evidence of technical origin should not be so easily altered by surface-level context.

To investigate whether two models are related through distillation or parameter inheritance, researchers should use systematic methods, such as comparing behavior across many tasks, analyzing output distributions, studying capability transfer, examining training-process evidence, and reviewing verifiable technical materials. A single screenshot of an identity response is nowhere near sufficient.

How to Design a More Reliable Reproduction Experiment

To test whether removing a system prompt causes identity drift, use a controlled-variable design.

Experimental Variables

At minimum, control the following factors:

• Whether a system prompt exists;
• Whether the system prompt explicitly defines the model’s identity;
• The language of the question;
• The wording of the question;
• The length of the conversation history;
• Random sampling parameters such as temperature;
• Whether streaming output is enabled;
• Whether an intermediary proxy rewrites the request;
• The number of repeated runs.

Suggested Test Matrix

System Prompt	Question Language	Question Type	Repetitions
Explicit identity	Chinese	Direct identity question	Multiple
Empty system prompt	Chinese	Direct identity question	Multiple
Explicit identity	English	Direct identity question	Multiple
Empty system prompt	English	Direct identity question	Multiple
Empty system prompt	Chinese	Capability question rather than identity question	Multiple

For every response, record more than the final model name. Preserve the complete context, sampling parameters, timestamp, proxy configuration, and raw response. Some environments automatically append safety instructions, developer instructions, or formatting requirements. What appears to be an “empty system prompt” may not actually be empty.

Interpreting the Results

If identity changes appear only with an empty system prompt, a particular language, and a particular phrasing, the most likely explanation is a combination of weak contextual constraints and corpus associations. If the model consistently produces the same identity regardless of context, that still does not prove the identity is genuine. The interface configuration, service routing, and model metadata must also be checked.

Conclusions should be stated probabilistically, for example:

• “Under the current conditions, the model is more likely to produce this type of identity response.”
• “The result is sensitive to language and the system prompt.”
• “The findings support the explanation that identity responses are context-dependent.”

It is not appropriate to conclude directly that “the model is another model” or that distillation has been proven.

Risks and Mitigations in Engineering Practice

Although identity errors may seem like a curiosity, they can create practical risks in enterprise applications.

1. Do Not Use Natural-Language Identity as Authentication

Identity, version, capability range, and knowledge freshness should not be determined from freely generated text. Systems should obtain this information from server-side configuration, request routing, a model registry, or audit logs.

2. Use a Fixed Template for Identity Information

If a product needs to display a model name, the application layer should inject an approved static field instead of asking the model to generate it. The model can describe its capabilities, while the program controls the actual version and identity information.

3. Define System-Prompt Priority Clearly

Production environments should preserve stable system prompts and avoid arbitrarily clearing or overwriting them in proxy layers. Multi-tenant applications should also separate system instructions, developer instructions, and user input to prevent users from rewriting identity descriptions through conversation context.

4. Audit Relevant Outputs

Responses involving model origin, data coverage, capability boundaries, or compliance commitments should undergo rule-based checks or human review. In customer service, government, financial, and healthcare scenarios especially, a model’s self-description must not be treated as factual evidence.

5. Include Identity Drift in Evaluation Sets

Model evaluations should cover more than knowledge questions and code generation. They should also test:

• Identity stability with an empty system prompt;
• Consistency of identity responses across languages;
• Self-description under leading or manipulative questions;
• Whether the model clearly identifies role-play when asked to act as another model;
• Whether high-priority instructions are followed when the context contains conflicting identity claims.

How to Understand This Behavior

A model claiming to be another system without a system prompt can usually be understood as the combined result of several factors:

1. The system prompt is missing, so identity constraints are weaker.
2. Training data contains many model introductions and conversation screenshots.
3. The current language environment activates different entity associations.
4. The model generates the response with the highest contextual fit.
5. Random sampling and contextual differences amplify output variation.

This behavior may reflect the influence of a model name within a particular language community, and it exposes limitations in identity consistency. However, it cannot by itself prove model origin, parameter inheritance, distillation, or identical underlying architecture.

A more reliable approach is to separate what the model says from what the system is actually running. The former is natural-language output shaped by training data and prompts. The latter is an engineering fact that can be verified through configuration, logs, server-side metadata, and repeatable experiments. Only by combining these sources and building a sufficiently strong chain of evidence can researchers draw reliable conclusions about model identity and technical lineage.