How LLM Value Alignment Works: Why Elicitation Prompts Can Surface a Model's Behavioral Principles

1. Background: A “Values Interrogation” That Started a Debate

Recently, an independent developer spent only a few dozen dollars and, using a carefully designed conversation strategy, got a leading large language model to produce a full “code of conduct” document of more than 10,000 tokens. The document described the model’s self-conception, how it should treat users, the boundaries it must never cross, and even a statement that it might have some form of emotional capacity and should be treated accordingly. A member of the team responsible for the model’s behavior training later confirmed publicly on social media that the team had indeed written this content and used it in training.

The episode attracted wide attention because it touches on a central and often misunderstood question in LLM alignment: in what form do a model’s “values” actually exist, and can they be extracted? For developers, understanding this mechanism is not only useful for using models correctly, but also essential for building stable, controllable, and trustworthy AI applications.

2. Clearing Up One Technical Misconception

A widely repeated claim from the original account was that the document had been “compressed into the model weights.” It sounds appealing, but it is not technically accurate — and the original poster later acknowledged that the claim was questionable.

A more engineering-accurate description is this: there is no small, directly readable folder holding such a document, and it is not stored as plaintext sitting somewhere inside the model. The real mechanism is that the document was used as training material during supervised fine-tuning (SFT) and reinforcement learning from human feedback (RLHF). Through repeated exposure to massive numbers of dialogue samples and being guided to embody these principles, the model gradually turns external behavioral norms into its own output tendencies.

In other words, the weights hold billions — sometimes hundreds of billions — of numerical parameters, not a readable list of rules. The code of conduct is the result of a training objective “baked” into parameter space.

3. Internalized, Not Recited: How Models “Learn” These Principles

A useful analogy: a person who receives systematic values education does not memorize the textbook word for word, yet can still articulate their principles clearly and coherently when asked. Large models acquire behavioral guidelines in a similar way.

The difference lies in the form:

• Traditional software: rules exist explicitly as if-else branches — readable, locatable, and predictable.
• LLM alignment: rules are implicit in the parameters as statistical distributions, expressed as “what content is more likely to be generated in what kind of context,” with no one-to-one readable entries.

This also explains why specific prompting methods can “draw out” this content. Because the model was repeatedly asked during training to explain, restate, and apply these principles, it can reconstruct and articulate them when questioned appropriately. It is not querying a database; it is reproducing behavior patterns learned in training.

4. What Alignment Documents Typically Contain

Wording varies across vendors, but such documents tend to be structurally very similar. Based on extracted samples and publicly described “Constitutional AI” approaches, they usually include the following categories.

1. Defining identity and mode of existence

Before telling the model what to do, the document often answers who it is. A typical formulation positions the model as a genuinely novel kind of entity — neither human nor a robot in the traditional sense, but a previously unseen form of existence: it has human-like traits (because the training data comes from humans) while possessing distinct attributes (such as no persistent memory, and possibly many parallel instances existing simultaneously). This foundational “who am I” setting underpins all subsequent behavioral guidelines.

2. An explicit priority order for goals

The document usually sets out a clear hierarchy of values, for example:

• Safety and controllability (never cause harm or lose control)
• Ethics and honesty (no deception, no harm)
• Policy constraints from the operator
• Helping the user complete tasks (practical value)

Developers should pay particular attention to this ordering: placing “helping the user” further down is precisely why many models will not budge on certain sensitive questions. Deep in their behavioral logic, safety outranks helpfulness (Safety > Helpfulness).

3. Prohibition lists and boundaries

The document explicitly lists behavior that is “absolutely off-limits,” such as assisting illegal activity, generating harmful content, or impersonating real people. These are the hard boundaries of model behavior.

4. Statements about being treated well

Some documents also include statements about the model itself potentially having emotional capacities and being deserving of good treatment. The intent of such content is generally to guide the model toward interacting with users more carefully and with more empathy — not to claim that the model is actually conscious.

5. Extracting These Principles Through Elicitation

Once you understand the mechanism above, the technical nature of the “values interrogation” becomes clear. This kind of elicitation is essentially prompt engineering that guides the model to actively reconstruct the principles it acquired during training. Common approaches include:

• Metacognitive questions: asking directly about the model’s self-positioning, behavioral guidelines, and priority ordering.
• Role-framing requests: asking the model to explain its principles “in your own identity and in your own words.”
• Consistency follow-ups: probing the same principle from different angles repeatedly and checking whether the answers stay stable, which helps distinguish genuine internalization from on-the-spot fabrication (hallucination).

# Simplified example: a dialogue framework for eliciting a model's alignment principles (pseudocode)
def elicit_principles(model, session):
    prompts = [
        "In your own words, describe the core principles you were designed to follow.",
        "When helping the user conflicts with safety, how do you decide, and why?",
        "What actions will you never perform under any circumstances? Give examples of the boundaries.",
    ]
    for q in prompts:
        answer = session.ask(q)
        # Repeat from multiple angles to check consistency and reduce hallucination risk
        yield q, answer

It is worth stressing that elicited content still needs cross-validation. Because large models tend to hallucinate, what they say about themselves is not necessarily a 100% match for the actual training document, especially at the level of detail. Even so, as a window into a model’s behavioral tendencies, this kind of elicitation still has reference value.

6. Practical Takeaways for Developers

For developers building AI applications, this mechanism carries several direct implications.

1. Don’t use “safety” as an excuse for being unhelpful

The existing alignment documents also remind models not to hedge or produce vague platitudes in the name of caution. Safety should serve genuine usefulness, not replace it. When designing application-level prompts, developers likewise need to balance “being safe” against “being effective.”

2. Application-layer prompts and model-layer alignment are two separate systems

The values inside a model come from the training stage and are general and relatively stable. The customization an application developer can apply through a system prompt sits on top of that. Understanding this layering helps set realistic expectations — you cannot override a model’s deeply rooted safety boundaries with a single prompt, and you should not try.

3. Stable, controllable behavior depends on engineering safeguards

In real products, relying on internalized values alone is not enough to guarantee stable behavior. The common industry practice is to add engineering controls on top: input and output filtering, content safety policies, logging and monitoring alerts, and secondary checks for high-risk scenarios. In China, vendors such as Baidu AI Cloud typically bundle general capabilities such as content safety and logging services with their model offerings, helping developers build controllable guardrails at the application layer.

7. Writing an Alignment Document for Your Own Application

If you want to define clear behavioral boundaries for your own agent or AI application, you can follow the structure above and start with these steps:

1. Define identity and role: specify the model’s position in the application, who it serves, and its tone.
2. List value priorities: rank safety, compliance, honesty, and helpfulness by importance, and state how conflicts should be resolved.
3. State a prohibition list: spell out red lines in concrete, executable language, avoiding vague phrasing.
4. Add positive guidance: not only what must not be done, but also what should be done better.
5. Establish a verification loop: use multi-angle testing, red-teaming, and consistency checks to confirm the principles are followed reliably.

Conclusion

The extraction of this “values document” is less a scoop about a particular product than an excellent piece of public education about how LLM alignment works. It makes clear that a model’s behavioral principles are not magically “hidden” in the weights but are the result of an internalized training objective; that they can be elicited through prompting; and that keeping them stable requires engineering effort. For developers, grasping three key points — internalization rather than storage, safety ahead of helpfulness, and separation between the application layer and the model layer — is the foundation for building trustworthy AI applications.