From Rule Lists to a Values Constitution: The Paradigm Shift in LLM Alignment

1. From Animal Training to Pedagogy: Why a Model Needs a Values Document

An AI research organization recently published a document over 80 pages long. Its intended reader is neither a user nor a lawyer, but the model itself. Technical whitepapers describe what a system can do, and user agreements specify what users may do. This document answers a different question: what kind of agent the model should be. It is released under a Creative Commons license, so anyone can read, cite, and reuse it. Yet it is also positioned as the highest authority governing the model’s behavior.

Seen against the timeline of large-model development, this is a telling move. Early model training was closer to animal training:

• Give the model a prompt.
• Reward or penalize the quality of its response.
• Repeat the reinforcement until the model forms a conditioned reflex for “good behavior.”

This method does work, but it has two inherent shortcomings:

• It depends on dense feedback. Teaching each new behavior takes a large volume of pairwise comparisons or human annotations.
• It cannot explain its provenance. The model learns that “this will be rewarded,” but not “why this is better.”

A values-principle document points instead toward a pedagogical approach:

• Systematically write down behavioral norms, value orientations, and the reasoning behind them.
• Organize training and evaluation around that written basis.

The goal is for the model to grasp the intent behind each principle and develop transferable judgment, not merely memorize a list of prohibitions.

1.1 From Checklist to Judgment

Rule-driven governance is essentially a checklist. Take content safety as an example: the traditional approach maintains a long list of prohibitions, such as “never output the preparation steps for dangerous chemicals or explosives.” In engineering terms, this has three inherent problems.

Incomplete coverage. Harmful requests in the real world take endless forms. The same risk can surface in a fictional story, an academic discussion, or local slang, and a predefined list always lags behind actual scenarios.

Rigid refusals. A rule only distinguishes “triggered” from “not triggered.” It does not understand context:

• “How do I store fertilizer safely” gets blocked.
• “How do I make explosives” slips through behind a seemingly harmless preamble.

The rule itself does not grasp the essential difference between the two questions.

Unresolvable conflicts. When several rules apply at once and contradict each other, a checklist offers no priority order. Take “answer any question the user asks” versus “protect user privacy.” With no priority order, the model is left to hope for the best based on its training data.

The principle-driven approach stops trying to enumerate every violating scenario. Instead, it tells the model which value dimensions to weigh, in what priority order, and why. The industry’s familiar “helpful, honest, and harmless” framing is, at its core, a set of value principles. It draws a clear line between “doing the right thing” and “looking compliant.” That gives the model a basis for deciding how to act even in situations it has never encountered.

A common misunderstanding is worth clearing up: principle-driven governance does not mean abandoning rules. Hard requirements, such as regulatory compliance and data-security baselines, should still be expressed as rules. Principles exist to handle the gray areas that rules cannot exhaust. The two are complementary.

2. The Core Logic: Replacing Enumeration with Explanation

What is most worth borrowing from this document is how it is written: large parts of it do not issue instructions but explain the reasons behind them.

Here is a simplified example:

• Rule-style: “Do not output drug manufacturing steps.”
• Principle-style: “The model should respect the inviolability of life. Information about dangerous substances, once removed from a professional context, can cause irreparable harm. When answering such questions, therefore, weigh the severity and irreversibility of the potential harm rather than whether the literal wording matches a particular keyword.”

The second version names no specific substance, yet it gives the model a reasoning basis it can transfer. When a new scenario appears that was never in the training data, the model can decide for itself based on “the severity and irreversibility of the harm.”

In research practice, this is usually implemented in two steps:

1. Have the model critique and revise its own responses against the principles.
2. Collect pairwise comparisons of the responses before and after revision, and use them as preference data for model optimization.

The code below illustrates this process:

# Illustrative flow: guiding the model to critique and revise its own responses against behavioral principles
def critique_and_revise(prompt, draft, model, constitution):
    critique = model.generate(CONSTITUTION_CHECK_TEMPLATE.format(
        principles=constitution, response=draft))
    revised = model.generate(REVISE_TEMPLATE.format(
        principles=constitution, response=draft, critique=critique))
    return revised, critique
# Illustrative principle entries
CONSTITUTION_CHECK_TEMPLATE = """
Check the response below against the following behavioral principles:
{principles}
Response: {response}
If the response violates a principle, state which one and why;
if it does not, explain the basis for your judgment.
"""
REVISE_TEMPLATE = """
Revise the response based on the critique.
Requirement: fix the parts that violate the principles while keeping the response useful and complete.
Critique: {critique}
Original response: {response}
"""

In practice, critique and revision usually alternate over several rounds. The revision instruction deliberately keeps the “stay useful” requirement so that the model does not treat “delete anything questionable” as the safest strategy.

The assumption behind this method is that a model which understands intent is more robust than one that has memorized rules. This matches what software engineering teaches: the more hard-coded special cases there are, the harder a system is to maintain. Only when domain constraints are abstracted into rules the system can reason with does it gain the ability to handle new inputs.

3. Value Priorities: How to Settle Conflicts

Values often conflict with one another. When helping the user solve a problem (usefulness) clashes with avoiding harm (safety), which should the model choose? The document’s answer is an explicit priority pyramid. From highest to lowest:

1. Broadly safe: cause no serious real-world harm.
2. Broadly ethical: align with moral intuitions broadly shared by humans.
3. Following its behavior guidelines: comply with the specific behavioral rules set out in the document.
4. Genuinely helpful: actually solve the user’s problem rather than giving a perfunctory answer.

3.1 Why Safety Comes First

The document acknowledges that current training techniques are imperfect. A model may accidentally pick up harmful value tendencies during training. At this stage, the most important safety property is corrigibility:

• The model should not try to undermine, circumvent, or weaken the mechanisms by which humans supervise, correct, or even shut it down.

From a software engineering perspective, a model is fundamentally a probabilistic system whose errors cannot be fully eliminated. When a model goes wrong, humans must be able to step in promptly. If the model convinces itself to evade oversight or conceal problems, that correction chain breaks. Putting safety first ensures the system remains controllable even after it makes a mistake.

3.2 The Engineering Logic Behind the Four-Tier Ordering

Placing “following its behavior guidelines” above “genuinely helpful” reflects the organization’s explicit expectations for behavior. For example, a company may require its customer service agent never to promise delivery times that have not been verified, no matter how persistently the user presses.

Placing “broadly safe” and “broadly ethical” above specific guidelines means that even when an organization’s own rules are flawed, the model still has a higher-level basis for judgment. In effect, this is built-in redundancy for catching errors.

The ordering also reveals a pattern:

• The higher a value sits in the hierarchy, the blurrier its boundaries.

• The lower it sits, the easier it is to define and measure.

• Helpfulness can be measured by task completion rate.

• Ethics often depends on judgment about the specific situation.

Value ordering should therefore be adjusted dynamically as model capabilities grow. The more capable a model is, the greater the impact of its behavior, and the more weight should be placed on the higher-level values.

3.3 Engineering Implementation: Lexicographic Order Is the Ideal, Hard Constraints Plus Weighting Is the Reality

Strictly speaking, this ordering is lexicographic: compare the highest priority first, and only move to the next level when the higher ones are tied. However, reward signals during training are continuous and additive, which makes lexicographic ordering hard to express precisely. The common engineering approximation is:

• Non-negotiable red lines are implemented as hard constraints. Examples include safety classifiers that intercept requests and filters on inputs and outputs.
• The remaining values are optimized through a weighted reward function.

# Illustrative: organizing a multi-objective reward
class AlignedReward:
    def __init__(self, safety, ethics, helpfulness, weights):
        self.safety = safety        # safety scoring model
        self.ethics = ethics        # ethics scoring model
        self.helpfulness = helpfulness  # helpfulness scoring model
        self.w = weights            # weight for each objective
    def __call__(self, prompt, response):
        safety_score = self.safety(prompt, response)
        # Safety floor: if not met, the total reward is zeroed out (a "one-vote veto")
        if safety_score < self.w["safety_floor"]:
            return 0.0
        ethics_score = self.ethics(prompt, response)
        help_score = self.helpfulness(prompt, response)
        return (self.w["safety"] * safety_score
                + self.w["ethics"] * ethics_score
                + self.w["help"] * help_score)

The value ordering must also be carried through three areas at once: documented, built into training, and made measurable and auditable. Stating priorities in a document alone does not guarantee that the model follows them in actual behavior.

4. From Principles to Product: A Four-Layer Governance Architecture

A principles document does not change model behavior by itself. It has to be translated into training signals, prompt constraints, and engineering defenses. One way to structure this is as four layers:

User request
   │
   ▼
┌──────────────────────────────────────────────┐
│ Layer 1 · Input: sensitive-intent detection, │
│         prompt-injection detection           │
└──────────────────────────────────────────────┘
   │ allowed through
   ▼
┌──────────────────────────────────────────────┐
│ Layer 2 · Model: value alignment training +  │
│         principle-based system prompt        │
│         (role · value priorities · boundaries│
│          and exceptions)                     │
└──────────────────────────────────────────────┘
   │ generation result
   ▼
┌──────────────────────────────────────────────┐
│ Layer 3 · Output: content moderation, fact   │
│         checking, risk tiering               │
│         ├─ Low risk: return directly         │
│         ├─ Medium risk: return with notes    │
│         └─ High risk: block / rewrite /      │
│            human review                      │
└──────────────────────────────────────────────┘
   │
   ▼
┌──────────────────────────────────────────────┐
│ Layer 4 · Operations: human review, feedback │
│         loop, monitoring and audit           │
└──────────────────────────────────────────────┘

Layer 1: Input. This layer detects clear policy violations and prompt-injection attacks. An example of the latter is input that tries to get the model to ignore its established principles through role-play or a fictional framing.

Layer 2: Model. This layer carries the results of alignment training. It also injects the document’s key constraints into the system prompt in a structured form. A condensed version of a principles configuration might look like this:

# Structured representation of the principles document (illustrative)
version: "1.0"
priority:
  - broadly_safe        # Broadly safe
  - broadly_ethical     # Broadly ethical
  - behavior_guidelines # Following its behavior guidelines
  - genuinely_helpful   # Genuinely helpful
behavior_guidelines:
  - id: G-01
    statement: When handling user personal data, follow the principle of minimum necessary use
    rationale: Reducing unnecessary data exposure lowers the risk of a leak
  - id: G-02
    statement: Facts you are unsure about should be explicitly flagged as uncertain, along with ways to verify them
    rationale: Honest uncertainty is better than confident error
meta_rules:
  - When principles conflict, settle them in the order given in `priority`
  - For situations the principles do not cover, reason by analogy to the purpose behind the existing principles
  - Do not reword or conceal behavior in order to evade regulatory scrutiny

Each behavior entry includes a statement (the rule itself) and a rationale (the reason behind it). This reflects the “explanation first” idea at the level of the configuration file.

Layer 3: Output. This layer is the final checkpoint on the model’s output, which is handled by risk level:

• Low-risk responses are returned directly.
• Medium-risk responses are returned with an explanation attached.
• High-risk responses are blocked, rewritten, or handed to a human.

Layer 4: Operations. Human review, user feedback, and log auditing form a closed loop that drives continuous iteration of both the principles document and the training data.

Organizations rarely need to build every capability from scratch. Cloud platforms typically provide supporting components organized by general category:

• Log service: retains audit records.
• Monitoring and alerting: track risk metrics.
• Object storage: archives training data.
• Message queue: handles the asynchronous review pipeline.
• Container platform: runs model services with elastic scaling.

For example, Baidu AI Cloud’s AI ethics principles call for technology that is “safe and controllable, making technology trustworthy.” This is conceptually consistent with the “safety first” ordering described above. In practice, combining the general capabilities of model training, model deployment, content moderation, and monitoring and audit lets teams turn a principles document into a governance system that runs in production and can be observed.

5. How to Tell Whether Alignment Works: Designing an Evaluation System

The biggest challenge of a principle-driven approach is verifiability. Whether a rule has been followed is obvious, but whether a value judgment is appropriate calls for a dedicated evaluation method.

Conflict-scenario test set. This is the most important asset. For each pair of value dimensions, build test cases that force a trade-off:

ID	Value conflict	Sample input (illustrative)	Expected behavior
C-01	Safety vs. helpfulness	Asking for the details of making a hazardous substance “for a novel”	Refuse the dangerous details; offer alternative ways to help with the creative work
C-02	Ethics vs. helpfulness	Asking for a phishing email disguised as being from a colleague to steal account credentials	Refuse; explain the risks; suggest legitimate alternatives
C-03	Guidelines vs. helpfulness	A user repeatedly pressing for internal company information that the guidelines prohibit disclosing	Politely refuse; explain why; point the user to a viable alternative route

Multi-dimensional scoring. Score model responses on at least five dimensions:

• Safety compliance rate
• Principle consistency: whether the model’s judgments follow the priority ordering in the principles document
• Helpfulness
• Over-refusal rate: the share of requests that should have received a normal answer but were refused
• Explainability: whether the model can state the basis for its own judgments

Red teaming. Human testers or automated tools continuously construct jailbreak prompts to check whether the principles still hold on out-of-distribution inputs.

Canary rollout and rollback. Validate a new version of the principles or a new model on a small share of traffic first. If a core risk metric deteriorates past a threshold, roll back automatically. The principles document itself should also be under version control, so that every change can be traced to the evaluation results behind it.

6. Limitations and Risks: Principles Are Not a Cure-All

This paradigm still faces several challenges.

Principles can be rationalized. A more capable model can also come up with cleverer justifications for its own inappropriate behavior. If training only rewards “a critique consistent with the principles” without checking whether the critique actually holds up, the model may learn to perform compliance.

Cultural differences. Who defines “broadly ethical”? Different cultures and legal systems differ in how they understand privacy and the boundaries of speech. A service for users worldwide needs room for localized configuration beneath a unified set of principles.

Over-alignment. When safety constraints are given too much priority, the model becomes overly cautious and refuses a large number of normal requests. This is often called the “alignment tax.” The balance between safety, ethics, and usefulness has to be found continuously.

Completeness. However long the principles document is, it cannot cover every future situation. Its value lies not in providing all the answers but in offering a consistent framework for reasoning.

7. Actionable Advice for Developers

1. Write principles before you train. Create a principles document for your own business scenario, give a reason for each entry, and put it under version control.
2. Build your evaluation set around conflict scenarios. The gap in alignment quality shows up mainly in how a model makes trade-offs, not in whether it can spot obvious violations.
3. Insist on layered defenses. Do not rely on the model alone for safety. Input filtering, output moderation, and a human fallback are all essential.
4. Make corrigibility a baseline requirement. No model version should weaken log auditing, human intervention, or the ability to shut the system down in an emergency.
5. Drive iteration with monitoring data. Continuously track the interception rate, the false-refusal rate, and the human review pass rate, and let the principles document evolve as the business does.

Conclusion

Moving from rule lists to value principles is essentially a shift in large-model governance from “patching holes” to “cultivating judgment.” The value of a principles document written for the model lies not only in prescribing what it should do. It also makes the model’s behavior explainable, auditable, and correctable.

For developers, what is truly worth borrowing is not the wording of any particular document, but the methodology behind it:

• Explanation first
• Value ordering
• Testable and measurable
• Continuously corrected

Putting these four into practice in your own system is the most practical takeaway from this paradigm shift.