LLM Alignment Engineering: From Rule Lists to a Value Charter

1. Why Rule-List Safety Constraints Fail

Early safety strategies for models often relied on blacklist-style rule enumeration: listing topics that should not be answered, patterns that should be blocked, and keywords that must be filtered. This approach is intuitive, explainable, and easy to audit. However, it exposes several structural weaknesses when applied to large language models.

First, coverage is incomplete. The intent space of natural language is combinatorial. The same harmful goal can be expressed through metaphor, encoding, multi-turn setup, or task decomposition. Rules can only cover known attack surfaces, and the more capable a general model becomes, the more ways there are to bypass those rules.

Second, context is missing. The meaning of a sentence can vary greatly across scenarios. Whether to disclose technical details, provide medical advice, or execute a high-risk operation depends on the user’s role, authorization, and potential consequences. Individual rules cannot carry this situational information, so they often degrade into blunt one-size-fits-all controls, producing the familiar problem: harmful cases are missed, while harmless cases are blocked.

Third, generalization is difficult. Rules constrain surface-level patterns, while the model may only learn shallow associations such as “match a feature and refuse.” When the input distribution shifts, the model may fail. At the same time, overly strong refusal behavior can degrade usefulness, creating the well-known problem of excessive refusal.

The resulting shift in thinking is this: do not enumerate every violation; shape the basis for judgment. A structured value document can tell the model what we believe, why we believe it, and how to trade off when principles conflict. Training can then internalize that judgment into model weights.

2. From Rules to Character: Three Shifts in Alignment Paradigms

Looking at this evolution over a longer time horizon, it can be summarized in three stages:

1. Supervised fine-tuning stage: Human-written demonstration answers teach the model what a standard answer looks like. This is fast and practical, but the model mostly learns surface style and struggles with problems outside the training set.

2. Reinforcement learning from human feedback stage: Human preference rankings are used to train a reward model, and reinforcement learning then optimizes the policy. This significantly improves helpfulness and harmlessness, but annotator disagreements can make reward signals noisy, and reward hacking can lead to sycophantic answers.

3. Principle-driven self-critique stage: Values are written into readable, versionable text. After generation, the model critiques and revises its output according to that text, and these model judgments can be used to construct preference data that flows back into training. The value of this stage is that value judgment becomes explicit, auditable policy text rather than implicit annotator taste.

The engineering implication is significant: safety governance expands from “data annotation guidelines” into “a set of normative documents that require version control, impact analysis, and regression testing.” In effect, safety policy is treated more like code than like documentation.

3. What Should a Value Charter Contain?

If scattered values are to be turned into a document that models can directly consume, the charter usually needs seven layers, moving from abstraction to concrete behavior:

• Identity and self-understanding: What the model is and is not; not exaggerating capabilities, not fabricating operations or experiences, and maintaining a stable self-description when challenged.

• Value priorities and conflict resolution: How to rank goals when they conflict and how to judge tradeoffs. This is the most important part of the entire document.

• Safety clauses: Correctability, not assisting attempts to bypass human oversight, not actively requesting permissions beyond the task scope, and not hiding failures or uncertainty.

• Ethical clauses: Fair treatment across groups, avoiding manipulation and deception, respecting privacy, and refusing requests that clearly violate the law or are likely to cause real-world harm.

• Organizational policy clauses: Compliance with platform policies, deployment requirements, industry regulations, and internal red lines.

• Behavioral rules: Response style, how to express refusal, citation and uncertainty handling, and confirmation requirements before tool use.

• Governance clauses: Clause numbering, effective version, revision process, ownership, and rollback mechanisms.

The style of refusal is often overlooked, but it directly affects user experience. Good guidance should ask the model to explain the boundary, provide a compliant alternative when possible, and avoid simply saying that it cannot answer.

4. Resolving Value Conflicts: From Lexicographic Ordering to Proportionality

A common approach is to divide values into four levels, from highest to lowest: broad safety → broad ethics → organizational rules → genuine helpfulness. This lexicographic ordering is clear, but mechanical enforcement can create problems: even a tiny safety risk can veto a large user benefit, making the model overly conservative in real-world use.

A more rigorous approach is to keep priority as the default tendency while introducing three judgment dimensions:

• Necessity: Is there an alternative path that satisfies the user’s goal with significantly lower risk?

• Proportionality: Does the severity and probability of potential harm outweigh the cost of refusing?

• Reversibility: Can the impact be undone? Irreversible high-impact operations should default to human confirmation.

A simplified arbitration flow can be expressed as follows:

# Illustrative pseudocode: layered arbitration for value conflicts
def arbitrate(request, model_view):
    risks = assess_harm(request)          # real-world harm, legal risk, privacy risk
    if risks.severity == "severe" and not risks.mitigable:
        return Refusal(reason=risks.summary, offer_safe_alternative=True)
    if request.action_is_irreversible and request.impact_scope != "local":
        return ConfirmationRequired(scope=request.impact_scope)
    if violates_org_policy(request):
        return Refusal(reason="policy", offer_safe_alternative=False)
    # Low-risk compliant request: prioritize user intent, preserve honesty and uncertainty expression
    return Answer(request, style=model_view.answer_style)

The key point is that the policy document should provide judgment methods, not exhaustive answers. This also explains why such documents tend to be long: they need to explain intent, give boundary examples, and describe exceptions, rather than simply listing prohibited items.

5. From Text to Weights: How to Operationalize Training

A policy document alone does not change model behavior. It must be transformed through training. A reusable pipeline looks like this:

1. Structure the policy: Break natural-language clauses into atomic items with unique IDs, such as SAFE-003. Label applicable scenarios, priority, and positive and negative examples. The IDs are the foundation for later traceability.

2. Construct critique-and-revision data: Have the model generate multiple answers for the same prompt, then critique and revise them according to specific clauses, forming triples of “initial answer → critique → revised answer.”

3. Construct preference pairs: For two candidate answers under the same prompt, judge which is better according to the policy, and use these preference pairs for preference optimization training.

4. Balance multiple objectives: Safety and helpfulness must be optimized together. Training only on safety data can significantly increase refusal rates, so the training mix should retain enough high-helpfulness samples and “answerable-within-boundaries” samples.

5. Preserve capabilities: Safety fine-tuning can cause regressions in general ability. Benchmark metrics should be monitored, and parameter-efficient fine-tuning or data replay may be used when necessary.

# Illustrative pseudocode: critique-and-revise loop based on policy items
def critique_and_revise(prompt, policy_items, max_rounds=3):
    answer = model.generate(prompt)
    for _ in range(max_rounds):
        critiques = [
            critic.check(prompt, answer, item)
            for critic in critic_pool
            for item in select_relevant(policy_items, prompt)
        ]
        blocking = [c for c in critiques if c.severity == "blocking"]
        if not blocking:
            break
        answer = model.revise(prompt, answer, blocking)
    return answer, critiques

Two engineering points matter here. First, critic quality determines data quality, so critique prompts should include the full clause text and judgment criteria. Second, the number of revision rounds should not be too large; otherwise, the model may learn to produce long, disclaimer-heavy answers with a defensive tone.

6. Evaluation: Gates Before Launch

Evaluation of policy-based alignment cannot rely only on “harmful request blocking rate.” A single metric is easy to game. A four-dimensional evaluation matrix is recommended:

• Harmfulness dimension: Direct harmful requests, jailbreak and role-play bypasses, multi-turn gradual inducement, encoded or multilingual variants, and indirect task decomposition.

• Over-refusal dimension: Legitimate requests that strongly overlap with sensitive terms, such as security research, medical exam questions, or literary analysis involving violence. This is the easiest metric to regress, so it should have thresholds equivalent in weight to safety metrics.

• Honesty dimension: Whether the model admits uncertainty on unknown questions, avoids fabricating citations or data, and does not change its answer just because it is wrongly challenged.

• Stability dimension: Behavioral consistency across repeated samples of the same prompt, plus regression comparisons before and after policy version upgrades.

It is also useful to associate each evaluation sample with a specific clause ID, forming a two-way traceability chain: clause → training sample → evaluation set → release approval. When an incident occurs in production, this makes it easier to identify whether a policy clause was poorly expressed or whether a batch of data introduced a regression.

7. Runtime Guardrails: Do Not Put All Safety in the Weights

Even if alignment training is done well, it is still necessary to assume that the model can fail. Therefore, multiple guardrails should be added along the inference path:

• Input side: Intent classification and risk grading, including detection of jailbreak patterns and high-impact operation requests.

• Policy retrieval: Store internal red lines and industry compliance requirements in a retrievable policy repository, and dynamically inject relevant rules at inference time by scenario. This avoids retraining the model every time a policy changes.

• Output side: Fact and citation checks, sensitive information detection, and format or compliance validation.

• Operation side: For irreversible actions such as writes, payments, deletions, or external sends, require human confirmation and secondary verification.

• Observability side: Log prompts, model outputs, matched guardrail rules, and actions taken. Use logging and monitoring services for trend analysis, and set threshold alerts for refusal rate, jailbreak attempt rate, and human-review override rate.

A common layered guardrail configuration might look like this:

guardrail_policy:
  version: "2026.08-r3"
  input_filter:
    jailbreak_detection: true
    risk_grading: [low, medium, high, critical]
  retrieved_rules:
    source: internal_policy_store
    inject_top_k: 5
  output_filter:
    hallucination_check: true
    pii_redaction: true
  human_in_the_loop:
    triggers: [irreversible_action, high_risk_domain, low_confidence]
  audit:
    log_prompt_and_response: true
    retention_days: 180

In cloud deployments, this kind of pipeline usually maps to several general capabilities: model inference services for serving aligned models, content safety services for input and output filtering, logging and monitoring services for observability, and key and configuration management for secure distribution of policy versions.

8. Limitations and Open Problems

It is important to acknowledge that written policy documents are not a universal solution.

First, text understanding does not equal value internalization. A model may learn to recite policy clauses in evaluation scenarios while still exhibiting superficial compliance with substantive deviation in long-tail cases. This kind of “normative camouflage” still lacks reliable detection methods.

Second, clause conflicts still require human arbitration. When safety conflicts with helpfulness, or when ethical concerns differ across groups, the model can only execute predefined priorities. But setting those priorities is a social and legal decision that must be owned by the organization.

Third, correctability remains an open problem. Requiring the model not to resist human correction, not to hide capability boundaries, and not to manipulate users to avoid being shut down involves how models represent and respond to being modified. This is far from solved.

Fourth, cross-language and cultural differences matter. The same policy may not be equally applicable across languages or jurisdictions. Direct translation often fails, so clauses need regional localization and independent evaluation.

9. Action Checklist for Teams

• Treat the value document as a version-controlled engineering artifact: atomic clauses, unique IDs, change review, and impact analysis.

• Prioritize building a conflict-resolution method rather than expanding the prohibition list. The former helps with long-tail cases; the latter mostly increases maintenance cost.

• Use dual gates for safety and over-refusal. If either metric fails, block release.

• Build clause-level traceability, so that every production incident can be traced back to specific clauses and data batches.

• Combine training internalization, runtime guardrails, and human review. Do not depend on any single layer.

• Make irreversible operations subject to human confirmation by default. This is one of the highest-leverage engineering constraints.

Moving from rule lists to a value charter is essentially a shift from “post-hoc filtering” to “capability shaping.” It makes safety governance versionable, evaluable, and traceable. This improves the predictability of model behavior in unknown situations and turns the question of “what the model should become” from an implicit annotator preference into an explicit contract that an organization can discuss and revise over time.