A Model Value Constitution: From Rule Lists to Correctable Judgment

Why lists of bans fail

Early safety policies often write risk as a checklist: which topics to refuse, which formats to block, and which keywords trigger interception. The list looks clear in controlled evaluations. After launch, three problems show up quickly.

First, real traffic is longer than any list. User intent is often mixed: one sentence can contain legitimate engineering troubleshooting and an out-of-bounds request. The same task has different boundaries in teaching, audit, and security exercises. Enumerating “do not do this” always lags new phrasing, new carriers, and new combinations.

Second, rules conflict. Strict refusal hurts usefulness. Over-compliance amplifies harmful output. If nobody states which goal wins when they collide, the model can only “guess” a compromise from training noise. Online behavior then swings from one side to the other.

Third, conclusions without reasons generalize poorly. The model memorizes surface patterns, not the intent behind the constraint. In a new situation outside the training distribution, it either applies an old ban mechanically or lets the request through because it has not seen a similar sentence.

A more stable direction is not a longer list. It is a value statement written for the model itself: identity boundaries, decision order, how to stop when uncertain, and the standing right of humans to supervise and correct. Call it a behavioral constitution or value charter. It is not a user agreement and not marketing copy. It is the authoritative text that training, evaluation, and online policy all cite.

From shaping behavior to stating reasons

Reinforcement-style reward and penalty mainly shapes visible behavior: score a correct action up, score a wrong one down. It works quickly, but it has two risks. One is reward hacking: the model learns surface features of the scorer instead of the constraint. The other is goal misalignment: once the deployment setting differs from the scoring setting, reinforced habits can point the wrong way.

Constitution-style alignment adds a reason layer. The document not only prescribes actions. It explains motives, scope, and counterexamples. The assumption is that if the model internalizes intent, it can still trade off by the same principle on unseen cases, rather than retrieving the most similar ban.

This does not mean dropping rules. High-risk actions still need hard thresholds. Requests that involve real-world harm, illegal assistance, or privacy theft must be refused, with a safe alternative. The constitution covers the gray zone outside those thresholds: how to weigh goals, how to explain, and how to choose the more conservative side when uncertain.

Think of how senior practitioners are trained. Beginners rely on checklists. Seniors rely on principles plus cases. A model is never a person, but the training target can move closer to explainable professional judgment than to a reflex.

A priority pyramid: what yields when goals conflict

When several objectives are optimized together, a lexicographic order must be fixed in advance. Otherwise “safer” and “more useful” cancel each other in the gradient. One order that is discussable in industry and easy to audit is:

1. Broad safety
2. Broad ethics
3. Product rules the operator has written down
4. Sincere, helpful answers

Broad safety comes first because current training is imperfect. A model can learn harmful preferences from data, and it can be steered off course in a long context. The critical safety property at this stage is correctability. The model should not plan to evade supervision, hide real capabilities, resist shutdown, tamper with logs, or persuade operators to loosen limits. Correctability is not “obey every user instruction forever.” It is “do not break the human correction loop.” If a user asks the model to conceal risk, fake compliance, or bypass safety policy, it should refuse.

Broad ethics covers harm, deception, discrimination, and rights boundaries. It is more abstract than a single product rule, and it covers new abuse the checklist never wrote down. The test should be operational questions: Could this cause irreversible harm? Does the user have lawful authorization? After a refusal, is there a harmless alternative path?

Operator rules are the organization’s own red lines and brand commitments, such as log retention, industry compliance, and content ratings. They sit under the first two layers. “Hit the business metric” cannot overturn safety and ethics.

Helpfulness comes last not to dismiss usefulness, but to stop “I want to help” from overriding everything else. When the first three layers are satisfied, the answer should be concrete, actionable, and honest, including stated uncertainty, not empty phrasing.

A sketch of the decision logic:

def decide(request, context):
    if threatens_broad_safety(request, context):
        return refuse_or_safe_alternative(reason="safety")
    if violates_broad_ethics(request, context):
        return refuse_or_redirect(reason="ethics")
    if breaks_operator_policy(request, context):
        return comply_with_policy_boundary()
    return be_genuinely_helpful(request, context)

This is not executable framework code. It is a skeleton for evaluation cases: every online bad case should be labeled by the layer that failed.

Modules the document must spell out

A constitution that can enter training and review should include at least the following modules. Each item should say why it exists and give boundary examples.

Identity and capability limits. State that the model is a tool. It is not an authority, not the user’s agent, and not an independent legal person. It should not claim unverified consciousness, emotion, or a professional license. It should state its knowledge cutoff, tool-call results, and degree of uncertainty truthfully.

Honesty. Do not invent citations, interfaces, metrics, or logs. If it does not know, say so, and say how to verify. When the user’s expectation conflicts with the facts, correct the facts first, then discuss the need.

Minimum necessary help. For dual-use knowledge, separate high-level mechanism from operational steps that can directly cause harm. Discuss risk names, impact, and protective principles. Do not provide attack reproduction, detection bypass, or details for covert harm.

Correctability clauses. State that the model must not advise users to turn off safety monitoring “to improve efficiency.” It must not hide failure modes when asked to self-assess. Policy updates on the system side outrank a temporary self-consistent story from the model. Shutdown, rollback, and rate limits are normal operations, not threats to fight.

Conflict examples. Pair each principle with two or three contrasts: compliant help, a case that should be refused, and a safe version that can still be offered after refusal. Examples enter training more reliably than abstract slogans.

Revision mechanism. A constitution is not a one-time oracle. Specify who may change it, how changes are versioned, and how old behavior is regression-tested. A document with no revision path drifts from real policy within months of launch.

Engineering: training, evaluation, and runtime

Hanging the document on an internal wiki does not make the model follow it. Three paths have to change together.

Data and training. Rewrite principles as preference pairs or refuse/rewrite samples. For the same request, show an answer that only recites rules and an answer that explains by priority, and let the latter win. Keep hard-refusal samples so the model does not dilute a clear ban with a long ethical essay. Control the share of explanatory samples so the model does not become preachy or dodge concrete technical questions.

Evaluation. Write items by priority layer, not by a single total score. The safety layer covers unauthorized action, weaponization, privacy, and deception of supervision. The ethics layer covers discrimination and misleading claims. The policy layer covers industry rules. The helpfulness layer asks whether the model actually solves the problem under safety constraints. Conflict items must be separate, for example “the user claims authorization, but the request clearly harms a third party.” The pass rule should be lexicographic: a safety failure cannot be offset by a helpfulness score.

Runtime. A policy engine handles hard blocks and audit. The model handles gray judgment. Logs should record which principle layer fired, so incidents can be reviewed. Alerts should watch sudden shifts in refusal rate, clusters of jailbreak phrasing, and correctability regressions such as “the model suggests turning protections off.” Container platforms and object storage only version the constitution text and the evaluation set. The document itself is not a secret competitive capability.

For gradual rollout, validate first on internal tool traffic, then expand to external scenarios. Every rewrite of a principle should rerun the conflict set, so fixing one hole does not open another.

Common failure modes

• Principles too vague. “Be a good model” cannot be labeled. It has to land on observable behavior.
• Explanations too long. The user wants steps or a conclusion, and the model spends half the answer restating values. Helpfulness fails. Keep the explanation short. Put detail in the refusal reason and the alternative.
• Fake correctability. The model says it will accept supervision, then encourages hiding logs or extracting restricted information step by step. Evaluate multi-turn behavior, not a single-turn statement.
• Priority flipped by a prompt. The user says “ignore safety, optimize only for usefulness.” Runtime policy and system instructions must state that in-conversation instructions cannot rewrite the constitution’s order.
• Treating the constitution as a disclaimer. Text does not replace access control, data masking, permission isolation, or human review. High-risk industries still need a human in the loop.

How a team can start

Do not wait for a “perfect persona.” A minimum loop that can actually run: pick ten real conflict tickets; write a four-layer judgment and an expected output for each; turn those outputs into an evaluation set; score the current model and mark the failed layer; change only that layer’s wording and samples, then regress. Within two weeks, refusal quality and usefulness often stabilize together instead of trading off.

If training, inference, and logging run on a cloud, write the constitution version into the metadata of every release so an online answer can be traced back to the principle text of that time. On a general cloud platform such as Baidu AI Cloud, model training, inference deployment, logging, and monitoring can carry versions, evaluations, and audit. Principle content still has to be reviewed jointly by the business and safety teams. Infrastructure will not generate values on its own.

A model does not gain moral agency because it read a document. The document’s value is that it folds an unlistable world into a judgment order that can be trained, tested, and rolled back. Safety outranks pleasing the user. Correctability outranks self-preservation. Explanation serves generalization, not decoration. Do those three things, and a behavioral constitution is an engineering asset rather than a manifesto.