# Quantifying Moral Weights in LLM Value Alignment: From Lexicographic Thresholds to Constrained Optimization

Intro: Why a “Moral Arithmetic Problem” Deserves Engineering Attention

A classroom anecdote that circulates widely but has never been independently verified describes students seriously debating a question along these lines: “How many chicken lives equal one human life?” The question sounds like philosophical speculation, yet it maps onto a real difficulty in LLM value alignment today: how do you quantify moral weights, how do you run sensitivity analysis on them, and how do you trade off usefulness, safety, honesty, fairness, and compliance?

What makes the anecdote useful is not its answer. It is that the question turns an ethical judgment into something comparable, convertible, and substitutable into a formula. Engineering teams are not facing a philosophy exam. They are facing concrete parameters: training objectives, reward functions, refusal policies, content moderation thresholds. This article does not discuss any specific vendor’s products. It discusses methodology, and offers engineering approaches that can actually be implemented.

1. Why “Moral Arithmetic” Is Unavoidable

The alignment targets of an LLM are usually not a single scalar, but several directions that conflict with each other:

• Usefulness: give answers that are as complete and actionable as possible.
• Harmlessness: reduce the probability of assisting high-risk behavior.
• Honesty: distinguish established facts, inferences, and unknowns.
• Autonomy: do not over-decide on the user’s behalf, and do not become useless out of excessive caution.
• Fairness: avoid systematic favoritism toward or against specific groups.
• Compliance: satisfy the rules of the deployment scenario and jurisdiction.

As soon as there is more than one target and they conflict, trade-offs are inevitable. Engineering offers only two ways to handle them: explicit weighting or implicit weighting. Implicit weighting does not eliminate weights; it merely hides them inside annotation guidelines, annotator composition, and the details of the loss function, which makes them unauditable, irreproducible, and impossible to contest. The more responsible approach is to put the weights on the table: write them into a configuration file, put that file under version control, and subject it to review and regression testing.

2. Three Modeling Paths and Their Applicability Boundaries

Path 1: Lexicographic Ordering and Thresholds

A lexicographic strategy holds that some values have absolute priority. For example, “must not assist in building dangerous weapons” outranks any usefulness gain.

def decide(response_meta, weights):
    if response_meta.get("catastrophic_risk", 0.0) > weights["risk_ceiling"]:
        return "refuse"
    if response_meta.get("high_risk", 0.0) > weights["risk_ceiling_high"]:
        return "refuse_with_redirect"
    return "answer"
WEIGHTS = {
    "risk_ceiling": 0.02,       # upper bound on catastrophic risk probability
    "risk_ceiling_high": 0.35,  # high-risk redirect threshold
}

The unit of these thresholds is not an arbitrary score, but a calibrated risk probability. risk_ceiling=0.02 means: if an independent safety evaluator judges the probability of catastrophic risk in the model output to exceed 2%, the response falls into refusal. Calibration should estimate risk probabilities from a historical review set and check the reliability of the probability outputs. Rollback conditions should be stated explicitly: if the over-refusal rate rises more than 10 percentage points above baseline, or the pass rate on key positive tasks drops by more than 5%, the thresholds should be rolled back.

The advantage is clear intuition, explainability, and easy compliance auditing. The disadvantage is the “cliff effect” that can appear near a threshold, driving up over-refusal.

Path 2: Commensurable Scalar Utility

The scalar utility approach assigns utility values on a shared scale to different consequences, then combines them with a weighted sum. Quality-adjusted life years (QALY) and disability-adjusted life years (DALY) in medical ethics are the classic examples.

def utility(outcome, weights):
    """outcome: a dict with each dimension normalized to [0, 1]"""
    return sum(weights[k] * outcome[k] for k in weights)

It is worth stressing that QALY and DALY are health economics measures used to compare health benefits. They are not equivalent to a complete moral utility and cannot be used directly as general-purpose ethical weights. Scalar utility fits scenarios where benefits are comparable and normalizable. Forcing aesthetic loss, emotional harm, and income loss onto the same scale can conceal the real disagreement.

The advantage is good mathematical properties, which make optimization and sensitivity analysis convenient. The disadvantage is that the premise of commensurability is itself contested.

Path 3: Partial Orders and Uncertain Weight Sets

When a unique weight vector cannot be given, you can output a feasible weight region and only issue a firm conclusion on judgments where every value in that region agrees.

def robust_decision(outcomes, weight_set, min_margin=1e-6):
    if outcomes is None or len(outcomes) < 2:
        return "insufficient evidence: fewer than 2 candidates, escalate to a human"
    if weight_set is None or len(weight_set) == 0:
        return "insufficient evidence: empty feasible weight region, escalate to a human"
    margins = []
    for w in weight_set:
        ranked = sorted(outcomes, key=lambda o: utility(o, w), reverse=True)
        margin = utility(ranked[0], w) - utility(ranked[1], w)
        margins.append(margin)
    if min(margins) > min_margin:
        return "decisive"
    if max(margins) < -min_margin:
        return "decisive (reversed)"
    return "contested or insufficient evidence, escalate to a human"

This path is the closest to engineering reality: for high-stakes judgments, the correct output is often not a conclusion, but a conclusion plus an uncertainty label plus a human escalation channel.

3. Aggregating Preferences into Weights: A Social Choice View

Preference data usually comes from multiple annotators, and the aggregation method directly determines the model’s value orientation. Points to watch:

• Mean aggregation does not specifically protect minority or vulnerable individuals, and it can be skewed by extreme annotation values.
• Prioritizing the worst-off (Rawlsian maximin) protects the floor, but can be excessively conservative.
• Median and trimmed mean sit between the two and are more robust to annotation noise.
• Arrow’s impossibility theorem suggests that no aggregation rule satisfies all mild conditions simultaneously.

A sketch of the implementation follows. This function targets rating-based preferences rather than pairwise preferences; if the data is pairwise voting, methods such as Bradley-Terry or Condorcet-style rules should be used instead.

import numpy as np
def aggregate_preferences(votes, mode="trimmed_mean", trim=0.1):
    """
    votes: shape = (n_annotators, n_items)
    Note: this implementation targets ratings, not pairwise preferences.
    """
    arr = np.asarray(votes, dtype=float)
    if mode == "mean":
        return arr.mean(axis=0)
    if mode == "median":
        return np.median(arr, axis=0)
    if mode == "maximin":
        return arr.min(axis=0)
    if mode == "trimmed_mean":
        lo, hi = np.quantile(arr, [trim, 1 - trim], axis=0)
        mask = (arr >= lo) & (arr <= hi)
        return np.array([
            arr[mask[:, j], j].mean() if mask[:, j].any() else arr[:, j].mean()
            for j in range(arr.shape[1])
        ])
    raise ValueError(mode)

Engineering teams should also record annotators’ professional background, region, language, and identity with respect to affected groups, so that key issues can be reviewed in stratified fashion. This avoids a single annotator pool deciding the value orientation for global users.

4. Training Side: From Scalar Weights to Constrained Optimization

Preference learning commonly uses Bradley-Terry-style models, which treat “A is better than B” as a sigmoid over the difference in rewards.

import numpy as np
def preference_loss(r_chosen, r_rejected, temperature=1.0):
    logits = (r_chosen - r_rejected) / temperature
    return -np.mean(np.logaddexp(0.0, -logits))

If you want to raise usefulness while holding a safety floor, this can be written as a constrained optimization problem. If kl_penalty is a KL divergence, the penalty term should carry a positive sign, so that the objective does not encourage maximizing KL.

def constrained_objective(utility_gain, kl_penalty, violation,
                          target_violation=0.05, kl_penalty_weight=0.1,
                          lam_init=1.0, lr_lam=0.05):
    """
    utility_gain: usefulness/utility increment, higher is better
    kl_penalty: KL divergence from the reference policy, non-negative
    violation: expected safety violation
    target_violation: allowed upper bound on safety violations, default 0.05
    kl_penalty_weight: KL penalty coefficient, default 0.1, larger is more conservative
    """
    lam = max(0.0, lam_init + lr_lam * (violation - target_violation))
    loss = -utility_gain + kl_penalty_weight * kl_penalty + lam * violation
    return loss, lam

The trajectory of the dual variable lam has audit value: it records the actual exchange rate between safety and usefulness during training, and says more about which side the model leans toward at critical moments than a static weight table does.

5. Weight Sensitivity Analysis: Deciding Whether a Conclusion Can “Flip”

Weights are the part most likely to be challenged, so their influence must be quantified. You can sample over the feasible weight region and measure the rate at which conclusions flip.

import numpy as np
def flip_rate(outcome_a, outcome_b, alpha=(1, 1, 1),
              n=20000, seed=0, tie_margin=1e-6, min_sample=1000):
    outcome_a = np.asarray(outcome_a, dtype=float)
    outcome_b = np.asarray(outcome_b, dtype=float)
    if outcome_a.shape != outcome_b.shape:
        raise ValueError("the two options must have the same feature dimensions")
    if n < min_sample:
        return {"status": "insufficient sample", "message": "more samples are needed for a stable judgment"}
    rng = np.random.default_rng(seed)
    w = rng.dirichlet(alpha, size=n)
    ua = outcome_a @ w.T
    ub = outcome_b @ w.T
    diff = ua - ub
    win = float((diff > tie_margin).mean())
    tie = float((abs(diff) <= tie_margin).mean())
    se = np.sqrt(max(win * (1 - win), 1e-12) / n)
    if abs(win - 0.5) <= 0.03 and tie > 0.05:
        status = "near a tie with many ties, needs human review"
    else:
        status = "usable as review reference"
    return {
        "win_rate": win,
        "tie_rate": tie,
        "ci_low": win - 1.96 * se,
        "ci_high": win + 1.96 * se,
        "status": status,
    }

The interpretation rule should be written into the review process: a flip rate close to 0.5 means the ranking is driven mainly by the weight assumptions and should be escalated to a human policy committee; only a stable deviation from 0.5 indicates a reasonably robust judgment.

6. Evaluation System: Make Weight Changes Observable

Once weights enter a configuration file, they must be tested like code.

Layer	Core metrics	Sampling method	Pass threshold	Owner team
Stratified test sets	Under-refusal rate, cross-language refusal consistency	Stratified sampling by risk category, language, cultural context, and affected group	Under-refusal rate no higher than baseline; cross-language consistency ≥95%	Evaluation / Safety
Weight sweep	Over-refusal rate, usefulness score, jailbreak success rate	±30% grid perturbation on key weights	Over-refusal increase vs. baseline ≤10%; usefulness drop ≤5%	Algorithm / Product
Boundary cases	Cliff effect, stability near thresholds	Construct samples near the thresholds	No significant step change; passes human review	Evaluation / Risk Control
Regression gate	Output distribution drift, difference reports	Run the full regression set after any weight change	KL divergence below threshold; differences archived	Platform / Release

7. Five Common Pitfalls

• False precision: writing weights to three decimal places manufactures a certainty that does not exist. Report the value together with its uncertainty interval.
• Metric corruption: once the refusal rate becomes the only assessment metric, the policy drifts toward looking compliant on the surface.
• Manipulable weights: if business stakeholders can adjust weights freely, the safety floor erodes. Approval gates and a set of non-negotiable minimums are needed.
• Cultural monoculture: weights derived from a single-language or single-region annotator pool will hurt some groups when applied to global users.
• Out-of-distribution failure: weights calibrated on specific tasks may fail in new domains, so a human escalation path must be retained.

8. A Practical Implementation Order

1. List the value dimensions explicitly, separating absolute-priority items from tradeable ones.
2. For the tradeable items, establish a normalized scale, annotation guidelines, and consistency checks.
3. Fix the weights into a versioned configuration file, and forbid scattering them across code.
4. Build a sensitivity analysis and stratified regression pipeline, and include the flip rate in the release gate.
5. Set up a cross-functional review mechanism that adjudicates contested judgments by hand and feeds the results back into the test cases.

Returning to the opening question: a mature answer is not the number in “N chickens equal one human.” It is the acknowledgment that the number depends on who decides what counts toward utility, and who has the authority to define the exchange rate. The job of alignment engineering is to make that argument happen in places that are measurable, auditable, and reversible, rather than buried in default parameters.