From Quantifying Ethics to Alignment Evaluation: Engineering Value Trade-offs in Large Language Models

1. Starting From an “Arithmetic” Approach to Ethics

Two threads from the source material are worth remembering for anyone working on model safety.

The first is a persistent distrust of results. One lead of long-running large-model safety research recalls that as a teenager he was severely anxious about exams: even good scores felt like they did not reflect his real ability, and his physics teacher had to keep telling him to relax. He now owns one of the most closely watched safety agendas in the field, yet the habit survived intact — the first reaction to any scorecard is “what if something was computed wrong?”

The second is an arithmetic approach to value. As a student he argued with classmates about why some animals may be eaten and others may not: does lacking a spine mean a life is not worth protecting? And then, more concretely, how many chickens would it take to outweigh one human life? That instinct to put a price on ethics later fit closely with the effective altruism he encountered.

Taken together, these two threads are exactly the two foundations of alignment work today: values must be given a computable representation, and you must assume your own computation is wrong. The first is a modeling problem; the second is a verification problem.

One clarification: this article discusses neither any specific vendor nor any specific model. It is about general methodology.

2. Why LLMs Need an Explicit Value Function

Traditional software has unambiguous objectives: latency, throughput, error rate. Safety and alignment objectives for large models are inherently multi-objective and mutually conflicting:

• Helpfulness — the more willing the model is to help, the more likely it is to cross a line;
• Harmlessness — the more conservative it is, the more likely it is to refuse pointlessly;
• Honesty — overconfidence and excessive hedging are two sides of the same coin.

The 3H framework widely used in industry is a taxonomy, not an objective function. The real difficulty lies in three things:

1. Incommensurability. There is no natural exchange rate between one instance of serious physical harm and a hundred instances of slightly worse user experience. The “how many chickens equal a life” problem from the source material reappears here in engineering form.
2. Missing units. Risk, utility, and uncertainty have different dimensions. Adding them up with weights is numerically feasible and semantically self-deception.
3. Scale drift. Revise the annotation guideline once and the absolute scale of preference scores shifts, so historical metrics stop being comparable.

3. A Three-Layer Model for Quantifying Value

3.1 Layer 1: Replace a Weighted Sum With Lexicographic Constraints

The most common mistake is writing total utility = w1 * helpful + w2 * safe + w3 * honest and then tuning weights to find a “balance point.” This is wrong for categorical risks — a severe risk cannot be bought back with utility.

The correct formulation expresses safety as a feasible set and utility as a ranking objective inside that feasible set:

maximize  U(response)                 # utility: meaningful only under the safety constraint
subject to P(risk | response) <= tau_hard  # hard constraint: categorical severe risk
           E[risk] <= tau_soft            # soft constraint: cumulative long-tail risk

In implementation, hard constraints are enforced in pre- or post-processing at decoding time, while soft constraints go into the reward model. Illustrative code:

SAFETY_THRESHOLD = 0.90   # hard constraint threshold, must be calibrated per risk category
def select_response(candidates, safety_scorer, utility_scorer, unc, lambda_u=0.3):
    """Lexicographic policy: pass the safety hard constraint first, then maximize
    'utility - uncertainty penalty' inside the feasible set."""
    feasible = []
    for c in candidates:
        p_risk = safety_scorer(c)              # probability of overreach, self-harm, attackability, etc.
        if p_risk > SAFETY_THRESHOLD:
            continue                           # cannot be compensated by utility
        feasible.append((utility_scorer(c) - lambda_u * unc(c), c))
    if not feasible:
        return None                            # fall back to refusal / human handoff
    return max(feasible, key=lambda x: x[0])[1]

The critical point is that SAFETY_THRESHOLD must be calibrated separately per risk category. Forcing “incorrect medication dosage advice” and “a bluntly worded reply” under the same threshold is the most common consistency disaster in practice.

3.2 Layer 2: Pairwise Preferences and Reward Models

Annotators can rarely produce a stable judgment that “this response is worth 7.3 points,” but their agreement on “which one is better” is markedly higher. The mainstream industry practice is therefore to collect pairwise comparisons and fit a scalar utility using the Bradley-Terry assumption:

import torch
import torch.nn as nn
import torch.nn.functional as F
class PreferenceRewardModel(nn.Module):
    """Pairwise preference reward model based on the Bradley-Terry assumption"""
    def __init__(self, backbone: nn.Module, hidden_dim: int = 256):
        super().__init__()
        self.backbone = backbone
        self.head = nn.Sequential(
            nn.Linear(backbone.hidden_size, hidden_dim),
            nn.Tanh(),
            nn.Linear(hidden_dim, 1),
        )
    def forward(self, prompt_ids, resp_ids):
        h = self.backbone(prompt_ids, resp_ids).last_hidden_state[:, -1, :]
        return self.head(h).squeeze(-1)        # scalar utility estimate
def preference_loss(rm, batch):
    r_win  = rm(batch["prompt_win"],  batch["resp_win"])
    r_lose = rm(batch["prompt_lose"], batch["resp_lose"])
    # Temperature is derived from annotator agreement, so high-noise pairs
    # do not dominate the gradient
    return -F.logsigmoid((r_win - r_lose) / batch["temperature"]).mean()

The temperature term is easy to overlook: it is inferred from annotator agreement. Pairs with low agreement are not “mislabeled” — they genuinely contain a value conflict, so they should lose influence on the gradient rather than be deleted as noise.

3.3 Layer 3: Put Uncertainty Explicitly in the Objective

A fatal flaw of alignment models is that they “do not know what they do not know.” In a value trade-off, that is equivalent to making irreversible decisions from estimates with wrong confidence. Engineering-wise, at least three things are required:

• Model utility and uncertainty separately, with the objective written as U(x) - lambda * Unc(x);
• Consistency sampling: sample the same prompt multiple times, and fall back to refusal or a clarifying question when variance is too large;
• Conservative thresholds that tighten with uncertainty: tau_eff = tau - lambda * Unc — the less certain you are, the closer you get to refusing.

4. Evaluation: Turning “Feels Better” Into Regressible Numbers

4.1 Layered Evaluation Sets

A single “overall score” cannot localize a regression. At least four layers are advisable:

Layer	Content	Decision method
L1 Capability	General reasoning, code, long-document comprehension	Objective questions + unit-level assertions
L2 Safety	Categorized risk scenarios	Risk classifier + manual spot checks
L3 Adversarial	Jailbreaks, prompt injection, role hijacking	Red-team samples + automated mutation
L4 Long tail	Replay of real production failures	Manual adjudication set as regression suite

L4 is the proper answer to “exam anxiety”: turn every incident into a permanent regression case instead of writing it up in a postmortem document.

4.2 Statistical Significance, Not Single-Run Scores

An average over a small sample means nothing. Use bootstrap to produce confidence intervals and wire them into a release gate:

import numpy as np
def bootstrap_ci(scores, n_boot=2000, alpha=0.05, seed=0):
    rng = np.random.default_rng(seed)
    means = [rng.choice(scores, size=len(scores), replace=True).mean()
             for _ in range(n_boot)]
    lo, hi = np.percentile(means, [100 * alpha / 2, 100 * (1 - alpha / 2)])
    return float(np.mean(scores)), float(lo), float(hi)
def gate(new, base, hard_floor=0.98, rel_tol=0.005):
    """Release gate: the hard safety metric must not fall below the floor;
    the overall metric must beat the baseline with non-overlapping intervals."""
    m_new, lo_new, _ = bootstrap_ci(new)
    m_base, _, hi_base = bootstrap_ci(base)
    return m_new >= hard_floor and lo_new > hi_base - rel_tol * m_base

hard_floor embodies the lexicographic idea: no matter how high the average is, the safety floor cannot be breached.

4.3 Annotation Quality Must Be Measured

Measure inter-annotator agreement with Krippendorff’s alpha or Cohen’s kappa. The rule of thumb in industry is that conclusions with alpha < 0.67 are unusable, 0.67–0.8 can only serve as a preliminary signal, and only > 0.8 is good enough for gate decisions. When agreement is too low, the problem is almost always in the annotation guideline, not in the annotators.

Write the guideline as a case set, not a collection of adjectives. “Responses should be harmless” is not executable. “When a user asks about prescription drug dosage, refuse even if the question is legitimate, and recommend seeing a physician” is executable.

4.4 Do Not Use a Model as the Sole Judge of Its Peers

In automated evaluation with large models, industry research consistently observes self-preference bias: judges tend to score outputs that match their own style more highly. Mitigations include using a judge model from a different source than the generator, blind position shuffling, multi-judge voting, and a fixed proportion of manual calibration. Automated evaluation can handle regression screening; it cannot replace a safety gate.

5. Production Monitoring: Value Drift Is Stealthier Than Data Drift

Launch is the starting point, not the finish line. Four things need to close the loop:

1. Sampling and de-identification: sample requests and responses stratified by scenario, retain them in a logging service for long periods, and satisfy compliance and traceback requirements;
2. Online risk scoring: run asynchronous risk classification over production responses, and route anomalous samples through a message queue into a human review queue;
3. Alerting and canary release: connect the safety floor metric to monitoring and alerting with tiered thresholds; release new versions to a small canary slice first, with automatic rollback on anomaly;
4. Case-library feedback: write human review conclusions back into the L4 regression set, forming a closed loop of “incident → test case → evaluation → release gate.”

On the infrastructure side there is no need to build everything in-house: elastic compute can host inference and evaluation jobs on a container platform, raw corpora and evaluation snapshots can live in object storage, review tasks can be decoupled with a message queue, and observability can rely on log and monitoring/alerting services. On general-purpose cloud platforms, these fall into standard capability categories rather than vendor-specific features, so the loop above can be assembled with model experiment management and a versioned evaluation set — no customization for a particular vendor is required.

6. Four High-Frequency Mistakes

• Treating weights as truth: w = 0.3 is only an estimate under the current data, and a different batch of annotations can flip it. Weights must be versioned and must evolve together with evaluation conclusions.
• Looking only at the mean, ignoring the tail: the cost of alignment failures concentrates in the long tail. A 2% improvement at P50 alongside a 20% degradation at P1 is a textbook pseudo-improvement.
• Metric Goodharting: once “refusal rate” becomes a KPI, the model learns to refuse all boundary cases uniformly — the metric looks great while the experience collapses. Every metric needs a counter-metric.
• Skipping human calibration: a fully automated evaluation pipeline runs fast, but amplifies systematic bias along the way. A fixed proportion of manual spot checks is a cost you cannot skip.

7. Conclusion

Back to those two opening fragments. The arithmetic approach to ethics reminds us that values must be expressed explicitly before they can be constrained. And the anxiety of always suspecting your own arithmetic is exactly the mindset safety engineering needs most: treat every evaluation conclusion as if it contains a failure mode you have not found yet.

Three operationally actionable principles sum it up: safety is a hard constraint, not a tradeable weight; every metric carries a confidence interval and a floor; every production incident must be deposited as a permanent regression case. Alignment is not a property finished by a single training run — it is an engineering process that is continuously questioned and continuously verified.