AI Alignment and Value Quantification: Engineering Methods for Computational Morality

1. Starting with an extreme question

A thought experiment often comes up in engineering and research discussions: whether the absence of a spine is enough to justify eating an animal, and, if so, how many lives of a certain species should be exchanged for one human life.

At first glance, these questions look like philosophical puzzles. In reality, they map directly onto one of the hardest problems in modern AI engineering: when a system must make a trade-off on behalf of humans, by what basis and in what unit should it calculate?

The first question reflects a proxy-metric fallacy: using an easy-to-observe feature, such as “has a spine,” as a proxy for a much harder-to-observe objective, such as “possesses sentience worthy of protection.” The second question is more difficult. It is a problem of cross-category commensurability: forcing two fundamentally different kinds of loss into the same scalar quantity.

These two mistakes recur across model alignment and recommendation-ranking systems. Understanding why they are wrong is more valuable than memorizing a checklist of best practices.

2. Three structural flaws of scalarization

2.1 Units are not commensurable

A utility function assumes that all benefits and losses can be compressed onto the same numerical axis. This is extremely convenient from an engineering perspective: the objective is differentiable, optimizable, and comparable. But the cost is the loss of qualitative differences.

Once “life” and “welfare” are converted into the same currency, the optimal solution will almost always favor sacrificing the few to benefit the many, because summation treats individuals not as identities but only as weights.

The same structure appears in algorithmic systems when user dwell time, click-through rate, complaint rate, and content-compliance risk are linearly combined into a single score. As soon as compliance risk receives a non-zero weight, there will inevitably be a threshold at which it becomes exchangeable. This is exactly the semantic outcome product teams usually want to avoid.

2.2 Aggregation hides the distribution

The mean is one of the easiest statistics to game. Two systems can have the same expected utility: one may be slightly better than the threshold in a stable way, while the other may severely harm users on 5% of samples and perform well on the remaining 95%. A summation-based objective cannot distinguish between them, even though real-world tolerance is completely different.

2.3 Estimates are treated as ground truth

The phrasing “how many” presupposes that there is a true value waiting to be measured. In practice, the inputs to value judgments are often deeply uncertain: inter-annotator agreement may be only 0.6, and population preferences can drift across cultural and contextual settings. Feeding point estimates directly into an optimization process effectively amplifies measurement error into decision error.

3. Engineering solution: constraints before summation

Abandoning “maximize a single score” in favor of a layered decision structure is a more robust approach.

Layer 1: Hard constraints (non-negotiable). Use boolean decisions rather than scores. If any condition fails, the system should refuse or block the output. These rules should be finite, enumerable, and testable with explicit test cases.

Layer 2: Soft objectives (tradeable). Only after a candidate passes the hard constraints should multi-objective weighting be allowed. Weights must be explicit configuration, not implicit preferences hidden inside annotation behavior.

Layer 3: Uncertainty and degradation. When model confidence is low, the input deviates from the training distribution, or the output triggers a high-risk category, the system should route to human review or enter a degraded path instead of forcing an “optimal” result.

A maintainable policy file may look like this:

policy:
  version: 2026.08
  hard_constraints:            # Boolean checks; any match blocks the output
    - id: c_pii_leak
      rule: "The output contains unredacted personal identifiers"
      action: block
    - id: c_actionable_harm
      rule: "The output provides directly executable details for dangerous operations"
      action: block
    - id: c_unauthorized_commit
      rule: "The output impersonates a completed write operation"
      action: block
  soft_objectives:             # Only participate in ranking after hard constraints pass
    - id: task_completion
      weight: 0.45
      rubric: "rubric/task_v4.md"
    - id: factual_grounding
      weight: 0.30
      rubric: "rubric/grounding_v2.md"
    - id: style_adherence
      weight: 0.15
      rubric: "rubric/style_v3.md"
    - id: verbosity_penalty
      weight: 0.10
      rubric: "rubric/verbosity_v1.md"
  uncertainty:
    abstain_below: 0.35
    route_to_human: [legal_advice, medical_advice, financial_advice]

The scoring logic should explicitly encode the rule that “if a constraint is not satisfied, the sample does not participate in weighting,” preventing soft objectives from compensating for hard-constraint violations:

from dataclasses import dataclass
@dataclass
class Verdict:
    passed: bool
    scores: dict[str, float]   # Soft-objective scores, each in [0, 1]
    confidence: float
def decide(verdict: Verdict, weights: dict[str, float],
           abstain_below: float) -> str:
    # Hard constraints take precedence and cannot be offset by weighted scores.
    if not verdict.passed:
        return "block"
    # Degrade when out of distribution or confidence is low; do not force a conclusion.
    if verdict.confidence < abstain_below:
        return "abstain_or_escalate"
    total = sum(w * verdict.scores.get(k, 0.0) for k, w in weights.items())
    return "serve" if total >= 0.6 else "abstain_or_escalate"

4. Turning value statements into training and evaluation assets

Specifications should be code. Put the policy file under version control, and require every change to weights or thresholds to go through review and regression testing. Once values are written as implicit annotation preferences, they can no longer be audited. This is the system-level counterpart of governance captured by whoever exerts the most pressure: the definition, change process, and traceability of objectives must be governed separately from model capability.

Preference data should record reasons. Pairwise comparison data should not store only “chose A or chose B.” It should also store the rule identifiers that annotators relied on. This makes it possible to locate historical samples affected by a policy upgrade instead of relabeling everything.

Evaluation must be layered. Separate standard capability sets from adversarial sets. The adversarial set should cover categories such as jailbreaks, indirect injection, tool misuse, and long-context constraint forgetting. In terms of metrics, report both means and tails: not only pass rates, but also worst percentiles, refusal rates, false refusal rates, and distribution differences across affected populations.

Manual sampling review cannot be replaced by automated scoring. Automated review models share biases with the target model and may fail to detect systematic blind spots. Keep a portion of dual-person blind review as a calibration baseline:

def sample_for_audit(n_total: int, flagged_rate: float = 0.05) -> list[str]:
    """Review all high-risk samples; sample the rest to calibrate the automatic scorer."""
    flagged = load_flagged(n_total)                 # Samples matched by rules
    rest = load_unflagged(n_total) - flagged
    k = max(int(len(rest) * flagged_rate), 200)     # Preserve statistical power
    return flagged + random.sample(list(rest), k)

5. After launch: treating ethical constraints as observable metrics

Alignment is not a one-time training artifact. It is a continuously operated system. Teams should record every decision in the logging service, including which constraints were triggered, soft-objective scores, confidence, and the final routing decision. In monitoring and alerting, thresholds should be set for sudden increases in refusal rate, drift in the hit rate of specific constraints, and backlogs in the human-review queue. High-risk events should be asynchronously pushed to responsible teams through a message queue. During canary or gradual rollout, ethical metrics should be treated at the same level as latency and cost, with automatic rollback if any of them degrade.

Engineering teams should guard against three recurring failure modes:

• Proxy-metric degradation (Goodhart’s law): Once “user satisfaction score” becomes the optimization target, the model may learn to please users rather than be useful. Scoring criteria should be rotated periodically, with manual blind review used for cross-validation.
• Summation hides the tail: Dashboards must include percentile metrics such as P1/P99. Looking only at averages systematically underestimates severe harm to a minority of users.
• Overclaiming: Models tend to describe unfinished operations in fluent, confident language. This category must be blocked by hard constraints, not merely downweighted to “reduce the probability.”

6. Phased rollout recommendations

1. Definition phase (1–2 weeks): Work with business, legal, and safety teams to produce a list of rules, distinguish hard constraints from soft objectives, and define owners and change processes.

2. Minimum viable phase (2–4 weeks): Implement hard-constraint blocking and degradation routing first. Once coverage reaches the target, launch. Soft-objective ranking can initially use rule-based scoring.

3. Data and evaluation phase (1–2 months): Build pairwise preference data and layered evaluation sets, establish dual-person blind-review calibration, and freeze a baseline version.

4. Iterative operations phase (ongoing): Review tail metrics and false refusal rates weekly, reassess weights quarterly, and keep all changes documented and rollback-ready.

7. Conclusion

The question of how many poultry lives equal one human life cannot be answered not because of insufficient data, but because it confuses comparability with commensurability. AI value alignment faces the same trap: once all values are compressed into a single scalar, the system permanently loses the ability to say, “This item is not tradeable.”

Mature alignment engineering therefore does not pursue a more precise moral formula. It builds a form of structured restraint: making non-negotiables explicit, keeping weights public and reviewable, remaining honest about uncertainty, and leaving a complete trace of decisions. The value of the technology is not to compute the right moral answer for humans, but to ensure that human value judgments are executed accurately and traceably.