Engineering Value Judgments in Decision Systems: Ethical Weights, Utility Models, and Auditability

1. The Starting Point: How Can Incomparable Values Fit on the Same Scale?

When people make choices, they often need to compare outcomes of different kinds. How should animal welfare be weighed against human life? Which forms of suffering should be alleviated first? These questions have long been debated in ethics. Some argue that a class of beings lacks a certain structure and can therefore be ignored; others ask whether that criterion implies such lives are not worth protecting; still others try to formulate the problem as a calculation that resembles mathematics. These discussions show that value comparison is itself a problem that requires explicit definition.

For engineers, this problem is not abstract. Recommendation ranking must trade off user experience against content risk. Resource scheduling must allocate capacity between latency and fairness. Medical triage, insurance review, and disaster response all require priorities to be set across different risks. Any system that automatically ranks options necessarily embeds a value function. The difference is whether that function is written explicitly and whether it has been reviewed.

2. A Basic Utilitarian Modeling Framework

The most common approach is the expected utility model. For each candidate option, list the possible outcomes, their probabilities, benefits, and harms, then compute a weighted sum:

U(option) = Σ p_i × (b_i − w × h_i)

Here, p_i is the probability of outcome i, b_i is the benefit, h_i is the harm, and w is the weight of harm. The value of w determines how sensitive the system is to harm. It is the most critical and most contested parameter in the entire model.

The advantage of this framework is its clear structure: the source of every term can be inspected. Its limitations are also obvious. Probabilities usually come from historical data or expert estimates, and they contain error. Benefits and harms may not be measurable in the same unit. The weight w is often set to 1 by default; this looks neutral, but it already contains a value judgment.

3. Three Common Biases

3.1 Proxy Metric Bias

A system cannot directly measure happiness or harm. It can only use substitutes such as clicks, dwell time, or accident counts. Once a proxy metric becomes disconnected from the real value it is meant to represent, optimization results drift away from the original intent. For example, a recommendation system that optimizes only for dwell time may keep pushing sensational content, damaging user experience and user interests over the long term.

3.2 Weight Provenance Bias

Weights are usually determined by a small number of designers, without participation from stakeholders. With the same model, a different set of weights can produce the opposite conclusion. If weights exist only as constants in code, external reviewers can hardly find problems.

3.3 Extrapolation Bias

Parameters calibrated on small samples or in a single context are often used directly for large-scale or cross-context decisions, where their conclusions can easily fail. Parameters should be labeled with their applicable scope and recalibrated when used outside that scope.

4. Engineering Implementation Options

4.1 Make Value Judgments Explicit

Do not hard-code weights in the codebase. Put them in separate configuration files and record the version, owner, approval time, and rationale. Configuration changes should go through the same review process as code.

4.2 Set Red Lines Before Weighting

Some outcomes should not be offset by any benefit, such as irreversible personal harm or clearly illegal acts. Such constraints should be implemented as hard filters and applied before weighted comparison. Only options that pass the constraints enter utility calculation. This avoids the flawed reasoning that “the total benefit is high enough, so a severe risk is acceptable.”

4.3 Perform Sensitivity Analysis

With the same dataset, the optimal option may differ under different weights. The system should run under multiple weight sets and observe whether the conclusion is stable. If the conclusion flips with the weights, the decision depends on a value judgment and should be explicitly chosen by humans rather than silently decided by the system.

4.4 Keep Auditable Decision Records

Every automated decision should record the input data, the configuration version used, the scores of each candidate option, and the reasons for filtering. When disputes arise later, the team can then locate whether the problem lies in the data, the parameters, or the logic.

Example (pseudocode):

struct Outcome: label, probability, benefit, harm, irreversible
struct Policy: harm_weight, max_irreversible_prob
function expected_utility(outcomes, policy):
    total = 0
    for each o in outcomes:
        assert 0 <= o.probability <= 1
        total += o.probability * (o.benefit - policy.harm_weight * o.harm)
    return total
function is_admissible(outcomes, policy):
    risk = sum of probability for all outcomes where irreversible is true
    return risk <= policy.max_irreversible_prob
function choose(options, policy):
    admissible = options that pass is_admissible
    if admissible is empty: return NONE and route to manual review
    return the option with the highest expected_utility

Pseudocode for sensitivity analysis:

for each w in [0.5, 1.0, 2.0, 5.0]:
    result = choose(options, Policy(harm_weight = w, max_irreversible_prob = 0))
    record (w, result)
if result differs across w:
    mark for human confirmation

The code should not contain fixed conversion constants such as “one entity of one class equals N entities of another class.” Such values are conclusions of ethical debate. They should exist as configuration after review, with applicable scope and owner clearly labeled.

5. Deployment Workflow

1. Requirements phase: list the stakeholders involved in the decision and identify which outcomes are red lines.
2. Design phase: define the utility function and document the source and rationale of each parameter.
3. Validation phase: replay historical cases and check model behavior in boundary situations.
4. Launch phase: run a staged rollout, compare manual decisions with system decisions, and route samples with large divergence to a review queue.
5. Operations phase: conduct regular reviews. When business objectives or social consensus change, reassess the weights.

6. What Should Not Be Quantified

Not all decisions are suitable for weighted-sum processing. Issues involving human dignity, basic rights, or non-compensable harm should prioritize rule-based constraints and human judgment. Quantifiable parts can assist decision-making, but they cannot replace the responsible decision-maker.