From Moral Computing to AI Ethics Engineering: Boundaries and Practice

1. The Technical Metaphor of Thought Experiments: Compressing Values into a Scalar

Thought experiments such as “How many chickens’ lives are equivalent to one human life?” are unsettling because they attempt to compress values that may not be comparable into a single numerical scale.

AI systems and software engineering face a similar challenge. Complex objectives are often represented by a single metric: click-through rate, conversion rate, risk score, user satisfaction, or percentage efficiency improvement. Metrics are not inherently problematic. The danger arises when a metric becomes the only objective. The system then evolves toward what is easiest to optimize rather than toward the values people actually care about.

Technical communities have sometimes debated whether the presence of a backbone should determine whether a life deserves protection, followed by questions about whether different forms of life can be exchanged or converted into one another. The value of these debates is not that they produce a universal answer. Instead, they expose an engineering reality: every quantitative model contains simplifications, and those simplifications must be declared, constrained, and audited.

Ethical computation cannot replace ethical judgment. It can only serve as one input into a broader decision-making process.

2. Why a Single Utility Function Is Dangerous

Representing an ethical problem with a single utility function creates several common risks.

Value Collapse

When multiple dimensions are combined through weighted addition, some non-negotiable boundaries may be offset by other benefits. For example, a system that optimizes “total benefit minus risk cost” may accept risks that should never be accepted simply because the expected benefits are high enough.

Metric Gaming

When a system or organization optimizes around a metric, the metric can shift from being a proxy for value to becoming value itself. This resembles a common problem in reward-function design: a model finds weaknesses in the evaluation process instead of achieving the designer’s actual intent.

Hidden Uncertainty

Ethical decisions often involve disagreement and uncertainty. A single score can create a false sense of certainty, causing cases that should receive human review to be approved automatically.

For these reasons, a more robust engineering approach is not to pursue a “perfect ethics score.” Instead, systems should prioritize constraints, keep multiple objectives visible, represent uncertainty explicitly, and make decisions traceable.

3. An Engineering Approach: Constraint-First, Layered Decisions

1. Treat Hard Constraints as Non-Negotiable

Start by expressing non-negotiable boundaries as hard constraints rather than as soft penalties. Examples include:

• Sensitive data must not be processed without the required authorization.
• High-risk scenarios involving minors must enter a dedicated review process.
• High-risk automated decisions must retain a path to human review.

Hard constraints should be evaluated at the earliest stage of policy assessment. No expected benefit should be able to override them.

2. Use Multiple Objectives Instead of a Single Score

Safety, fairness, privacy, usability, and cost should be modeled as separate objectives. The system should preserve the score and confidence interval for each dimension rather than compressing them into an opaque total.

A system may provide a recommendation, but it should not hide how different dimensions contributed to that recommendation. In scenarios that require ranking, reviewers should be able to see why an item was placed near the top.

3. Establish Decision Layers and Escalation Rules

Decision-making should be layered:

• Low-risk, high-confidence requests can be processed automatically.
• Medium-risk requests can go through rule checks and sampled review.
• High-risk or highly uncertain requests should require human review.

Escalation rules should be written into policy configuration rather than relying on individual developers’ experience.

4. Version and Roll Back Policies

Ethics policies require version management just as models do. Each policy change should record its rationale, scope of impact, evaluation results, and rollback conditions.

Release gates should include an ethics impact assessment, red-team testing, regression testing against evaluation sets, and monitoring and alerting configuration.

4. Evaluation and Auditing: Making Ethical Decisions Verifiable

Ethical requirements need to be translated into testable engineering specifications. Three categories of evaluation are useful.

• Rule evaluation: Check whether hard constraints have been violated, including requirements related to authorization, consent, age protection, and data minimization.
• Behavioral evaluation: Observe system outputs in representative and adversarial scenarios, recording false approvals, false blocks, and missing explanations.
• Process evaluation: Verify whether human reviews are completed in a timely manner, audit logs are complete, and alerts are properly resolved.

Logging and monitoring are general-purpose infrastructure for this work. Logs should capture the input context, policy version, scores for each dimension, review conclusions, and final action.

Object storage can be used to preserve evaluation sets and audit snapshots. The specific product is less important than maintaining a data chain that is traceable, reproducible, and accountable.

5. Example: Policy Evaluation Pseudocode

The following platform-independent pseudocode shows how hard constraints, uncertainty thresholds, and human review can work together:

function evaluate(context):
    constraints = [
        context.authorization_verified == true,
        context.sensitive_data_minimized == true,
        context.minor_protection_required == false,
        context.risk_level <= allowed_risk
    ]
    if any constraint is false:
        return BLOCK(reason=constraint_violation, audit=true)
    scores = multi_objective_score(context)
    // scores include safety, fairness, privacy, utility, and confidence intervals
    if uncertainty(scores) > review_threshold:
        return HUMAN_REVIEW(reason=high_uncertainty, audit=true)
    if scores.safety.lower_bound < safety_floor:
        return BLOCK(reason=safety_floor, audit=true)
    if scores.risk_level == medium:
        return HUMAN_REVIEW(reason=medium_risk_sampling, audit=true)
    return ALLOW(scores=scores, policy_version=current_version, audit=true)

The key point is not the complexity of the code, but the order of operations:

1. Check non-negotiable constraints.
2. Evaluate multiple objectives.
3. Escalate highly uncertain cases to human reviewers.
4. Allow automated execution only after these checks pass.

6. Organizational Processes for Sustainable Ethics Engineering

Technical mechanisms need to be supported by organizational processes. At a minimum, organizations should establish four practices:

• Perform ethics impact assessments early: Identify affected groups and high-risk scenarios during the requirements phase.
• Use release gates: Make evaluation-set regression tests, red-team results, and monitoring configuration conditions for deployment.
• Conduct regular reviews: Update policies in response to model drift, complaints, incidents, and post-incident analysis.
• Run cross-functional reviews: Include product, engineering, legal, security, and business stakeholders in high-risk decisions.

An effective-altruism-style approach emphasizes using evidence and scale to evaluate the impact of an action. This offers a useful perspective for engineering evaluation: measure outcomes where possible, compare alternatives, and revise decisions continuously.

However, engineering must also recognize the limits of this approach. Values that are not commensurable should not be forcibly converted into a common scale. Quantitative results cannot replace democratic deliberation or legal compliance. Model uncertainty must be made explicit and incorporated into decision-making.

7. Conclusion: Quantification Is a Tool, Not an Answer

The thought experiment of asking how many chickens are equivalent to one human life shows both the appeal and the danger of reducing values to numbers.

For AI systems, the right goal is not to design a universal ethics score. It is to build decision systems that are constrained, explainable, reviewable, and auditable.

Hard constraints protect fundamental boundaries. Multi-objective evaluation preserves complexity. Explicit uncertainty triggers human review. Logging and version management make decisions traceable.

Quantification can improve consistency, but ethical responsibility ultimately remains with people, institutions, and the processes through which decisions are made.