06 · Ethics

This chapter sits in Part I, the Court of Philosophy — the sixth hearing, where philosophy asks the machine whether a decision keeps its moral weight once a system makes it.

Can a decision be delegated without losing its morality?

On a spring afternoon in 2014, eighteen-year-old Brisha Borden and a friend grabbed an unlocked kid's bike and scooter in a Fort Lauderdale suburb, rode them half a block, and were arrested for burglary and petty theft — total value, $80. The previous summer, forty-one-year-old Vernon Prater had been picked up for shoplifting $86.35 of tools from a Home Depot; he had already been convicted of two armed robberies and an attempted armed robbery. Both were scored by COMPAS, a proprietary risk-assessment tool used across Broward County, Florida, to predict the likelihood of reoffending. Borden, Black and a first-time adult offender, scored 8 out of 10 — high risk. Prater, white and a repeat violent felon, scored 3 — low risk. Two years later, ProPublica's investigation found the algorithm had gotten it backward: Borden was not charged with any further crime; Prater was soon incarcerated again (Angwin et al., "Machine Bias," ProPublica, 23 May 2016). Across more than 7,000 Broward County cases, Black defendants who did not reoffend were misclassified as high-risk at roughly twice the rate of white defendants who did not reoffend.

Northpointe, COMPAS's maker, did not deny the numbers — it denied that they proved bias. Its scores were calibrated: among defendants who scored, say, 7, roughly the same share reoffended whether they were Black or white. ProPublica had measured a different quantity — the error rates within each racial group — and by that measure the tool was plainly unequal. Both sides were right about their own arithmetic, and this is where the story stops being a scandal about one company's software and becomes a theorem. Jon Kleinberg, Sendhil Mullainathan, and Manish Raghavan, and independently Alexandra Chouldechova, proved in 2016–17 that when the base rate of the outcome differs between two groups — as recidivism rates did between Black and white defendants in this data — no risk score can generally satisfy both calibration and equal error rates at once, except in narrow special cases (Kleinberg, Mullainathan & Raghavan, "Inherent Trade-Offs in the Fair Determination of Risk Scores," arXiv:1609.05807; Chouldechova, "Fair Prediction with Disparate Impact," Big Data 5(2), 2017). COMPAS and ProPublica were not arguing about facts. They were standing on opposite sides of an impossibility result and each calling the other's axiom the wrong one — which means "make the algorithm fair" is not an engineering instruction until someone first answers a question philosophy has never closed: fair by which of several mutually exclusive criteria, and who gets to choose.

Delegating the decision does not just relocate the fairness question — it can dissolve the chain of responsibility that used to answer for it. A system that adapts from data after deployment can produce an output its own engineers cannot predict or fully reconstruct, which breaks the ordinary assumption that a designer's intent traces forward into a machine's behavior. Andreas Matthias named this the responsibility gap: when a learning automaton's actions no longer trace predictably to any human's decision, neither the manufacturer nor the operator can be justly held liable by the old rules, and yet someone has been harmed (Matthias, "The Responsibility Gap," Ethics and Information Technology 6(3), 2004). Helen Nissenbaum had already catalogued the mechanics of this a decade earlier — "many hands" spreading a single harm across enough contributors that no one holds the whole of it, bugs treated as acts of nature, blame displaced onto "the computer," and software shipped under licenses that disclaim liability by default (Nissenbaum, "Computing and Accountability," CACM 37(1), 1994). A gap in responsibility is not an absence of harm. It is harm with the addressee missing.

If responsibility can slip away from a system, philosophy has to ask what a system is, morally — object, agent, or something newly in between. Luciano Floridi and Jeff Sanders argued that "moral agent" need not require free will, consciousness, or intention at all: an entity that is interactive (responds to stimuli by changing state), autonomous (can change state without external stimulus), and adaptable (can change the very rules by which it changes state) qualifies as a moral agent at the relevant level of abstraction, whatever else it lacks (Floridi & Sanders, "On the Morality of Artificial Agents," Minds and Machines 14(3), 2004). That loosens moral agency enough to fit a recommendation engine or a trading bot without claiming it has an inner life. It leaves untouched, and does not answer, the harder question now being asked of the systems this book keeps discussing: whether something interactive and adaptive enough could also become a moral patient — able to be wronged, not just to act — a question no consensus method yet resolves.

Alignment research turns out to be metaethics wearing an engineer's badge. Iason Gabriel's 2020 analysis distinguishes several things "align an AI with human values" could mean — instructions, revealed preferences, idealized preferences, interests, or values — and argues the central difficulty is not discovering the one true set of moral principles, but designing principles fair enough to win reflective endorsement across people who deeply and permanently disagree about ethics (Gabriel, "Artificial Intelligence, Values, and Alignment," Minds and Machines 30, 2020). That is exactly the problem political philosophers call the problem of legitimacy under pluralism — alignment did not invent a new question; it gave an old one a training loop and a deadline.

Privacy submits to the same delegation problem in a quieter register. Helen Nissenbaum's contextual integrity, developed across the 1990s and set out fully in Privacy in Context (2010), argues that privacy is neither secrecy nor control over information but the expectation that information flows according to the norms proper to the context it came from: a diagnosis flowing from doctor to insurer for billing is a different flow, governed by different norms, than the same fact flowing from doctor to employer (Nissenbaum, Privacy in Context, Stanford UP, 2010). Most data-collection systems are context-blind by construction — a field in a database carries no memory of which norms it was gathered under — which is why so much that feels like a privacy violation involves no secret at all, only information moved into the wrong context.

Google convened an eight-member external AI ethics council, ATEAC, on 26 March 2019 and dissolved it entirely nine days later, after employee protest over two of its members made the board's continued existence untenable (MIT Technology Review, "Google Has Now Cancelled Its AI Ethics Board," 5 April 2019). An advisory board's authority depends entirely on the goodwill of the institution that convened it and can be withdrawn as easily as it was granted. Mechanism design does not ask for goodwill: a fairness constraint written into a training objective, or a differential-privacy bound written into a query system, keeps operating on the day nobody is in the room to enforce it — which is a narrower kind of ethics than a board's, but a harder one to dissolve by memo.

Connections

Chapter 02 supplies the deeper question of what kind of thing a "decision" made by software even is. Chapter 07 takes platform power into the register of governance, where the question stops being who is accountable and becomes who is sovereign. Chapter 13 returns to the responsibility gap through interpretability — can you hold a system accountable for a decision no one, including its designers, can fully read back out of it?

Open questions

If calibration and error-rate balance cannot both hold, is choosing between them a technical decision at all, or is every deployed risk score a silent vote on which kind of injustice a society will tolerate? Does Floridi and Sanders's minimal moral agency dissolve responsibility further by making blame diffusible to "the system," or does it at least make the diffusion visible enough to argue about? And if legitimacy under pluralism was always the real problem behind alignment, what does computer science have to offer political philosophy that centuries of contract theory did not already try?

Go deeper

• Cathy O'Neil, Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy, Crown, 2016 — the general-audience case this chapter's theorem underwrites.
• Jon Kleinberg, Sendhil Mullainathan & Manish Raghavan, "Inherent Trade-Offs in the Fair Determination of Risk Scores," arXiv:1609.05807, 2016, read against Alexandra Chouldechova, "Fair Prediction with Disparate Impact," Big Data 5(2), 2017 — the impossibility result from both directions.
• Stanford Encyclopedia of Philosophy, "Ethics of Artificial Intelligence and Robotics" — the standing map of this chapter's whole terrain.