Chapter 6 — Where the Machine Has Been Wrong

This chapter pays for the last two.

The pattern that follows is the most useful thing in the book, and it is worth stating before the evidence rather than after: in every documented case below, the software did approximately what it was built to do. It returned a candidate. The harm came from a human treating the candidate as an answer. Not one of these is a story about a computer being stupid.

Prediction

Geolitica, the renamed successor to PredPol, sold place-based crime forecasting to police departments. In 2023 the journalists at The Markup audited roughly 23,600 predictions made for Plainfield, New Jersey, against crimes subsequently reported.

Fewer than 100 matched — a success rate below 0.5%. Arrest rates in predicted areas were unchanged whether or not a prediction had fired that day. [ESTABLISHED] (The Markup; methodology)

Earlier, RAND ran one of the first controlled tests of place-based prediction in Shreveport, Louisiana, and found no meaningful reduction in property crime against matched control areas. [ESTABLISHED] (RAND, via R Street)

The Los Angeles Police Department used PredPol from around 2011. Its own Office of Inspector General concluded there was insufficient data to show the tool reduced crime — after eight years, the department could not measure its effect. LAPD dropped it in 2020. [ESTABLISHED] (MuckRock; GovTech)

Eight years of operational use without a measurement. Remember Chapter 2's Move: ask for the denominator first. This is what its absence buys.

Prediction about people

Chicago's Strategic Subject List scored individuals for their risk of involvement in gun violence. RAND, contracted to evaluate the 2013–14 cohort, found that people on the list were neither more nor less likely to become a shooting victim than a matched control group. RAND also found the programme lacked clear guidance on what an officer should actually do with a name once it appeared — the instruction amounted to "take action." The department contested the finding. The programme was decommissioned as grant funding ended and was confirmed dead in January 2020. [ESTABLISHED] (RAND; Chicago OIG)

LAPD's Operation LASER, built on Palantir Gotham, targeted "chronic offenders." The department's Inspector General audited it in 2019 and found that almost half of the people on chronic-offender lists had zero or one violent-crime arrest, that around 10% had no recorded quality interactions with police at all, and that training was informal and varied between divisions. LASER was shut down that year. [ESTABLISHED] (Courthouse News)

And in New Orleans, a Palantir deployment ran from 2012 to 2018 without the city council or the public being told, structured as a philanthropic gift specifically to avoid public procurement review. No efficacy evaluation was ever published. [ESTABLISHED] (The Lens; ACLU)

Across all five: no independent evaluation found any of these tools working as advertised. In three of the five, the finding directly preceded discontinuation. That is not a selection of failures — it is what the independent evaluation record for predictive policing contains.

Identification

Four wrongful arrests, all in the United States, all documented in court records or settlements.

Robert Williams, Detroit, January 2020. Michigan State Police ran security-camera stills through facial recognition; the system flagged Williams's expired driver's licence photo. Police built on that with a photo lineup and nothing else. He was arrested in front of his family and held thirty hours. In June 2024 Detroit settled, adopting what the ACLU called the strongest police facial recognition policy in the country: no arrest based solely on a match or on a lineup that followed one, plus an audit of every facial-recognition-derived arrest warrant since 2017. [ESTABLISHED] (ACLU)

Porcha Woodruff, Detroit, February 2023. A facial recognition match plus a victim misidentification, no other corroboration, produced a carjacking warrant. She was arrested eight months pregnant, held eleven hours, experienced contractions in custody. Charges were dropped a month later. Her own civil suit against the city was subsequently dismissed — the wrongful-arrest facts are not in dispute, but her personal remedy failed. [ESTABLISHED] (Washington Post; CBS Detroit)

Nijeer Parks, New Jersey, January 2019. A shoplifter presented a fake ID bearing Parks's photo. Parks had never held a driver's licence and lived thirty miles away with a solid alibi. Ten days in jail; case dismissed for lack of evidence. [ESTABLISHED] (NJ Monitor)

Randal Reid, arrested near Atlanta in November 2022 on a Louisiana warrant. A detective relied solely on a Clearview AI match to a stolen-credit-card purchase. Reid had never been to Louisiana. The use of facial recognition was omitted from the arrest warrant affidavit. Six days in jail. Jefferson Parish settled the federal civil-rights suit for $200,000 in May 2025. [ESTABLISHED] (Biometric Update)

One failure pattern, four times: an algorithmic match treated as sufficient grounds for a warrant without independent corroboration. In the Reid case the tool's involvement was actively concealed from the court. The ACLU has counted more than a dozen known wrongful facial-recognition arrests in the US; every publicly identified subject to date has been Black. [ESTABLISHED] (ACLU)

That last sentence has a measurable substrate. NIST has tested around 200 algorithms from about 100 developers across more than 18 million images. False-positive rates differ across demographic groups by up to a factor of roughly 7,203 — and they do so even on high-quality images, so this is not an artefact of bad photographs. The highest one-to-many false-positive rates were for African American women; rates were lowest for people of Eastern European descent; false positives are also elevated for women, children and the elderly. False-negative differentials are far smaller, around a factor of three. [ESTABLISHED] (NISTIR 8280; NISTIR 8429)

NIST's own caveat is the one that matters most here: it measures algorithms in isolation. It does not measure the police workflow around them — the human review, the corroboration practice — which is precisely where all four cases actually broke.

Fabrication

Chapter 4 told the West Midlands story. It belongs in this chapter's ledger too, as the only clean, named, dated case of a general-purpose language model producing a wrongful law-enforcement decision outside biometrics: a fabricated football fixture entering an intelligence assessment, a ban imposed, a chief constable's resignation, and twenty-one forces continuing to use the tool. [ESTABLISHED]

The gap I could not fill

I searched for a documented case of an AI system producing a wrongful outcome in fraud or financial crime specifically — a named person wrongly arrested, charged, or frozen out of their money on an algorithm's say-so. I did not find one comparable in specificity to the facial-recognition cases. [ABSENT]

This is not a clean bill of health and must not be read as one. False positives in AI transaction monitoring are extremely common by design — an entire industry literature exists on "false positive fatigue." The difference is that the harmed party is usually a delayed or blocked transaction rather than a person in a cell, which generates no court record, no settlement, and no journalism. The absence of documentation is a fact about what gets documented.

Anyone in Thailand building on the Central Fraud Registry described in Chapter 5 should read that paragraph twice. A system that suspends a person's accounts at every bank simultaneously, on a flag, is a system with exactly this harm profile and exactly this documentation deficit.

What the regulators did

The response so far, for the record:

[ESTABLISHED] (EU AI Act Article 5; NPCC AI Strategy; Stateline)

And the independent verdict on the most-cited safeguard: Northumbria University research in 2026 found that the UK Covenant's "human in the loop" principle often exists in name only, providing false assurance rather than a genuine check, and that England and Wales lack any central independent body to coordinate AI standards, oversight and procurement across 43 forces. [ESTABLISHED] (Northumbria University)

"A human reviewed it" is the sentence every one of these deployments could truthfully say. A human reviewed Robert Williams's photo lineup. A human signed Randal Reid's affidavit. A human put the invented football match into the intelligence document. Human oversight is not a safeguard unless the human is structurally able to disagree — which means seeing the underlying material, knowing the tool's error rate, and having time.

The asymmetry, named

Hold this chapter next to the successes and notice the difference in grades.

Failures here are [ESTABLISHED]: inspector-general audits, RAND evaluations, court settlements, NIST measurements, a journalist's audit of 23,600 predictions. Successes are mostly [CLAIMED]: vendor blog posts and force self-reports.

That asymmetry does not mean the technology fails more than it works. It means failure produces adversaries — a person who was jailed, a civil liberties organisation, an auditor — and adversaries produce documents. Success produces marketing. If you weigh the two categories by the quality of their evidence, you will systematically underrate the technology. If you weigh them by their volume, you will systematically overrate it. There is no honest way to avoid holding both facts at once.

The Move — Write the corroboration before you act on the match. When a system returns a candidate — a face, a cluster, a name — record, before taking any action, what independent evidence would have to exist for the candidate to be right, and then go and find out whether it does. Not evidence consistent with the match: evidence obtained without reference to it. Every case in this chapter is a case where that step was skipped, and in one of them the tool's involvement was hidden from the court so the step could not be demanded by anyone else.

What would change my mind: if an independent evaluation showed a deployed predictive or identification system reducing crime or improving accuracy against a proper control, the balance of this chapter would shift and I would say so. As of August 2026 the independent evaluation record contains no such finding, and the pattern of five programmes and four arrests is consistent enough that I would want a strong study, not a promising one.