Recourse Is the AI-Ethics Question That Ships

Most writing about AI ethics operates at an altitude where nothing can be checked. It worries about bias, transparency, autonomy, and alignment, all real, all so broad that a company can nod along to every word and change nothing on Monday. I want to come down to the altitude where the harm actually lands, because I build the kind of systems this is about, and I find bugs in other people’s for a living. From down here the picture is narrower and much less comfortable.

Start with what “bias” really means in the wild, because the abstraction hides it. In 2016, ProPublica examined COMPAS, a tool used in American courts to score how likely a defendant was to reoffend. Black defendants who did not go on to reoffend were flagged as high-risk at nearly twice the rate of white defendants who did not. The vendor disputed the framing, and there is a genuine statistical result underneath the fight: you cannot satisfy every reasonable definition of fairness at once when the base rates differ. That is not a bug someone forgot to fix. It is a mathematical fact, proven, and it means “make the model fair” is not a coherent instruction. Somebody has to choose which fairness to sacrifice, and that choice is political, and right now it is being made silently by whoever tunes the threshold.

The pattern repeats wherever you look closely. In 2018, Joy Buolamwini and Timnit Gebru tested commercial face classifiers and found error rates under one percent for light-skinned men and up to a third for dark-skinned women. Same product, same model, wildly different reliability depending on who you are, and nobody in the failing group was asked or warned. “Garbage in, garbage out” is the slogan people reach for, but it is too kind. The data was not garbage. It was a faithful record of a world that already over-samples some faces and under-serves others, and the model did its job, which was to learn that world and keep it.

Transparency is the wrong demand, and it lets everyone off the hook

The standard fix offered here is transparency: make the model explainable. I have come to think this is mostly theatre, and I want to say why plainly.

A modern model finds patterns in data that its own builders cannot fully name. You can generate a story after the fact about which inputs mattered, and that story can be wrong, or worse, technically true and useless. “Your loan was declined because of feature 47” explains nothing to the person holding the letter. Demanding an explanation the system cannot honestly give produces a paragraph of plausible words, and everyone treats the paragraph as accountability. It is not. It is a receipt for a decision no one will own.

Even where explanation is possible, notice what it does. It shifts the burden onto the affected person to understand a system they never chose to be judged by, and it lets the operator say they were transparent while nothing about the decision changed. Transparency, as usually demanded, is a way of looking responsible.

The question that protects someone

Here is the test I apply, and it is deliberately small enough that you cannot wave it away. When a system decides against a person, three things have to be true, or the ethics conversation was decoration.

The person has to be told a decision was made and that a model made it. Not buried in terms of service. Told. A silent automated rejection is the most common and least discussed harm, because the victim never learns there was anything to contest.

The person has to be able to contest it and have that contest actually reviewed, not routed back into the same model that produced the result. An appeal that the machine adjudicates is not an appeal.

And a specific human being has to be accountable for the outcome, someone whose job is on the line, not “the algorithm,” which is the most effective liability shield ever invented. The moment a decision can be blamed on a system, no person has to answer for it, and that is usually the point.

Call this recourse. It says nothing about the model’s internals, and that is the strength of it. A perfectly fair, perfectly transparent model with no recourse still traps people. A crude model with real recourse is survivable, because its mistakes can be caught and reversed by someone who can be held responsible. Recourse is checkable. You can walk into an operation and ask to see the appeal, and either it exists and works or it does not.

Why I trust this frame more than the grand one

I am not a philosopher and I am not going to pretend the alignment of future superintelligence is a solved or even well-posed problem. But I notice that the grandest framings are also the most convenient. They put the hard question far away, in a lab, in the future, in the math, anywhere except the loan desk and the parole hearing and the hiring filter where models are quietly deciding real outcomes today with no one to appeal to.

Bias will never be fully removed, because it lives in the world the data comes from, not only in the code. So stop promising to remove it. Build the thing that assumes you failed: tell people, let them fight back, and put a human name against the outcome. That is less inspiring than a manifesto about the future of humanity. It is also the part you can actually ship, and the part a person on the wrong end of a model would trade every white paper to have.

Sources

• ProPublica’s investigation of the COMPAS risk score: Machine Bias.
• Buolamwini and Gebru, on face-classifier error rates by skin tone and gender: Gender Shades.