Every organisation deploying a model has published a set of values. Fairness, transparency, accountability, human oversight. Nobody objects to any of it, which is a reliable sign that the statement is not doing much work.
The useful conversation is one level down, where the trade-offs are real and mutually exclusive. This piece stays at that level.
Bias is a measurement problem before it is a moral one
The word bias covers several distinct things, and conflating them prevents anyone from fixing anything.
Data bias. The training data does not represent the population the system will be used on. A diagnostic model trained largely on one demographic performs worse on others. This is the most common and most tractable form.
Label bias. The thing you measured is not the thing you care about. A hiring model trained on "was this candidate hired" learns previous hiring decisions, including whatever prejudices they contained. The data is accurate; the target is wrong.
Proxy bias. You excluded a protected attribute, but other features encode it. Postcode correlates with ethnicity. Employment gaps correlate with parenthood and disability. Removing the obvious field does not remove the information.
Deployment bias. The model is statistically fine and used badly — applied to a population it was not built for, or by people who over-trust its output.
There are several reasonable definitions of fairness — equal accuracy across groups, equal false positive rates, equal false negative rates, calibrated probabilities within each group — and it has been formally proven that you cannot satisfy all of them at once unless base rates happen to be identical across groups. There is no configuration that is fair by every definition. Somebody has to choose which definition applies and be accountable for that choice. Most organisations avoid making the choice explicitly, which means it gets made implicitly by whoever wrote the code.
That result reframes the whole discussion. The question is not "is this system fair" but "which fairness criterion did you select, why, and who agreed to it".
Choosing a criterion, concretely
The right answer depends on what a mistake does to the person.
For a loan application, a false negative denies someone credit they would have repaid. Equal false negative rates across groups is a defensible target.
For a medical screening tool, a false negative means a missed illness and a false positive means an unnecessary follow-up. These are wildly unequal harms, so equalising overall accuracy is the wrong objective.
For content moderation, over-removal silences people and under-removal exposes them to abuse. The correct balance differs by category of content, and pretending one threshold serves all is how platforms end up in the news.
The pattern: identify who bears the cost of each error type, then equalise the thing whose cost is highest. This is a policy decision that requires people who understand the domain, and it cannot be delegated to the team that trained the model.
Privacy, with the specifics
Data protection law is mostly older than the current wave of systems, and it applies anyway. The requirements that bite hardest:
Purpose limitation. Data collected for one purpose generally cannot be repurposed for training without a fresh legal basis. "We already had the data" is not a basis.
Data minimisation. Collect what you need. Models tend to be built by teams that want everything, and this is directly in tension with the law.
The right to erasure. Genuinely difficult. If someone's data was in a training set, removing it from the database does not remove its influence on the weights. Retraining is expensive. This is an unresolved practical problem, and the honest positions are to retrain periodically, to use techniques that limit individual influence, or to avoid training on personal data in the first place.
Automated decision-making rules. Where a decision is fully automated and has legal or similarly significant effects, individuals typically have rights to information about the logic involved, to contest it, and to request human review. "The model decided" is not a lawful explanation.
Memorisation. Models can reproduce fragments of training data verbatim, particularly unusual strings. If personal data went in, it may come out. This needs testing rather than assuming.
What the current regulation actually asks for
Requirements vary by jurisdiction, but the emerging shape is consistent enough to plan against. The dominant model is risk-tiered.
| Tier | Examples | Typical obligations |
|---|---|---|
| Prohibited | Social scoring, manipulative systems targeting vulnerability, certain biometric surveillance | Not permitted |
| High risk | Employment, credit, education access, essential services, medical devices, law enforcement | Risk management, data governance, documentation, logging, human oversight, accuracy and robustness testing, conformity assessment |
| Limited risk | Chatbots, synthetic media | Disclosure that the user is interacting with a machine or viewing generated content |
| Minimal risk | Spam filters, recommendations, game systems | General law only |
Two practical observations. First, the obligations cluster on documentation and process rather than on model architecture — regulators are asking whether you can demonstrate diligence, not whether you used a particular technique. Second, sector rules often bind harder than general AI rules. If you are in medical devices or financial services, your existing regulator's expectations arrive first and matter more.
A governance process that is not theatre
What separates real governance from a published principles page:
- A register of deployed systems. You cannot govern what nobody has listed. Most organisations do not have this and are surprised by what turns up.
- Documented purpose and prohibited uses, per system. Written before deployment.
- Performance measured by subgroup, not just in aggregate. Aggregate accuracy conceals exactly the failures that matter, and this single practice catches most bias problems.
- A named owner who is accountable for outcomes and is senior enough to stop a launch.
- A route for affected people to contest a decision, reaching a human with authority to overturn it.
- Monitoring for drift. Populations change; a model validated two years ago is not validated now.
- A recorded decision on the fairness criterion, with reasoning. This is the item almost everyone skips, and the one that would be most useful in an audit.
None of that requires unusual expertise. It requires deciding that someone owns it.
Human oversight, and why it usually fails
"A human reviews the output" is the most common control and the weakest one as typically implemented, for well-documented reasons.
Automation bias. People defer to machine recommendations, particularly when the machine is usually right. A reviewer approving the ninetieth confident recommendation of the day is not exercising independent judgement.
No time. If the reviewer has ninety seconds per case and the model took a full case file into account, meaningful review is not possible.
No authority. If overturning the model requires escalation and justification while agreeing requires a click, the incentive is obvious.
No information. A score without the factors behind it cannot be assessed.
Oversight that works has fewer cases, more time, real authority, visible reasoning, and measurement of how often the human actually disagrees. If the override rate is near zero, you do not have oversight — you have a rubber stamp, and you should either fix the process or stop claiming the control exists.
Where I think the real gap is
Not in principles, which are broadly agreed, and not in the law, which is arriving. It is in the ordinary organisational habit of measuring what a system does to different groups of people, writing down what you chose and why, and giving somebody the authority to say no.
That is unglamorous work. It is also nearly all of the difference between a system that treats people reasonably and one that does not, and it has very little to do with the model.