Biased Data, Biased AI
About this lesson, for parents and teachers
A model can score 96% and still fail one group of people again and again. Train one yourself, audit it, and repair it.
Ages 15–18 · about 9 minutes · builds on Overfitting: When a Model Memorizes
Try this
- Press Train. The overall accuracy looks excellent. Would you ship it?
- Press Audit by group. Compare the error bars for group A (circles) and group B (triangles).
- Drag the group B share slider up, which means collecting more group B examples, and retrain. Watch B's error fall.
- Find a training mix where both groups' errors are within 2 points of each other.
How it works
This classifier is k-nearest neighbours: to label a new point, it finds the 7 closest training examples and takes a vote. Groups A and B follow exactly the same true rule, the curved line between yes and no. The only difference is where they sit in feature space. Group A's examples are mostly on the left and group B's on the right, the way different groups of people can have different typical measurements.
In the data as collected, only 5% of the 200 training examples come from group B. Around group B there are too few examples to trace the curve, so votes are often decided by far-away points and the model guesses wrong. This is representation bias: the algorithm is not prejudiced, but the data under-represents one group, so the errors land on that group.
Why didn't the overall accuracy warn you? Because it is an average. If 95% of test cases are from group A, a model can fail a quarter of group B and still score about 96% overall. That is why responsible teams do a disaggregated evaluation: they measure error separately for every group the system affects, and look at the gap between them, not just the headline number.
Here the fix is better data: with a balanced training set, both groups get low, similar error rates and the overall score hardly changes. Real life is often messier. Labels can carry human bias, features can act as hidden stand-ins for group membership, and different fairness definitions can conflict. But auditing by group and fixing the data is where good practice starts.
Where engineers use this
A 2018 study of commercial face analysis systems found error rates were far higher for darker-skinned women than for lighter-skinned men, largely because of unbalanced training and test data. Companies now publish model cards that report accuracy per group.
Words to know
- Bias (in AI)
- A systematic pattern where a model performs worse for, or treats unfairly, some groups.
- Representation bias
- When some groups are under-represented in training data, so the model learns them poorly.
- k-nearest neighbours
- A classifier that labels a point by a vote of the k closest training examples.
- Disaggregated evaluation
- Measuring a model's errors separately for each group instead of only overall.
Talk about it
Goal: Train a model on lopsided data, see who it fails, and fix it by fixing the data.
- Why can a model with 96% accuracy still be unfair? Who should decide what error gap is acceptable?
- Can you think of an app or device that might work worse for some people because of who was in its training data?
- Collecting more data costs time and money. Who should pay for making AI work well for everyone?
Without a screen: Draw a big map with two neighbourhoods. Put 20 sticky notes marked yes or no in one neighbourhood but only 2 in the other. Ask someone to guess labels for new spots by looking at the 3 nearest notes. Count mistakes in each neighbourhood, then add 10 more notes to the sparse one and try again.
Curriculum links: CSTA K-12 3A-IC-24 (evaluate the ways computing impacts personal, ethical, social, economic and cultural practices); AI4K12 Big Idea 5 (Societal Impact); AP CSP Impact of Computing
Questions
What is AI bias?
AI bias is when a machine learning system makes more mistakes for some groups, or treats them unfairly. It often comes from training data that under-represents those groups or reflects unfair past decisions.
How do you fix bias in AI?
Start by measuring errors separately for each group. Then improve the data by collecting more examples from under-represented groups, check labels for human bias, and re-test. Sometimes the model or how it is used must change too.
Is a more accurate model always fairer?
No. Overall accuracy is an average, so it can hide large errors for a small group. A model can become more accurate overall while one group gets worse, which is why per-group audits matter.