Mansori

Overfitting: When a Model Memorizes

About this lesson, for parents and teachers

A model that scores perfectly on its practice data can still fail badly on new data. Fit curves of rising complexity and catch overfitting in the act.

Ages 15–18 · about 8 minutes · builds on Backpropagation: Training a Network Live · leads to Biased Data, Biased AI

Try this

  1. Drag degree from 1 up to 12. The red curve is the least-squares fit to the filled training dots.
  2. Watch the chart: the red train error keeps falling, but the dashed blue test error turns around and shoots up.
  3. Visit at least six degrees and stop on the one with the lowest test error.
  4. Set degree 12 and raise the penalty λ one notch at a time until the wiggles calm down.

How it works

Hidden behind the dots is a smooth curve, 0.8 sin(3x), plus random noise. The model is a polynomial of degree d, which has d + 1 weights. Your browser finds the weights by least squares: it solves the normal equations (AᵀA) w = Aᵀy, which gives the curve with the smallest possible total squared distance to the 14 training points.

We score the fit with RMSE, the root mean squared error. Training RMSE always goes down as you add degrees, because a bigger model can always do at least as well on the data it was fitted to. At degree 13 a polynomial could pass through all 14 points exactly. That is not learning, it is memorizing, and the noise gets memorized too.

The honest test is data the model never saw. The 40 hollow test points come from the same curve with fresh noise. Low degrees underfit: a straight line cannot follow the wave, so both errors are high. High degrees overfit: the curve swings wildly between training points, so test error explodes. The sweet spot is in between, and finding it is called model selection.

Engineers rarely just pick a small model. Instead they use regularization. Ridge regression adds a penalty λ · Σ w² to the loss, which punishes large weights. Large weights are what make wild wiggles, so a little λ lets even a degree 12 model stay smooth. Too much λ squashes everything flat and underfits again.

This trade-off is often called the bias-variance trade-off. Simple models are biased, wrong in the same way every time. Complex models have high variance, changing a lot with every new sample of noise. Press New noisy data at degree 12 and watch how much the curve jumps around. A good model is one whose predictions barely change when you collect a new sample of the same kind of data.

Where engineers use this

Every machine learning team keeps a held-out validation set and stops training or tunes regularization when validation error starts rising. Weight decay, dropout and early stopping in deep learning are all defences against overfitting.

Words to know

Overfitting
When a model fits the training data's noise so closely that it does worse on new data.
Underfitting
When a model is too simple to capture the real pattern, so it does badly on training and test data.
Test set
Data held back from training and used only to measure how well the model generalizes.
Regularization
A penalty on model complexity, such as λ · Σ w², that discourages wild solutions.

Talk about it

Goal: Fit curves of rising complexity and see training error fall while test error climbs.

  • Why does training error always go down when the model gets more complex, but test error does not?
  • A student memorizes last year's exam answers. How is that like overfitting?
  • If you had 1000 training points instead of 14, would degree 12 still overfit so badly? Why?

Without a screen: Draw 10 dots roughly along a gentle hill on graph paper. One person draws a smooth curve through the middle of them; another draws a wiggly line that hits every dot exactly. Now add 5 new dots from the same hill. Measure with a ruler which line is closer to the new dots.

Curriculum links: Common Core HSS-ID.B.6 (fit a function to data and use residuals); AP Statistics (regression and residuals); CSTA K-12 3A-DA-12 (create computational models that represent relationships in data); AI4K12 Big Idea 3 (Learning)

Questions

What is overfitting in machine learning?
Overfitting is when a model learns the training data too well, including its random noise, so it performs worse on new data. You spot it when training error keeps falling while validation or test error rises.

How do you prevent overfitting?
Use more training data, choose a simpler model, add regularization such as ridge or weight decay, or stop training early when validation error stops improving. Always measure on data the model has not seen.

What is the bias-variance tradeoff?
Simple models have high bias: they miss the real pattern. Very flexible models have high variance: they change a lot with each new sample of data. The best model balances the two to get the lowest error on new data.