Mansori

Gradient Descent: Rolling Downhill to Learn

About this lesson, for parents and teachers

How does an AI get better? It measures how wrong it is, then takes a small step in the direction that makes it less wrong. Over and over. Be the one who picks the step size.

Ages 12–14 · about 9 minutes · builds on A Perceptron Learns to Draw a Line · leads to Backpropagation: Training a Network Live

Try this

  1. Press Step with the tiny starting learning rate. Is the ball getting to the bottom fast?
  2. Raise the learning rate and reach the bottom in 10 steps or fewer. Drag the ball first to choose a different start.
  3. Find a learning rate that zigzags across the valley but still settles.
  4. Push the learning rate above 1 and press Run. What happens to the loss?

How it works

Machine learning models have numbers called weights. A loss function measures how wrong the model is for a given weight. Here the loss is (w - 2)², so the best weight is 2, where the loss is 0. Real models have millions of weights and a loss landscape we cannot see, but the idea is the same: find the lowest point.

The gradient is the slope of the loss at the current weight. For our curve it is 2 * (w - 2). A positive slope means the loss goes up to the right, so we should move left, and vice versa. That is why the update rule subtracts: w = w - lr * grad. You can see the slope as the purple tangent line touching the ball.

The learning rate lr sets how big each step is. Too small and the ball crawls, needing hundreds of steps. Just right and it slides in quickly. A bit too big and it overshoots, jumping past the bottom and zigzagging back. For this curve each step multiplies the distance to the bottom by (1 - 2·lr).

When lr is bigger than 1, that multiplier is bigger than 1 in size, so every step lands further away than the last. The loss grows without limit: the training diverges. Engineers training real neural networks watch the loss curve for exactly this and often shrink the learning rate over time, taking big steps early and careful ones near the end.

Where engineers use this

Every large AI model, including image generators and chatbots, is trained with gradient descent or close relatives like Adam. Picking the learning rate is one of the first things engineers tune.

Words to know

Loss function
A formula that measures how wrong a model is. Training tries to make it as small as possible.
Gradient
The slope of the loss. It points uphill, so learning steps go the opposite way.
Learning rate
The size of each step. Too small is slow, too big overshoots or diverges.
Divergence
When training steps make the loss grow instead of shrink, until the numbers explode.

Talk about it

Goal: Tune the learning rate and watch a ball roll down an error curve to the best answer.

  • If you were blindfolded on a hill and wanted to reach the bottom, how would you decide which way to step?
  • Why might a learning rate that works well at the start be too big near the end?
  • What learning rate made the ball land exactly on the bottom in one step? Why?

Without a screen: Play 'hot and cold' with numbers: one person secretly picks a number from 0 to 100, the other guesses, and the first replies with how far off the guess is. Try moving by half the distance each time, then by double the distance, and compare.

Curriculum links: CSTA K-12 3B-AP-08 (artificial intelligence algorithms), 3A-DA-12 (computational models); AI4K12 Big Idea 3 (Learning); Common Core HSF-IF (interpreting functions) and 8.F (rate of change)

Questions

What is gradient descent in simple words?
Gradient descent is a way to find the lowest point of an error curve by repeatedly checking the slope and taking a small step downhill. Machine learning uses it to adjust weights so the model makes fewer mistakes.

What does the learning rate do?
The learning rate decides how big each downhill step is. A small rate is safe but slow, a larger one is faster but can overshoot, and one that is too large makes the error grow out of control.

Why does a large learning rate cause divergence?
Each step jumps past the bottom and lands further up the other side than where it started. Repeating that makes the distance, and the error, grow with every step.