Mansori

Backpropagation: Training a Network Live

About this lesson, for parents and teachers

XOR stumped early AI researchers for years: no single straight line can split its four points. Train a small network yourself and watch it learn to bend the line.

Ages 15–18 · about 10 minutes · builds on Gradient Descent: Rolling Downhill to Learn · leads to Overfitting: When a Model Memorizes

Try this

  1. Press Train and watch the heatmap: red means the network says 1, blue means 0. The black line is where it is exactly 50/50.
  2. Watch the loss curve fall below the dashed "guessing" line, then below 0.05.
  3. Switch to 1 neuron and train for 500 epochs. The loss gets stuck. Why?
  4. Try 2 neurons. If training stalls on a plateau, press Re-init and train again.

How it works

The network has two inputs, a hidden layer of tanh neurons and one sigmoid output. We measure how wrong it is with the cross-entropy loss, L = -[y log(p) + (1-y) log(1-p)], averaged over the four XOR examples. Random guessing scores ln 2 ≈ 0.693, which is the dashed line on the chart, so anything below it means the network has learned something.

Backpropagation works out how the loss changes when each weight changes, the gradient ∂L/∂w. It starts at the output, where for a sigmoid with cross-entropy the error signal is simply p - y, then uses the chain rule to pass that signal backwards: through each output weight, through the derivative of tanh, which is 1 - h², and on to the first-layer weights.

With every gradient in hand, gradient descent moves each weight a small step downhill: w ← w - η · ∂L/∂w. The learning rate η sets the step size. Too small and training crawls; too big and it can overshoot. One pass over all the training data is called an epoch. This lesson really runs that loop in your browser, every frame.

Why does one hidden neuron fail? A single tanh neuron followed by a sigmoid can only split the plane with one straight line, and XOR's two red corners sit diagonally opposite each other. Two hidden neurons give two lines, and the output neuron combines them into a band. That is the deep idea behind layers: each layer builds new features that make the next layer's job easier.

Training is not guaranteed to succeed. Starting from unlucky random weights, the loss can sit on a flat plateau or fall into a poor solution, especially in tiny networks. Bigger networks, better initialisation and optimisers like Adam make this much rarer, which is part of why modern models are so large.

Where engineers use this

Every modern neural network, from phone face unlock to large language models, is trained with backpropagation and a variant of gradient descent. Libraries like PyTorch and JAX compute the gradients automatically, a technique called automatic differentiation.

Words to know

Loss
A single number measuring how wrong the model is on the training data. Training tries to make it small.
Gradient
The slope of the loss with respect to a weight, ∂L/∂w. It says which way, and how strongly, to change that weight.
Learning rate
The step size η used in each gradient descent update.
Epoch
One complete pass through all the training examples.

Talk about it

Goal: Train a small network on XOR in your browser and watch the decision boundary bend.

  • Why can't a single straight line separate the XOR points? Try sketching it.
  • What happened when you made the learning rate very large? Why might that help or hurt?
  • The same network sometimes solves XOR and sometimes gets stuck. What does that say about training big AI models?

Without a screen: Play blindfold hill descent: on a staircase or a sloped garden, a blindfolded player feels the ground with one foot and always steps a fixed distance in the downhill direction. Try big steps and small steps. Big steps get down fast but can overshoot a dip; tiny steps are safe but slow. That is the learning rate.

Curriculum links: CSTA K-12 3B-AP-08 (describe how artificial intelligence drives software and physical systems); AI4K12 Big Idea 3 (Learning); AP Calculus chain rule and derivatives; Common Core HSF-BF (building functions)

Questions

What is backpropagation in simple terms?
Backpropagation is the method for working out how much each weight in a neural network contributed to the error. It applies the chain rule from calculus, starting at the output and moving backwards layer by layer, so every weight gets its own gradient.

Why can't a single perceptron learn XOR?
A single neuron draws one straight boundary, but XOR's positive examples sit on opposite corners, so no line separates them. Adding a hidden layer lets the network combine two boundaries, which solves it.

What is a good learning rate?
It depends on the model and data. Common starting points for real models are around 0.001 with the Adam optimiser, while this tiny network trains well around 0.3 to 1. Engineers usually try several values and watch the loss curve.