Inside a Neural Network
About this lesson, for parents and teachers
A neural network is not magic. It is multiplication, addition and one simple squashing function, repeated. Tap any neuron and read its maths as the signal flows through.
Ages 15–18 · about 10 minutes · builds on A Perceptron Learns to Draw a Line · leads to Words as Points: Embeddings
Try this
- Drag the x1 and x2 sliders and watch the dots flow faster along strong connections. Push the output above 0.80.
- Tap neuron h3, then lower its bias until its weighted sum goes negative. ReLU turns it off to exactly 0.
- Tap Output and make
v1negative. The same inputs now give the opposite answer. - Red lines are positive weights, blue lines are negative. Thicker means a bigger weight.
How it works
Each neuron does two things. First it computes a weighted sum: multiply every input by its weight, add them up, then add a bias. For hidden neuron h1 that is z = w1·x1 + w2·x2 + b. A positive weight means "this input pushes me up", a negative weight means "this input pushes me down", and the bias sets how easily the neuron switches on.
Second, the neuron passes z through an activation function. Our hidden layer uses ReLU, which is simply max(0, z): negative sums become exactly zero. The output neuron uses the sigmoid, 1 / (1 + e^-z), which squashes any number into the range 0 to 1 so it can be read as a probability or a confidence.
Without activation functions, stacking layers would be pointless: a sum of weighted sums is still one weighted sum, so the whole network would collapse into a single straight-line model. The bend in ReLU is what lets each layer carve the input space into regions, and lets deeper networks represent curved, complicated decision boundaries.
This network has 2 inputs, 3 hidden neurons and 1 output, which means 9 weights and 4 biases, so 13 parameters. Running inputs through it from left to right is called a forward pass. A large language model does exactly the same kind of forward pass, but with billions of parameters, and its layers are mostly huge matrix multiplications that a GPU computes in parallel.
Here you set the weights by hand, one slider at a time. In real machine learning nobody does that: a training algorithm called backpropagation measures how wrong the output is, works out how much each weight contributed to that error, and nudges every weight a tiny step in the direction that reduces it. Repeat that over thousands of examples and good weights appear. That is the next lesson.
Where engineers use this
Every image classifier, speech recognizer and chatbot runs forward passes like this one. Engineers measure models by parameter count, and frameworks such as PyTorch store each layer's weights as a matrix so a whole layer is computed in one multiplication.
Words to know
- Weight
- A number on a connection that scales the signal passing through it. Positive weights excite, negative weights inhibit.
- Bias
- A number added to a neuron's weighted sum that shifts how easily it activates.
- Activation function
- The non-linear function applied to a neuron's sum, such as ReLU
max(0, z)or sigmoid. - Forward pass
- Computing a network's output by sending inputs through every layer in order.
Talk about it
Goal: Change inputs and weights and watch signals flow through layers of neurons to an output.
- Why would a network with no activation functions be no better than a single neuron?
- If you double every weight in the output neuron, what happens to how confident the output is?
- A model has 7 billion parameters. What does each one actually do, based on this lesson?
Without a screen: Play a human neural network: three people are hidden neurons, each with a card listing two weights and a bias. Call out two input numbers, each person computes their weighted sum with a calculator and says 0 if it is negative. A fourth person combines the three answers with their own weights and decides yes or no.
Curriculum links: CSTA K-12 3B-AP-08 (describe how artificial intelligence drives software and physical systems); AI4K12 Big Idea 3 (Learning); Common Core HSF-IF (interpreting functions) and HSA-SSE (seeing structure in expressions)
Questions
How does a neural network work in simple terms?
Each neuron multiplies its inputs by weights, adds them up with a bias, and passes the result through an activation function. Layers of these neurons feed into each other, and the final layer gives the answer.
What is ReLU in a neural network?
ReLU stands for rectified linear unit. It outputs the input if it is positive and 0 otherwise, written max(0, z). It is cheap to compute and gives the network the non-linearity it needs.
What is the difference between weights and biases?
A weight scales one particular input, so it says how much that input matters and in which direction. The bias is added once per neuron and shifts the point at which the neuron starts to fire.