Attention: How Transformers Focus
About this lesson, for parents and teachers
When you read "it was tired", you know "it" means the cat, not the mat. Transformers work that out with a calculation called attention. Click a word and watch where it looks.
Ages 15–18 · about 10 minutes · builds on Words as Points: Embeddings
Try this
- Tap the word it. The bars under each word, and the highlighted row of the heatmap, show how its attention is shared out.
- With it selected, drag
q·animaldown andq·thingup until most of its attention moves to mat. - Drag sharpness to 0. Every word now gets an equal share. Push it up and attention snaps onto one word.
- Turn on the GPT mask and tap the first word. Why can it only see itself?
How it works
In a transformer, each token's vector is used to make three new vectors. The query says what this word is looking for, the key says what this word offers, and the value is the information it will hand over. In a real model these come from multiplying the token's embedding by three learned weight matrices. In this lesson we hand-made 4-number queries and keys with readable meanings: animal, thing, action and grammar.
To decide how much word i should attend to word j, the model takes the dot product of the query and key, q·k, which is large when they point the same way. It divides by √d, here √4 = 2, so the scores do not get huge as vectors get longer. Then a softmax along each row turns the scores into weights that add up to 100%. That full table is the attention heatmap.
The output for word i is the weighted average of every word's value vector, Σ weight × value. So after attention, "it" carries a lot of the cat's information: the model has worked out what "it" refers to. The whole operation is written softmax(QKᵀ/√d)V and it runs for every word at once as a few matrix multiplications, which is why GPUs handle it so well.
Models that generate text, such as GPT and Claude, use a causal mask: a word may only attend to itself and earlier words, because while writing, the future words do not exist yet. Real models also run many attention heads in parallel, each with its own learned queries and keys, stacked in dozens of layers. Their dimensions are not labelled like ours; researchers study them to work out what each head has learned.
Where engineers use this
Attention is the core of the transformer architecture from the 2017 paper "Attention Is All You Need", which underlies ChatGPT, Claude, Gemini, machine translation and many image and protein-structure models.
Words to know
- Query, key, value
- Three vectors made from each token: what it looks for, what it offers, and what it passes on.
- Self-attention
- Each token in a sequence computing weighted averages over all tokens in the same sequence.
- Causal mask
- A rule that blocks a token from attending to tokens after it, used in text-generating models.
- Attention head
- One independent set of query, key and value projections. Transformers run many heads in parallel.
Talk about it
Goal: Click a word and see which other words it pays attention to, the core trick inside ChatGPT and Claude.
- In "the trophy did not fit in the suitcase because it was too big", what does "it" mean? What about "too small"?
- Why might a model need many attention heads instead of one?
- Attention compares every word with every other word. What happens to the amount of work when the text gets ten times longer?
Without a screen: Write a sentence on strips of paper, one word each. For every word, give each other word a score from 0 to 3 for how much it helps explain it. Turn each row of scores into fractions that add up to 1. You have built an attention table by hand.
Curriculum links: CSTA K-12 3B-AP-08 (describe how artificial intelligence drives software and physical systems); AI4K12 Big Idea 2 (Representation and Reasoning); Common Core HSN-VM (vector and matrix quantities)
Questions
What is the attention mechanism in simple terms?
Attention lets each word in a sentence look at the other words and decide which ones matter for understanding it. It scores every pair with a dot product, turns the scores into percentages with a softmax, and mixes in information from the highest-scoring words.
Why divide by the square root of d in attention?
Dot products of long vectors tend to be large, which pushes the softmax to give nearly all the weight to one word and makes training unstable. Dividing by the square root of the vector length keeps the scores in a comfortable range.
How do transformers work?
A transformer turns tokens into vectors, then passes them through many layers. Each layer has attention, where tokens share information, followed by a small neural network applied to each token. The final layer predicts the next token.