Mansori

Words as Points: Embeddings

About this lesson, for parents and teachers

To a language model, every word is a list of numbers, a point in space. Tap words on this map, then add and subtract them like vectors and see where you land.

Ages 15–18 · about 9 minutes · builds on Inside a Neural Network · leads to Attention: How Transformers Focus

Try this

  1. Tap any word. Dashed lines join it to its three nearest neighbours, with their distances.
  2. Press Word maths, then tap king, man and woman. The red arrow is the vector from man to king, moved so it starts at woman.
  3. Build your own: Paris - France + Japan, kitten - cat + dog or walked - walk + swim.
  4. Try an analogy that mixes groups, like pizza - apple + king. Where does it land?

How it works

Computers cannot do maths on letters, so language models turn each word (more precisely each token) into a vector: a list of numbers called an embedding. The model learns these numbers during training so that words used in similar contexts end up with similar vectors. "Cat" and "dog" appear in similar sentences, so their points sit close together.

Closeness is measured with a distance or similarity. On this map we use ordinary Euclidean distance, √((u₁-u₂)² + (v₁-v₂)²). Real systems usually use cosine similarity, which compares the directions of two vectors and ignores their length. Finding the closest vectors to a query is called nearest-neighbour search, and it powers semantic search and retrieval for chatbots.

The surprising part is that relationships become directions. In real word2vec embeddings, the arrow from man to woman points roughly the same way as the arrow from king to queen. So king - man + woman lands near queen. When we look up the answer we skip the three input words, which is what real analogy tools do too, otherwise the nearest word is often just king.

Be honest about this toy: we hand-placed these 38 words in 2 dimensions so the patterns are easy to see. Real embeddings are learned from billions of words and have hundreds or thousands of dimensions, so a word can be close to many different groups at once. The analogies there are approximate, they fail often, and they can also capture unfair stereotypes present in the training text.

Where engineers use this

Embeddings sit at the input of every large language model, and embedding search finds relevant documents for chatbots, recommends songs and products, and detects near-duplicate images. Vector databases exist just to store embeddings and search them fast.

Words to know

Embedding
A learned vector of numbers that represents a word, token, image or other item.
Vector
An ordered list of numbers, which you can picture as an arrow or a point in space.
Nearest neighbour
The stored vector closest to a query vector under some distance measure.
Cosine similarity
A score from -1 to 1 that measures how closely two vectors point in the same direction.

Talk about it

Goal: Explore a map where similar words sit close together, and do word arithmetic like king - man + woman.

  • Why might "bank" be hard to place on a single map? Think about river banks and money banks.
  • If embeddings are learned from text written by people, what unfair patterns could they pick up?
  • Why does a real embedding need hundreds of dimensions instead of two?

Without a screen: On a big sheet of paper, place 15 word cards (animals, foods, family words, places) so that similar words are near each other. Draw an arrow from "puppy" to "dog", then copy the same arrow starting at "kitten". Does it point at "cat"? Rearrange until most arrows agree.

Curriculum links: CSTA K-12 3B-AP-08 (describe how artificial intelligence drives software and physical systems); AI4K12 Big Idea 2 (Representation and Reasoning); Common Core HSN-VM (vector and matrix quantities)

Questions

What are embeddings in AI?
Embeddings are lists of numbers that represent words, sentences or images so a computer can compare them. Items with similar meaning get similar vectors, so the distance between vectors measures how related they are.

Does king minus man plus woman really equal queen?
In real word2vec embeddings the result lands closer to queen than to most other words, once you exclude the input words. It is approximate rather than exact, and many other analogies fail.

How many dimensions do word embeddings have?
Classic word2vec and GloVe vectors often had 100 to 300 dimensions. The token embeddings inside modern large language models have thousands of dimensions.