Mansori

How Chatbots Pick the Next Word

About this lesson, for parents and teachers

A chatbot writes one word at a time by asking: given everything so far, what comes next? Here is a tiny language model you can watch think, bar by bar.

Ages 15–18 · about 9 minutes · builds on Coin Flips and Chance

Try this

  1. Press Next word. The bars show every word that ever followed the current word in the training text, with its count.
  2. Drag temperature below 0.3 and generate. The model almost always picks the top word and starts repeating itself.
  3. Push temperature above 1.5. The solid bars flatten out, and rare words start winning.
  4. Switch to Trigram so the model looks at the last two words. Does the writing make more sense?

How it works

A language model gives a probability to every possible next token. Ours is an n-gram model: when the lesson starts, your browser reads a short story and counts which word follows which. After "the", it saw "cat" and "dog" many times and "rug" only twice, so those counts become the model's odds. A bigram model looks back one word; a trigram looks back two.

Large language models such as ChatGPT and Claude do the same job in a far more powerful way. Instead of a lookup table, a transformer network reads the whole conversation and outputs a score, called a logit, for each of roughly 100,000 tokens. A softmax turns those scores into probabilities that add up to 1, and then one token is picked and appended. Repeat, and you get a paragraph.

Temperature controls how the pick is made. Each logit is divided by T before the softmax: p = exp(z / T) / Σ exp(z / T). Here the logit is log(count), so at T = 1 you get the plain counted odds. Small T exaggerates the gaps, so the top word almost always wins. Large T shrinks the gaps, so unlikely words get a real chance.

Neither extreme is best. Very low temperature, close to always taking the top word (called greedy decoding), is predictable but loops and repeats. High temperature is varied but drifts into nonsense. Chat apps often use a temperature somewhere around 0.7 to 1, sometimes combined with top-p sampling, which only samples from the smallest set of words whose probabilities add up to, say, 90%.

Where engineers use this

Every AI chatbot, code assistant and phone keyboard suggestion uses next-token prediction. Developers set temperature in API calls: low for factual answers and code, higher for brainstorming and stories.

Words to know

Token
A chunk of text a model reads or writes, often a word or part of a word.
Language model
A model that assigns probabilities to the next token given the tokens before it.
Temperature
A number that divides the logits before softmax. Low makes choices safer, high makes them more random.
Softmax
A function that turns any list of scores into positive probabilities that sum to 1.

Talk about it

Goal: Build a tiny language model from text and change the temperature to see creative versus safe choices.

  • Our model can only ever write words it has seen. Can a large language model write a sentence nobody has written before? How?
  • When would you want a chatbot to use low temperature, and when high?
  • The trigram model sounds more sensible but copies the story more. Why does a longer memory cause copying with so little training text?

Without a screen: Write a short paragraph, then make a card for each word listing every word that comes right after it. Start with "the", draw a random next word from its card (put repeats on the card more than once), and keep going. Read your new story aloud. That is a bigram model.

Curriculum links: CSTA K-12 3B-AP-08 (describe how artificial intelligence drives software and physical systems); AI4K12 Big Idea 3 (Learning); Common Core HSS-CP (conditional probability and the rules of probability)

Questions

How do chatbots like ChatGPT decide what to say?
They predict one token at a time. A neural network scores every possible next token given the conversation so far, the scores become probabilities, and one token is sampled. The chosen token is added to the text and the process repeats.

What does temperature mean in AI?
Temperature controls randomness when a language model samples its next token. Low values make it choose the most likely tokens, giving focused but repetitive text, while high values make less likely tokens more probable, giving more varied but riskier text.

What is next token prediction?
It is the training goal of large language models: given some text, predict the token that comes next. By practising this on enormous amounts of text, models learn grammar, facts and patterns of reasoning.