How AI chatbots write word by word
Chatbots do not look up answers. They predict the next small piece of text, again and again, using patterns learned from a huge amount of writing.
Key takeaways
Chatbots split text into tokens: whole words, word pieces or punctuation.
Attention lets each token weigh the earlier ones to build context.
The model predicts one token, adds it, then predicts the next.
It generates plausible text, so important facts still need checking.
When you ask a chatbot a question, the reply appears a few words at a time. That is not a typing effect added for drama. It is how the model actually works: a large language model writes by predicting what comes next, one small piece at a time, for every piece of every answer.
Step 1: Text becomes tokens
Language models do not read letters or whole words. They read tokens: chunks of text that might be a short word, part of a longer word, or a punctuation mark. A tokenizer splits your message into these chunks and swaps each one for a number from a fixed vocabulary, usually tens of thousands of entries or more. A word like “unbelievable” might become un, believ and able. In English, a token works out to roughly three quarters of a word on average.
Each token number is then turned into an embedding: a long list of numbers that places the token in a mathematical space where related meanings sit close together.
Step 2: Attention builds context
The model itself is a transformer, a tall stack of layers. Inside each layer, a mechanism called attention lets every token look back at the tokens before it and decide which ones matter for its meaning. In “The trophy did not fit in the suitcase because it was too big”, attention is what helps the model connect “it” to “trophy”. Layer by layer, each token's numbers are refined until they reflect the whole conversation so far.
Step 3: Predict, pick, repeat
At the top of the stack, the model gives a score to every token in its vocabulary. Those scores become probabilities: how likely each token is to come next. Then a simple loop runs:
- The system picks one token, usually by sampling from the most likely options. A setting called temperature controls how adventurous that choice is.
- The chosen token is added to the end of the text.
- The slightly longer text goes back through the model to predict the next token. In practice the model caches its work on earlier tokens, so each step only computes what is new.
The loop continues until the model produces a special end token or reaches a length limit. That is why answers stream in: each token depends on all the ones before it.
Step 4: Where the knowledge comes from
The model learned to predict during training. It read an enormous amount of text and adjusted billions of internal numbers, called parameters, every time its guess for the next token was wrong. Later rounds of training on example conversations and human feedback teach it to follow instructions and answer helpfully.
A chatbot does not retrieve a stored answer. It generates a new one, token by token, from patterns it absorbed in training.
Why it matters
This explains the quirks. Because the model predicts plausible text rather than checking facts, it can sound confident and still be wrong, which is why good tools add web search, citations or code execution. It also explains why rephrasing a question changes the answer, and why long replies take longer to finish. Understanding the loop makes you better at prompting, and at knowing when to double-check.
