promt.engineer Working with AI: The Basics/Lesson 1 of 8
Lesson 1 of 8 · ≈ 12 min

How a language model writes an answer

What happens between a question and its answer, where the model gets this ability, and why the answers differ each time.

Start reading Lesson text for an AI agent

What this lesson is about

In this lesson we will look at how a language model writes an answer to a message. A language model is a program trained to work with text: it takes in text and writes text in response. Models like this run inside ChatGPT, Claude, Gemini, DeepSeek and other AI assistants. People use them to draft letters, work through documents, and write and check code. You can use them without ever thinking about how the model works.

Without that understanding, though, much of its behavior looks strange. The model may give two different answers to the same question. It may confidently cite a source that does not exist. A small edit to the wording can noticeably change its answer. All of this follows from the way the model puts its answer together, which is why the course starts here.

First, we will look at what the model does with the text it receives. Then we will see where this ability comes from and how it turns the model into an assistant that answers questions. At the end of the lesson, we will gather the conclusions. They will be useful in your work, and the following lessons build on them.

The model predicts how a text continues

Let’s start with the most important point: what the model actually does with the text it receives. Its answers can be very complex, but they all rest on a single operation. The model looks at the text so far and estimates which piece is most likely to come next.

Let’s look at a simple example. Take the start of the sentence “Let’s reschedule the meeting for”. It could go on in several ways: “tomorrow”, “Friday”, “Monday”, “next week”. The model weighs all the possible options at once and gives each one a probability. The result might look like this (the numbers are made up, purely for illustration):

“Let’s reschedule the meeting for …”
ContinuationProbability
tomorrow31%
Friday18%
Monday12%
next9%
……

The probabilities fall this way because the model saw sentences like this many times during training. It learned which continuations usually follow them. After “reschedule the meeting for”, the word “tomorrow” comes up often and “potatoes” almost never. The model’s estimates reflect that. We will see how exactly it learned this a little further on.

The pieces of text the model works with are called tokens. A token can be a whole word, part of a word, or a punctuation mark. We will look at tokens in detail in Lesson 2. For now, to keep things simple, let’s assume the model predicts the next word.

How an answer is built from single predictions

A single prediction gives only one next token. But a model’s answer usually runs to several sentences, sometimes to pages. So the next question is how these single steps add up to a whole answer.

The answer is built by repeating the same step. The model picks a token, adds it to the end of the text and makes a new prediction. This time the text is one token longer. Then the model picks the next token, adds that too, and so on. This continues until the model picks a special token that marks the end of the answer. Each new prediction takes into account both the original message and everything the model has written so far.

This is why the answer appears gradually in the chat app: the interface shows the tokens as the model picks them. There is no finished answer waiting anywhere. The model writes it as it goes, and the screen only shows that work. One more thing follows from this: the model does not erase what it has already written. If a mistake appears early in the answer, the rest of the text is written with that mistake taken into account. The model may notice the mistake and correct itself, but often it carries on in the same direction.

That leaves the question of where this work actually happens. The models in this course run on servers owned by the companies that make them. These servers are powerful computers in data centers. When we press “Send”, the app or website sends our message over the internet to one of these servers. There the model writes its answer, sending the tokens back to our device in small batches as they are ready. The phone or laptop only sends the message and shows the answer, and the model itself is not on it.

A message’s journey: from your laptop to the company’s servers and back
  1. We type a message in an app or on a website and press “Send”.
  2. The app sends the message over the internet to the developer’s servers.
  3. The model runs on these servers. It picks the first token of the answer.
  4. Each token goes back to the laptop as soon as it is ready, so the answer appears gradually.
  5. The laptop only sent the message and showed the answer. The model itself did not run on it.
Did you know? can you run a model yourself

Small models can also run on your own computer, for example in programs such as LM Studio or Ollama. Then your message stays on your machine. But these models are noticeably weaker than the ones that run in data centers. The largest models run on several server graphics cards at once and would not fit on an ordinary computer.

Where the model gets this ability

So the model estimates the probabilities of possible continuations and builds its answer from them. That leaves the question of how it knows which continuations are likely. Nobody gave it a table saying that “tomorrow” most often comes after “Let’s reschedule the meeting for”. This ability comes from training.

Inside the model are a great many numbers that its calculations depend on. They are called parameters. Modern models have billions of them, and the largest have trillions. When the model receives a text, the parameters determine what probabilities it gives to the next token. Change the parameters and the probabilities change too. So training a model comes down to finding the parameters that make it predict well.

The parameters are tuned on a huge amount of text: books, articles, websites, documentation and program code. The model is shown a piece of this text and asked to predict the next token. The correct answer is known in advance: it is the token that actually comes next in the text. The model’s prediction is compared with that token. If the model gave the correct token a low probability, the parameters are adjusted slightly so that this probability goes up in similar situations. This step is repeated an enormous number of times, on trillions of tokens, and each time the model’s predictions get a little more accurate.

The model learns to predict what comes next by training on a huge amount of text

Note that similar openings are followed by different words in different texts. In some emails the meeting is rescheduled for tomorrow, in others for Friday. So the model ends up learning how often each continuation occurs. That is where the probabilities in the table come from.

It is worth looking more closely at what the model learns from this training. Memorizing common word combinations is not enough to predict how all kinds of texts continue. The model has to pick up grammar and facts, such as the fact that Paris is the capital of France. It also has to learn how letters and contracts are usually laid out, how code is organized and how a line of reasoning unfolds.

Nobody writes these rules in by hand: they gradually take shape in the parameters because they help the model predict the next token more accurately. So a task that looks simple at first, guessing how a text continues, grows into something much larger. It becomes the ability to write coherent text on all sorts of topics.

How a language model becomes an assistant

Still, a model that can only continue text does not yet answer questions or do what it is asked. To see the difference, imagine a model that has had only the training described above. We give it the text “How can I politely turn down a customer who asks for a discount?” The model will continue it the way similar texts usually continue in its training data. Online, questions like this often come in lists, for example in FAQ sections or in the tables of contents of articles. So a perfectly plausible continuation is more questions:

How should I answer a customer who demands a refund? How can I calm down an unhappy buyer?

From the point of view of training, the model made no mistake: it continued the text plausibly. But the continuation is of no use. This is why the models behind AI assistants go through one more stage of training. At this stage, the model is shown many dialogues in which questions get good answers and tasks get done. People also compare different answers from the model and mark which are more useful, more accurate and safer. The model’s parameters are then adjusted to match these ratings.

After this training, the model continues text in the role of an assistant. If the message contains a question, the most likely continuation becomes an answer. If the message contains a request, the most likely continuation is to carry it out. The mechanism itself stays the same. The model still writes its answer token by token, choosing each next token by its probability. The only change is in which continuations the model considers likely.

The mechanism is the same: the answer is written token by token; only the continuations the model considers likely have changed

Both stages of training happen before the model is put to use. Once the assistant is answering our messages, the model’s parameters no longer change. Everything we write affects the answer only as part of the text that the model continues.

So if we correct the model in a chat, the correction becomes part of the conversation. It will affect later answers, but it will not change the model itself. Developers may use the conversation to train future versions, but that will be a different model. We will look at how the chat app keeps earlier messages in Lesson 3.

Why the answers differ each time

Now we can explain one of the features mentioned at the start of the lesson. If you ask the model the same question twice, the answers will almost certainly differ. The reason lies in the step where the model chooses the next token.

So far we have said that the model chooses a token based on probabilities, but not exactly how. The simplest way is to always take the most likely token. However, texts written this way often come out monotonous. Sometimes they even start going in circles and repeating themselves. This is why the token is usually chosen at random, but weighted by the probabilities. In our example, “tomorrow” will come up about a third of the time, “Friday” less often, and unlikely options almost never.

Because of this randomness, two answers to the same question can part ways within the first few words. Each new token depends on the ones before it. So a small difference at the start sends the rest of the text in another direction. Different answers to the same question are therefore normal behavior for the model. Nothing has gone wrong.

Each token is picked at random, weighted by the probabilities

A setting called temperature controls how random the choice is. The lower the temperature, the more alike the answers to the same question will be. In chat apps, the developers set the temperature in advance, and the user usually does not see it.

Did you know? why it is no longer set

Until recently, you could set the temperature of almost any model through the API, that is, when connecting the model to your own program. People often set it to zero to get the same answer every time. They raised it to get more varied texts, although a high temperature also brings more errors. Once models that reason before answering appeared, providers began removing this option. For many of the newest models, you cannot set the temperature at all.

What follows from this

Let’s sum up. A language model writes an answer by predicting the next token, again and again. It learned to make these predictions by training on a huge amount of text. Three things follow from this. They matter when you work with a model, and we will come back to them in later lessons.

First, the model does not look up answers in a ready-made store. It has no database of answers to questions. What it learned during training is spread across its parameters.

Even a well-known poem is reproduced the same way. The model saw it so often in training that it can predict it line by line. The model writes every answer from scratch. This is why it can write a letter or any other text it has never seen before.

Second, the model writes plausible text, and plausible does not mean correct. The model picks continuations that fit well with what it has already written. Often the plausible answer and the correct one are the same. Well-known facts appeared many times in the training texts, so the model reproduces them correctly.

But when the model lacks the information it needs, it can still write a coherent, confident text. In that case the content is made up. During training, assistants are taught to admit when they lack information. This does not always work, though. So a confident tone says nothing about whether an answer is correct. We will discuss how to check answers in Lesson 7.

Third, the user’s message is the beginning of the text that the model continues. Everything in the message affects the probabilities of the next tokens, and so it shapes the whole answer. If you say who a letter is for, its tone will change. If you attach a document, the model will draw on its content.

This is why the wording of a request matters so much. A small edit to a question can noticeably change the answer. In Lesson 8 we will look in detail at how to write requests with this in mind.

Self-check questions

Question 1 of 6

You ask a model the same question twice and get two different answers. Why?