> The course “Working with AI: The Basics”, promt.engineer. Lesson 3 of 8. Lesson page: https://promt.engineer/en/lessons/3/. All lessons: https://promt.engineer/en/llms.txt

# Lesson 3. Chat and memory: what the model remembers from a conversation

## What this lesson is about

In this lesson we will look at how a chat conversation with the model works and what the model actually remembers. In the first two lessons we treated the model as a program that takes text split into tokens and continues it token by token. In a chat, however, we usually hold a conversation. We ask a question, get an answer, ask a follow-up, or ask the model to redo something. The model takes into account everything said earlier. So it is natural to assume that it remembers the conversation the way a conversation partner would.

This picture is convenient, but it does little to explain what every frequent user of an AI assistant runs into sooner or later. In a new chat, the model knows nothing about the previous one, even if that chat was open a minute ago. In a very long conversation, it starts to forget what was agreed at the start, mix up details and repeat itself. Sometimes the assistant acts as if it had received instructions the user never gave. It keeps to a certain tone, avoids some topics, or offers to search the internet unprompted. And some assistants remember things about the user from past conversations.

All of this comes from how the chat works. Once we understand that, it is much easier to tell when to continue a conversation and when to start a new one.

First, we will look at what the model receives each time we send a message in the chat. Then we will see what else the chat app adds to our conversation. We will also come back to the context window from Lesson 2 to see why long conversations get worse over time. At the end of the lesson, we will find out how the memory feature in many AI assistants works and who else sees our conversation. After that, we will work out when it is better to continue a conversation and when to start over.

## What the model receives with each message

Let’s start with what we already know about the model. It receives text and writes a continuation of it. Once the answer is ready, its work on the request is done. As we saw in Lesson 1, the model’s parameters do not change during a conversation. So the model has no way to record anything about our conversation inside itself. How, then, does it know what was said several messages earlier?

The chat app stores the conversation. Each time we send a new message, the chat app collects the whole conversation so far: our messages and the model’s answers. It adds the new message at the end and passes all of this to the model as one text. The model reads the conversation from the beginning, from the first message to the last. Then it continues the conversation with one more answer.

It is as if a new employee answered each message. Along with the question, this employee gets a printout of the whole conversation so far. They remember nothing themselves, but everything they need is right in front of them.

Here is an example. Suppose the user and the assistant have already exchanged two messages each, and the user is now asking a third question. The model will then receive roughly this text:

```
User: My name is Anna, and I am running a project to move our office.
Assistant: Nice to meet you, Anna! How can I help with the move?
User: Make a list of tasks for the first week.
Assistant: 1. Agree on the moving date. 2. Choose a moving company. 3. Make an inventory of the furniture and equipment.
User: What is my name, and what project am I working on?
```

So that the model can tell who wrote which message, the chat app marks where each message starts and ends with special tokens. In the example, we replaced these tokens with the labels “User” and “Assistant” to make the text easier to read. With this text in front of it, the model has no trouble answering. The user’s name is Anna, and she is running the office move. Both facts are right there in the text that the model continues.

We tested this on two current models. When the question came with the earlier conversation, both models correctly named Anna and the office move. When we sent the same question on its own, without the conversation, both said they did not know the name or the project. That information was not in the text they received.

This is exactly what happens in a new chat. The chat app starts collecting the conversation from scratch, and nothing said in other conversations goes into the request. The one exception is the memory feature, which we will cover at the end of the lesson.

## What else the chat app adds to the conversation

The conversation is not the only thing that goes into the request. Before the user’s first message, the chat app inserts a text written by the assistant’s developers. This text is the **system prompt** (also called system instructions), and the user usually does not see it. It explains who the assistant is and how it should behave. It covers the tone of the answers and how to handle sensitive topics. It also says when to offer a web search and which other features of the chat app the assistant can use.

The developers also add information that the model would not otherwise have, such as today’s date. A model knows about the world only what was in the texts it was trained on. Without this hint, it would have no way of knowing what day it is.

System prompts are usually not published, but there are exceptions. Anthropic, for example, publishes the system prompt of the Claude chat app on its website. Among other things, it gives the current date and the model’s approximate knowledge cutoff. This is the point in time up to which the texts for its training were collected. The system prompt also has rules for sensitive topics and guidance on how to behave after making a mistake. Finally, it lists the chat app features that the model can suggest to the user.

All of this works just like the conversation. The system prompt is ordinary text at the start of the request, marked with special tokens. During training, models learn to give such instructions priority over the user’s messages. This is why the assistant usually follows them, even when the user asks for the opposite. This explains much of the behavior we mentioned at the start of the lesson. Tone, restrictions and the habit of suggesting a search come partly from training and partly from instructions the user never saw.

We ran into something similar while preparing this lesson. To test the example with Anna, we first ran the model through a program that adds a folder path to the system prompt. It is the path of the folder on the computer where the program was started. With no earlier conversation, we asked: “What is my name, and what project am I working on?” The model read both the user name and the project name from the folder path, although the question did not mention the folder. That information was in the text the model received, and the model used it.

Settings that users choose themselves go into the request in the same way. Suppose the assistant’s settings say how to address the user or in what style to answer. The chat app then adds this text to the request. It does the same when the conversation takes place inside a project with shared instructions. Attached files and web search results also reach the model this way. We will look at them more closely in Lesson 4.

Personal instructions and project instructions go to the model with every request. So they should contain only what is true for any conversation: what the user does, what style of answer they want, and what to avoid. It is best to keep them short and specific. They use up tokens in every request, and the model follows specific rules more reliably than vague ones.

## Why long conversations go wrong

Now that we know what a request is made of, we can return to the context window from Lesson 2. As a reminder, it is the largest number of tokens the model can take into account at once. The chat app sends the whole conversation to the model with every message. So the request grows longer with each new message.

Let’s look at how fast it grows. Suppose each exchange (a question and its answer) takes about 600 tokens, or roughly 500 words of English text. After ten exchanges, each new request will take about 6 thousand tokens. After fifty, it will take about 30 thousand, not counting the system prompt. Because the model reads the conversation again each time, the tokens of all fifty requests add up. Over fifty answers, the model will have to process more than 750 thousand tokens in total.

A chat subscription does not charge separately for tokens, but it has usage limits too. Where these limits are measured in tokens, a long conversation uses them up faster than a short one. Through the API, you pay for tokens, and each of those fifty requests is paid for in full. So long conversations cost noticeably more than short ones.

**Prompt caching** cuts part of this cost. The server that runs the model can remember for a while how it processed the beginning of a request. If the next request begins in exactly the same way, the server does not need to process that part again. Such input tokens cost 10 to 20 times less than ordinary ones.

In a conversation, this happens on its own. Each new request is the earlier conversation plus one new message, so it begins the same way as the previous request. Prompt caching also works with a subscription. With some services, input taken from the cache uses up less of the usage limit.

It is worth looking more closely at two conditions that caching depends on. First, the beginning of the request must match exactly, word for word. Suppose you change something near the start, such as the system prompt or the first message. Then everything after the change is processed and paid for again. Second, the cache does not last long, usually from five minutes to an hour after the last request. After a long pause, the whole conversation is processed and counted again.

> **For developers.** Put the fixed part of the request (instructions, documents, examples) at the beginning and the parts that change from request to request at the end. The cache will then work more often.
>
> Only a long enough beginning is cached, usually from a few hundred to a thousand tokens. Some providers cache automatically, while others need a separate request parameter. Writing to the cache sometimes costs a little more than ordinary processing. The `usage` data in the API response shows how many tokens came from the cache.

When the conversation no longer fits in the context window, the chat app has to do something about it. Assistants handle this in different ways. Some drop the oldest messages without telling the user, some replace them with a short summary, and some offer to start a new conversation. In the first two cases, the model no longer sees the whole beginning of the conversation. Anything agreed in the first messages may then be lost.

The second approach is worth a closer look. It is called compaction: the chat app retells the conversation in a short summary. Claude works this way, for example, and so do developer tools such as Claude Code. As the conversation nears the limit of the context window, the chat app sends the model a separate request asking for a summary. From then on, the model receives this summary instead of the old messages, and the conversation continues from there. In Claude, the full conversation stays on the screen, so compaction is easy to miss.

The summary is much shorter than the original conversation, and the model itself decides what goes into it. Details that seemed unimportant at the moment of compaction may drop out. The developers warn about this themselves. The Claude Code documentation says the summary keeps the user’s requests and the key pieces of code. Detailed instructions from early in the conversation, however, may be lost. The ChatGPT help pages say something similar about the summary you can use to start a new chat: it may miss details.

So anything that must stay in force until the work is done is safer in the project instructions or your personal instructions. As we saw in the previous section, the chat app adds these to every request, so compaction does not lose them.

> **For developers.** In Claude Code and Codex, you can start compaction yourself with the `/compact` command. In Claude Code, you can also add a note on what the summary should focus on.
>
> Compaction is available in the API as well. With Anthropic, you turn it on with a request parameter, and the API summarizes the conversation once input tokens reach a set threshold. Later requests drop everything before the summary, and you can write your own instruction for it. The OpenAI Responses API has a separate method that does this and returns a compacted window for the next request.

Quality can drop much earlier, though, long before the context window fills up. The longer the text, the less reliably the model keeps track of each detail in it. A requirement from the first message ends up buried under dozens of later messages. The model misses it more and more often.

There is one more reason, and it follows from Lesson 1. The model’s answers stay in the conversation and become part of the text the model continues. If an early answer contained a mistake, it stays in the request even after the user points it out. It goes on shaping the answers that follow. This is why, in a long conversation full of failed attempts, the model often slips back into a mistake that was already corrected.

This leads to a practical conclusion. If a conversation has grown long and the model is getting confused, you will usually do better in a new chat. In the first message, briefly state everything important: the task, what you have agreed on and what you have already worked out. That request will be shorter and cleaner. The model will not have to wade through all the drafts that have piled up.

## How the memory feature works

One question from the start of the lesson is still open: how some assistants remember the user between conversations if the model stores nothing. The answer will be familiar from the previous sections. The chat app manages the memory, and the model only receives text.

The memory feature works differently in different assistants, but it usually relies on two techniques. The first is notes about the user. The chat app keeps short facts separate from the conversation: the user’s name, what they do and what kind of answers they prefer.

The model itself can decide that a fact is worth remembering. When it notices something in the conversation that may be useful later, it writes a special command in its answer. The chat app recognizes the command and saves the note. The same happens when the user asks the assistant outright to remember something. Lesson 5, on tools, covers this exchange of commands between the model and the chat app in detail. In a new conversation, the chat app adds the notes to the request, next to the system prompt.

The second technique is searching past conversations. The chat app finds passages that relate to the new question and adds them to the request as well.

The model itself still does not change. Everything the assistant “remembers” is stored in the chat app and reaches the model as part of the request. Two things follow from this, and both matter in everyday work. First, the notes can become outdated or inaccurate. They will then affect the answers without you noticing, because the model treats them like any other text in the request. Second, the memory may hold things the user would rather not carry into other conversations, such as details of a work project.

So it is worth checking the memory settings from time to time. Many assistants let you view, correct or delete the saved facts, and you can also turn the memory off. For individual conversations there is temporary chat. In temporary chat nothing is added to the memory, and often the memory is not used at all.

## Who else sees the conversation

The way a chat works has one more consequence that matters for your job. The chat app stores the conversation, not the model. So the whole conversation, along with any attached files, sits on the developer’s servers. What happens to it next depends on the rules of each service.

Many services use conversations from personal plans to train their next models. In ChatGPT and Gemini, for example, this is on by default, and Claude asks about it when you sign up. You can usually turn it off in the settings, but there is a catch. If you rate an answer with the thumbs-up or thumbs-down button, the whole conversation may still be used for training. OpenAI and Google say so openly.

Each service has its own rules, and they change. Here is how they looked on November 19, 2026.

| Service | Training on conversations | Temporary chat |
|---|---|---|
| ChatGPT | on by default, can be turned off in settings | yes, kept for up to 30 days |
| Claude | chosen at sign-up, can be changed in settings | yes (incognito), kept for 30 days |
| Gemini | on by default, can be turned off in settings | yes, kept for about two months |
| DeepSeek | anonymized, can be turned off in settings | none found |

People also read some conversations. Google, for example, says that human reviewers read some Gemini chats. They rate the quality of the answers and look for rule violations. The chats they review are kept for up to three years. Anthropic says its staff see a conversation only in two cases. Either the user sent it in with feedback, or a rule violation needs checking.

Business plans and the API work differently. By default, OpenAI, Anthropic and Google do not train on conversations from business plans or on requests made through the paid API. This is why company documents are safer to handle on such a plan.

A simple rule follows from this. Do not put anything into a regular chat that you would not show an outsider. That includes passwords, customers’ personal data and anything the company treats as secret.

The developers themselves ask users to do this. Google advises against entering confidential information that you would not want a reviewer to see. OpenAI asks users not to share sensitive information in their conversations. If you need to discuss something personal, at least turn on temporary chat, which we described above. In ChatGPT, Claude and Gemini, these conversations are not used for training, though they stay on the servers for a while.

## What follows from this

Let’s sum up. The model stores nothing between messages; the chat app stores the conversation. With each message, it sends the conversation to the model along with the system prompt, and sometimes with notes from memory. If the conversation is too long, the chat app sends a shortened version. Everything the model knows about the conversation and the user comes from this text. Four things follow from this that matter when you work in a chat.

First, the model knows about the conversation only what is in the request. It learned facts about the world during training, but it has no other source for anything about this conversation or this user. A new chat starts with a clean slate, apart from what the memory feature adds. If a task depends on something from an earlier conversation, it is safer to provide it again. For the same reason, when the assistant behaves oddly, the cause often lies in other parts of the request. These can be the system prompt, the settings or the memory.

Second, the longer the conversation, the higher the risk that the model will get confused. The request grows, and things agreed early on get lost among later messages. Mistakes in earlier answers also keep influencing new ones. When the model starts to get confused, it is better to open a new chat and give it a short summary of everything important.

Third, the assistant’s memory is also just text that the chat app adds to the request. It shapes the answers like any other part of the request. So it is worth checking what the memory contains from time to time.

Fourth, the conversation is stored on the developer’s servers. On personal plans, it may be used for training and shown to reviewers. So anything you would not show to an outsider is better kept out of an ordinary chat.

In the next lesson, we will keep looking at what goes into the request. We will see where the model gets information about the world: what it learned in training and what it receives from search and attached files.

## Self-check questions

**1. In a new chat, someone wrote: “Let’s pick up where we left off yesterday.” The model asked what they meant. Why?**

<details><summary>Answer</summary>

A new chat starts with a blank slate, so yesterday’s conversation is not part of the request. If the memory feature saved nothing, the model has no way to know about it. Give it a brief reminder of what you discussed, or go back to the earlier chat.

</details>

**2. At the start of a long conversation, a user asked the model to avoid bureaucratic jargon. A few hours later, the jargon came back. Why?**

<details><summary>Answer</summary>

The instruction was left at the very start of a long conversation, under dozens of newer messages. As a result, the model began to follow it less closely. Repeating the instruction helps. So does starting a new chat with a short summary of the task.

</details>

**3. The model was trained on texts up to a certain date, but the assistant knows today’s date. Where does it get it from?**

<details><summary>Answer</summary>

Most likely, the chat app adds the date to the system prompt. The model then finds it in the request, like any other text.

</details>

**4. A colleague says: “The memory feature keeps training the model on my conversations.” What is wrong with this?**

<details><summary>Answer</summary>

The model itself does not change. The chat app stores the memory. It holds notes about the user and parts of past conversations, and the app adds them to the request as ordinary text.

</details>

**5. The model got a budget total wrong and accepted the correction. A few messages later, it used the old total again. Why?**

<details><summary>Answer</summary>

The wrong answer is still in the conversation. It goes back into the request along with the correction, so it can affect the answer again. It is safer to start a new chat and give the correct figures from the outset.

</details>

**6. A company holds long conversations with the model through the API, and its costs grow faster than the number of messages. Why?**

<details><summary>Answer</summary>

With each new message, the whole conversation is sent to the model again. So each request is longer than the one before, and the company pays for all of it again. Prompt caching makes the repeated beginning cheaper. Older parts of the conversation can also be replaced with a short summary.

</details>

---

Next lesson: [Lesson 4. What the model knows and where its mistakes come from](https://promt.engineer/en/lessons/4/index.md)
