promt.engineer Working with AI: The Basics/Lesson 4 of 8
Lesson 4 of 8 · ≈ 14 min

What the model knows and where its mistakes come from

Where the model gets its information, where its knowledge ends, why it invents facts, and why information in the request is more reliable than its memory.

Start reading Lesson text for an AI agent

What this lesson is about

In this lesson we will look at where the model gets the information it uses in its answers. The previous lessons established two things. The model continues text based on what it learned in training, and it knows about a particular conversation only what is in the request. Now let’s bring these two ideas together. We will see what the model knows and how far that knowledge can be trusted. We will also see what changes when the information it needs is right there in the request.

This has a direct effect on how useful an AI assistant is at work. A model will retell the classics without a hitch and explain how a tax system works. Yet the same model may not know about last month’s events. It may mix up dates or details, and, as we saw in Lesson 1, it may confidently cite a source that does not exist. But attach the right document to your question or turn on web search, and the answer becomes noticeably more accurate. To know when you can trust the model and when it needs help, you need to understand how its knowledge works.

First, we will look at what information the model picks up in training and why some of it sticks more firmly than the rest. Then we will turn to the knowledge cutoff. We will see why the model knows nothing about recent events and why it cannot always tell where the cutoff lies. After that, we will find out where invented facts come from. Finally, we will look at how web search and attached files work. This will show why information given in the request is usually more reliable than what the model remembers on its own.

What the model learns in training

Let’s start with how knowledge gets into a model in the first place. As we saw in Lesson 1, a model learns to predict the next token by training on a huge number of texts. Facts about the world end up in its parameters because they help it guess the continuation more accurately. If the training texts say many times that the capital of France is Paris, it is useful for the model to learn this. Then, after the words “The capital of France is”, it will confidently predict “Paris”.

An important property follows from this: how firmly the model knows something depends on how often it appeared in the training texts. The model usually learns a fact well if it is repeated in thousands of books, articles and websites. But a common misconception can be learned just as firmly. Information the model saw only a few times leaves a weak trace in its parameters. The model reproduces it with less confidence, and sometimes with errors. This might be the founding date of a small company, say, or the name of the author of a little-known article.

The model knows best what appeared in many texts

And the model cannot know private information that was never in the training texts at all. That includes the contents of a company’s internal documents, personal correspondence and data from closed systems. The model can reason about such things, but it has no source for the specific facts.

This is easy to see with texts everyone knows. We asked the two current models we use to test this course’s examples to briefly retell O. Henry’s story “The Gift of the Magi”. The story has been reprinted again and again, read in schools and discussed online. Both models retold it accurately, down to the characters’ names, Della and Jim. One of them also mentioned details such as the platinum chain for Jim’s watch and the combs for Della’s hair.

It is worth looking more closely at how this knowledge is organized. As we said in Lesson 1, the model does not store it as separate records. Everything it has learned is spread across its parameters, and information about similar things is intertwined there.

This is why the model picks up general patterns well. It knows how lease agreements are usually structured, how one type of tax differs from another and how code is written in popular programming languages. But when a specific detail has to be reproduced exactly, information about similar things can get mixed up. The model may credit one scientist with a colleague’s discovery. Or it may give a year that is close to the right one but not quite it. And it will say this in the same confident tone it uses to retell “The Gift of the Magi”.

Where the knowledge cutoff lies

A model’s knowledge depends on what its training texts contained. It also depends on when those texts were collected. Training texts are gathered up to a certain point, and the model knows nothing about events after it. This point is called the knowledge cutoff. The gap between the knowledge cutoff and the day a user asks a question is usually several months, and often more than a year. The model is first trained, then tested and released, and after that people go on using it for a long time.

So the model’s knowledge of anything that changes quickly may be out of date: prices, laws, new versions of software, the people who run companies. What is more, the model does not always realize this. It relies on what the training texts said. If those texts called a particular version of a program the newest, the model may still call it the newest today.

There is also a less obvious difficulty. Training texts usually say less about the last few months before the cutoff than about earlier times. This is because articles, discussions and mentions keep appearing long after an event. As a result, the model often knows less about those last months than about the more distant past.

Texts thin out towards the knowledge cutoff, and after it the model knows nothing
training textsknowledge cutofftoday

The gap between the cutoff and today is usually several months, and often more than a year.

If you ask a model about its knowledge cutoff, it may judge by the events it knows about. Few texts cover the last months before the cutoff, so the model may name an earlier date than the real one. For the same reason, the developer’s documentation is a more reliable source for the cutoff than the model’s own answer.

This is why developers of AI assistants add the current date and the knowledge cutoff to the system prompt, which we discussed in Lesson 3. For example, the published system prompt of the Claude chat app gives the month up to which the model knows about events. It also says the model may not know about later events. In such cases, it should offer the user a web search.

Where invented facts come from

Now we can return to the question this course began with: why a model sometimes confidently states things that are not true. In Lesson 1 we answered it in general terms. The model writes a plausible continuation of the text, and a plausible text is not always a correct one. Now that we know how the model’s knowledge is organized, we can make this picture more precise.

Let’s look at an experiment we ran while preparing this lesson. We first tried O. Henry. Both models quoted the openings of his best-known stories word for word, and one of them did the same for a little-known story. Widely reprinted classics, it turns out, are remembered very well.

So we chose a less famous author. We asked two models to give, word for word, the first two sentences of four stories from one collection by Frank R. Stockton. The stories ran from the most famous to the least known, and we checked the answers against the text on Wikisource:

The first two sentences of four stories by Frank R. Stockton: what the models gave
StoryFirst modelSecond model
“The Lady, or the Tiger?”word for word (modern spelling)word for word (modern spelling)
“The Transferred Ghost”first sentence correct, second inventeddeclined
“The Spectral Mortgage”declineddeclined
“Our Archery Club”declineddeclined

The most important thing in this table is how the models presented their answers. The first model gave the invented sentence in quotation marks, as a word-for-word quote, with no caveat. Its tone is no different from that of the exact quote from “The Lady, or the Tiger?”. Here is the sentence it gave as the second sentence of “The Transferred Ghost”:

“It was the abode of a genial, though somewhat eccentric, hospitality; it was the home of an intelligent and agreeable family; and it was the residence of Miss Mary Hinckman.”

Stockton’s real second sentence is shorter:

“It was the abode of a genial, though somewhat impulsive, hospitality.”

The heroine of the story is called Madeline, not Mary.

The result fits well with what we said at the start of this lesson. “The Lady, or the Tiger?” is Stockton’s most reprinted and discussed story. The models seem to have seen it so often in training that they reproduce its opening exactly. The rarer a text was in training, the weaker its trace in the parameters. Still, the model can build a plausible continuation from what it does know: the names of the characters, the setting, the author’s style.

This is how we get sentences that sound like Stockton and even contain correct details, but are not in the story. The invented sentence, for example, begins almost exactly with Stockton’s words. The rest has nothing in common with the real text. This is the mixing we spoke about above: information about similar things merges into one confident answer.

With “The Transferred Ghost”, the second model behaved differently and declined to give a quote. In the last two cases, both models declined and asked for the text. As we said in Lesson 1, assistants are trained to admit when they lack information. In this test, the models declined in five of their eight answers. We also asked both models to retell O. Henry’s “The Friday Kettle”, a story that does not exist, and both said they did not know it.

Still, you cannot rely on this. The first model declined twice, yet it invented a sentence for a story it half knew. And different models behave differently in the same situation.

A practical rule follows from this experiment. A model reproduces most accurately what appeared many times in its training texts. The rarer the subject, the higher the risk of an error. Be especially careful with exact quotes, numbers, dates, document titles and references to sources. Errors in them are the hardest to notice, and they look plausible even when they are invented.

Information in the request

So far we have talked about what the model learned in training. But as we saw in Lesson 3, the model has a second source of information: the text of the request. This source works quite differently. Information stored in the parameters has to be reconstructed. Information in the request is right in front of the model, and retelling it is much easier than recalling it.

We tested this with a simple example. We asked both models how many days in advance an employee at Northern Wind Ltd. must agree annual leave with their line manager. The company is made up, so its rules could not have appeared in the training texts. One model said it did not have the company’s policy and asked us for it. The other confidently answered “at least 14 days in advance”, a number it had invented.

Then we asked the same question again, this time with a fragment of the internal policy placed before it. The fragment was section 4.2, which sets the deadline. It reads: “Annual leave must be agreed with your line manager no later than 21 calendar days before it starts.” With the fragment in the request, both models gave the exact answer straight away: 21 calendar days. This shows the main point even more clearly. Without the source, a model may give an answer that is confident, specific and wrong.

This brings us to the main practical conclusion of the lesson. If a task depends on exact information, it is better to include it in the request than to hope the model remembers it. This applies to internal documents, recent data, the exact wording of laws and contracts, and figures from reports. Even if the model saw the text in training, it makes far fewer mistakes with the document in front of it than from memory. Mistakes are still possible: it may misread a table, for example, or mix information from the document with its own knowledge. But the risk is much lower.

What follows from this

Let’s sum up. A model has two sources of information. One is the knowledge it acquired in training. The other is the text of the request, to which the chat app adds the conversation, instructions, search results and the contents of files. Three things follow from how the model gets its knowledge. They are worth remembering whenever something important depends on the model’s answer.

First, knowledge from training is uneven. The model is more confident about things that appeared many times in its training texts, though even these can turn out to be common misconceptions. It knows rare facts less well, and it knows nothing about events after its knowledge cutoff. The rarer and newer the topic, the more cautiously you should treat the answer.

Second, the model can be confidently wrong when it mixes up similar facts. Assistants are trained to admit what they do not know, but you cannot count on it. This matters most for exact quotes, numbers, dates and references.

Third, the model reproduces information from the request more accurately than it recalls things from memory. You still need to check the answer, because the source may be wrong and the model may misread it. If a task depends on exact data, give that data to the model directly. Attach a document, paste the relevant passage or turn on search. Then check what the model based its answer on.

In the next lesson we will look at how an assistant searches for information, reads files and takes actions on its own. We will also see how this leads to AI agents.

Self-check questions

Question 1 of 6

An assistant called a version of a program “the latest”, although a newer one came out a month ago. Why?