What this lesson is about
In this lesson we will look at where the model gets the information it uses in its answers. The previous lessons established two things. The model continues text based on what it learned in training, and it knows about a particular conversation only what is in the request. Now let’s bring these two ideas together. We will see what the model knows and how far that knowledge can be trusted. We will also see what changes when the information it needs is right there in the request.
This has a direct effect on how useful an AI assistant is at work. A model will retell the classics without a hitch and explain how a tax system works. Yet the same model may not know about last month’s events. It may mix up dates or details, and, as we saw in Lesson 1, it may confidently cite a source that does not exist. But attach the right document to your question or turn on web search, and the answer becomes noticeably more accurate. To know when you can trust the model and when it needs help, you need to understand how its knowledge works.
First, we will look at what information the model picks up in training and why some of it sticks more firmly than the rest. Then we will turn to the knowledge cutoff. We will see why the model knows nothing about recent events and why it cannot always tell where the cutoff lies. After that, we will find out where invented facts come from. Finally, we will look at how web search and attached files work. This will show why information given in the request is usually more reliable than what the model remembers on its own.
What the model learns in training
Let’s start with how knowledge gets into a model in the first place. As we saw in Lesson 1, a model learns to predict the next token by training on a huge number of texts. Facts about the world end up in its parameters because they help it guess the continuation more accurately. If the training texts say many times that the capital of France is Paris, it is useful for the model to learn this. Then, after the words “The capital of France is”, it will confidently predict “Paris”.
An important property follows from this: how firmly the model knows something depends on how often it appeared in the training texts. The model usually learns a fact well if it is repeated in thousands of books, articles and websites. But a common misconception can be learned just as firmly. Information the model saw only a few times leaves a weak trace in its parameters. The model reproduces it with less confidence, and sometimes with errors. This might be the founding date of a small company, say, or the name of the author of a little-known article.
And the model cannot know private information that was never in the training texts at all. That includes the contents of a company’s internal documents, personal correspondence and data from closed systems. The model can reason about such things, but it has no source for the specific facts.
This is easy to see with texts everyone knows. We asked the two current models we use to test this course’s examples to briefly retell O. Henry’s story “The Gift of the Magi”. The story has been reprinted again and again, read in schools and discussed online. Both models retold it accurately, down to the characters’ names, Della and Jim. One of them also mentioned details such as the platinum chain for Jim’s watch and the combs for Della’s hair.
It is worth looking more closely at how this knowledge is organized. As we said in Lesson 1, the model does not store it as separate records. Everything it has learned is spread across its parameters, and information about similar things is intertwined there.
This is why the model picks up general patterns well. It knows how lease agreements are usually structured, how one type of tax differs from another and how code is written in popular programming languages. But when a specific detail has to be reproduced exactly, information about similar things can get mixed up. The model may credit one scientist with a colleague’s discovery. Or it may give a year that is close to the right one but not quite it. And it will say this in the same confident tone it uses to retell “The Gift of the Magi”.
Where the knowledge cutoff lies
A model’s knowledge depends on what its training texts contained. It also depends on when those texts were collected. Training texts are gathered up to a certain point, and the model knows nothing about events after it. This point is called the knowledge cutoff. The gap between the knowledge cutoff and the day a user asks a question is usually several months, and often more than a year. The model is first trained, then tested and released, and after that people go on using it for a long time.
So the model’s knowledge of anything that changes quickly may be out of date: prices, laws, new versions of software, the people who run companies. What is more, the model does not always realize this. It relies on what the training texts said. If those texts called a particular version of a program the newest, the model may still call it the newest today.
There is also a less obvious difficulty. Training texts usually say less about the last few months before the cutoff than about earlier times. This is because articles, discussions and mentions keep appearing long after an event. As a result, the model often knows less about those last months than about the more distant past.
The gap between the cutoff and today is usually several months, and often more than a year.
If you ask a model about its knowledge cutoff, it may judge by the events it knows about. Few texts cover the last months before the cutoff, so the model may name an earlier date than the real one. For the same reason, the developer’s documentation is a more reliable source for the cutoff than the model’s own answer.
This is why developers of AI assistants add the current date and the knowledge cutoff to the system prompt, which we discussed in Lesson 3. For example, the published system prompt of the Claude chat app gives the month up to which the model knows about events. It also says the model may not know about later events. In such cases, it should offer the user a web search.
Where invented facts come from
Now we can return to the question this course began with: why a model sometimes confidently states things that are not true. In Lesson 1 we answered it in general terms. The model writes a plausible continuation of the text, and a plausible text is not always a correct one. Now that we know how the model’s knowledge is organized, we can make this picture more precise.
Let’s look at an experiment we ran while preparing this lesson. We first tried O. Henry. Both models quoted the openings of his best-known stories word for word, and one of them did the same for a little-known story. Widely reprinted classics, it turns out, are remembered very well.
So we chose a less famous author. We asked two models to give, word for word, the first two sentences of four stories from one collection by Frank R. Stockton. The stories ran from the most famous to the least known, and we checked the answers against the text on Wikisource:
| Story | First model | Second model |
|---|---|---|
| “The Lady, or the Tiger?” | word for word (modern spelling) | word for word (modern spelling) |
| “The Transferred Ghost” | first sentence correct, second invented | declined |
| “The Spectral Mortgage” | declined | declined |
| “Our Archery Club” | declined | declined |
The most important thing in this table is how the models presented their answers. The first model gave the invented sentence in quotation marks, as a word-for-word quote, with no caveat. Its tone is no different from that of the exact quote from “The Lady, or the Tiger?”. Here is the sentence it gave as the second sentence of “The Transferred Ghost”:
“It was the abode of a genial, though somewhat eccentric, hospitality; it was the home of an intelligent and agreeable family; and it was the residence of Miss Mary Hinckman.”
Stockton’s real second sentence is shorter:
“It was the abode of a genial, though somewhat impulsive, hospitality.”
The heroine of the story is called Madeline, not Mary.
The result fits well with what we said at the start of this lesson. “The Lady, or the Tiger?” is Stockton’s most reprinted and discussed story. The models seem to have seen it so often in training that they reproduce its opening exactly. The rarer a text was in training, the weaker its trace in the parameters. Still, the model can build a plausible continuation from what it does know: the names of the characters, the setting, the author’s style.
This is how we get sentences that sound like Stockton and even contain correct details, but are not in the story. The invented sentence, for example, begins almost exactly with Stockton’s words. The rest has nothing in common with the real text. This is the mixing we spoke about above: information about similar things merges into one confident answer.
With “The Transferred Ghost”, the second model behaved differently and declined to give a quote. In the last two cases, both models declined and asked for the text. As we said in Lesson 1, assistants are trained to admit when they lack information. In this test, the models declined in five of their eight answers. We also asked both models to retell O. Henry’s “The Friday Kettle”, a story that does not exist, and both said they did not know it.
Still, you cannot rely on this. The first model declined twice, yet it invented a sentence for a story it half knew. And different models behave differently in the same situation.
A practical rule follows from this experiment. A model reproduces most accurately what appeared many times in its training texts. The rarer the subject, the higher the risk of an error. Be especially careful with exact quotes, numbers, dates, document titles and references to sources. Errors in them are the hardest to notice, and they look plausible even when they are invented.
Information in the request
So far we have talked about what the model learned in training. But as we saw in Lesson 3, the model has a second source of information: the text of the request. This source works quite differently. Information stored in the parameters has to be reconstructed. Information in the request is right in front of the model, and retelling it is much easier than recalling it.
We tested this with a simple example. We asked both models how many days in advance an employee at Northern Wind Ltd. must agree annual leave with their line manager. The company is made up, so its rules could not have appeared in the training texts. One model said it did not have the company’s policy and asked us for it. The other confidently answered “at least 14 days in advance”, a number it had invented.
Then we asked the same question again, this time with a fragment of the internal policy placed before it. The fragment was section 4.2, which sets the deadline. It reads: “Annual leave must be agreed with your line manager no later than 21 calendar days before it starts.” With the fragment in the request, both models gave the exact answer straight away: 21 calendar days. This shows the main point even more clearly. Without the source, a model may give an answer that is confident, specific and wrong.
This brings us to the main practical conclusion of the lesson. If a task depends on exact information, it is better to include it in the request than to hope the model remembers it. This applies to internal documents, recent data, the exact wording of laws and contracts, and figures from reports. Even if the model saw the text in training, it makes far fewer mistakes with the document in front of it than from memory. Mistakes are still possible: it may misread a table, for example, or mix information from the document with its own knowledge. But the risk is much lower.
How search and files work
Many AI assistants can search the internet and read attached files on their own. From the outside, it looks as if the model went online or opened a document. In reality, as with the memory feature in Lesson 3, the chat app deals with the outside world, and the model receives text.
Here is how search works. When the model gets a question, it decides that it needs up-to-date information.
It then writes a special command with a search query in its answer. The chat app runs the search, loads the pages it found or parts of them, and adds this text to the request. The model then carries on with the information in front of it. It writes the answer, often with links to its sources. We will look at this exchange of commands between the model and the chat app in Lesson 5.
Files work in much the same way. When the user attaches a document, the chat app extracts its text and adds it to the request. Models that can work with images can also receive pictures, such as document pages or diagrams. These pictures become part of the same request.
A very large file, or many files at once, may not fit in the context window. Chat apps handle this in different ways. Some say the file is too large, and others cut it off. Many search the document for the parts that best match the question and put only those parts in the request.
It is worth looking more closely at pictures, because people often send screenshots to the model. Images, too, reach the model as tokens. The chat app divides an image into small squares, for example 28 or 32 pixels on each side. Each square becomes a token. Images that are too large are scaled down first. So a picture, like text, takes up space in the context window, and its cost depends on its size in pixels.
We took a screenshot of an expense table with 12 rows and sent it to three current models. A screenshot of the whole 1920×1080 screen cost about 2,500 to 2,700 tokens. The table cropped from it cost about 200. That is even cheaper than the same table as text, which took 286 to 405 tokens.
The models read the pictures well. In every run, they copied all 36 numbers correctly, both from the cropped table and from the whole screen, where the font was small.
Errors appeared only when we shrank the screenshot to half its size. Then one model got two numbers in one row wrong, in both of its attempts. The wrong numbers looked quite plausible: 333.94 instead of 333.84. So it is better to send the model only the part you need, not the whole screen. If the model has copied numbers from a picture, check them, especially if the picture is small or compressed.
For developers how to set the picture size in the API
In the API, you can set the size of a picture yourself. OpenAI models have the detail parameter for this. At the low setting, the image is scaled down to 512 by 512 pixels and costs few tokens. With original, it keeps its full size. Claude scales down images whose long side is over 2,576 pixels or that would take more than 4,784 squares.
Two things follow from this, and both matter in everyday work. First, an answer can be no more reliable than the information in the request. If the search found an outdated article or an unreliable website, the model may rely on it as confidently as on a good source. Current models often notice an old date or sources that contradict each other. They do not always do so.
Second, the model sees only what the chat app put into the request. If the chat app included the wrong parts of a large document, the model may answer as if the section you need is missing. It may also fill the gaps from memory. So when accuracy matters, check which sources the model cited. It also helps to paste the relevant section of the document straight into your message. We will look at how to check answers in Lesson 7.
What follows from this
Let’s sum up. A model has two sources of information. One is the knowledge it acquired in training. The other is the text of the request, to which the chat app adds the conversation, instructions, search results and the contents of files. Three things follow from how the model gets its knowledge. They are worth remembering whenever something important depends on the model’s answer.
First, knowledge from training is uneven. The model is more confident about things that appeared many times in its training texts, though even these can turn out to be common misconceptions. It knows rare facts less well, and it knows nothing about events after its knowledge cutoff. The rarer and newer the topic, the more cautiously you should treat the answer.
Second, the model can be confidently wrong when it mixes up similar facts. Assistants are trained to admit what they do not know, but you cannot count on it. This matters most for exact quotes, numbers, dates and references.
Third, the model reproduces information from the request more accurately than it recalls things from memory. You still need to check the answer, because the source may be wrong and the model may misread it. If a task depends on exact data, give that data to the model directly. Attach a document, paste the relevant passage or turn on search. Then check what the model based its answer on.
In the next lesson we will look at how an assistant searches for information, reads files and takes actions on its own. We will also see how this leads to AI agents.