> The course “Working with AI: The Basics”, promt.engineer. Lesson 2 of 8. Lesson page: https://promt.engineer/en/lessons/2/. All lessons: https://promt.engineer/en/llms.txt

# Lesson 2. Tokens: the pieces the model builds text from

## What this lesson is about

In this lesson we will look at how the model sees the text it receives and what pieces it uses to build its answer. In Lesson 1, to keep things simple, we said that the model predicts the next word. In fact, it works with tokens. We only mentioned then that a token can be a whole word, part of a word or a punctuation mark. Now it is time to see what these pieces are and where they come from.

Tokens explain several things that anyone who works with an AI assistant runs into. A model that easily writes several pages of coherent text sometimes gets it wrong when asked to count the letters in a word. The limits on the length of a message, an answer and the whole conversation are also measured in tokens. Finally, in many models the same text takes more tokens in other languages than in English.

Such text hits the limits sooner. If you work through an API, it also costs more. An API is a programming interface for connecting the model to your own applications. Through an API, you pay for every token. With a subscription to an ordinary chat app, the price does not depend on the number of tokens. A subscription still limits how much work you can do, though.

First, we will look at what an ordinary phrase looks like once it is split into tokens. Then we will see why the model finds such pieces easier to work with than single letters or whole words. We will also see where the model’s set of tokens comes from. After that, it will be clear why some languages take more tokens than others and why counting letters is hard for the model. At the end of the lesson we will talk about limits: what exactly they restrict and how to allow for them in your work.

## What text looks like after it is split into tokens

Let’s start with what an ordinary phrase looks like once it is split into tokens. We will take the phrase from Lesson 1 and add its most likely continuation: “Let’s reschedule the meeting for tomorrow”. Before the text reaches the model, it passes through a **tokenizer**, a program that splits text into tokens. Here is how the tokenizer of current OpenAI models does it (the vertical bars show where one token ends and the next begins):

```
Let's| res|chedule| the| meeting| for| tomorrow
```

The result is seven tokens. The words “tomorrow”, “meeting”, “for” and “the” stay whole, each with the space before it. To the tokenizer, a space is a character just like a letter, and it most often comes at the start of a token. “Let’s” is also a single token. The word “reschedule”, however, splits into two parts: “res” and “chedule”. They do not match the prefix “re” and the word “schedule”.

The same tokenizer splits the Russian version of this phrase, “Давай перенесём встречу на завтра”, into 10 tokens. We will see why a little later in the lesson. For now, note something else. The split depends on the tokenizer, so different models divide the same phrase in different ways. On Kimi K3.1 Pro, for example, the English phrase takes 6 tokens and the Russian one 16.

It is worth looking more closely at what the model actually receives. The tokenizer has a dictionary: a list of every token it knows, drawn up in advance. In the OpenAI tokenizer, this dictionary has about two hundred thousand entries. Each token in it has a number, and the model receives a sequence of these numbers. So for the model, the word “tomorrow” with the space before it is one number. It cannot directly see which letters make up the word.

The answer is built from the same tokens. As we saw in Lesson 1, the model picks one token from the dictionary at each step. The tokenizer then turns the chosen numbers back into text.

## Why the model works with tokens

A natural question is why we need such odd fragments at all. It might seem simpler to give the model text letter by letter or word by word. The split would then be easier to follow. To answer this, let’s look at what would happen in each case.

If we split text into single characters, the dictionary would be small, but the text would become very long. The meeting phrase has 41 characters, so it would take 41 steps instead of 7. A one-page answer would take about two thousand. The model would have to make many more predictions to write the same answer and work with much longer sequences. All of this would make it slower and more expensive.

If we make every word a token instead, we get the opposite problem. A language has a huge number of words, and many of them come in several forms. English has fewer forms than many languages, but even here we have “meet”, “meets”, “met”, “meeting” and “meetings”. On top of that come names, company names, terms, typos, words from other languages and pieces of program code. Such a dictionary would be huge and still incomplete. A new word or a typo would not be in it, so the model could neither read nor write it.

Tokens get around both problems. Frequent words go into the dictionary whole, so text made of them stays short, while rare words are built from several parts. The dictionary also holds very short fragments, down to single characters. Because of this, tokens can make up any text, even a word the model has never seen. For example, “unpredictability” is not in the dictionary as a whole, so the tokenizer writes it in three parts: un|predict|ability.

That leaves the question of where the dictionary comes from and why it contains these particular fragments. It is built automatically from a large collection of texts, before the model is trained.

Here is a simplified version. At first, the dictionary holds only single characters. Then a program looks for the pair of neighboring fragments that occurs most often in the collection. Suppose it is the letters “t” and “h” standing next to each other. A new token “th” then enters the dictionary, and from now on this pair counts as one fragment in every text.

After this replacement, a different pair becomes the most frequent, say “th” and “e”. The step repeats again and again until the dictionary reaches the target size, for example two hundred thousand entries.

As a result, frequent combinations gradually grow into whole words, while rare ones stay built from parts. The pair “schedule” and “reschedule” shows this well. The first, with the space before it, is so common that it went into the OpenAI tokenizer’s dictionary whole. The rarer second word takes two tokens, “res” and “chedule”, and the boundary ignores the prefix “re”. This is why token boundaries do not match prefixes and roots: the tokenizer knows nothing about grammar. It only tracks which combinations appeared more often in the texts.

> **Did you know?** Inside a computer, every character is stored as one or more bytes, and the smallest fragments in the tokenizer’s dictionary are single bytes. That is why the tokenizer can write any text, even one it has never seen. It is also why a token boundary sometimes falls right inside a character. Emoji show this well.
>
> In the OpenAI tokenizer, the smiley 🙂 takes one token, a country flag such as 🇯🇵 takes four, and the “programmer” emoji 🧑‍💻 takes six. It is built from two emoji, a person and a laptop, with an invisible joiner between them.

## Why some languages take more tokens

Now we can return to the difference we noticed at the start of the lesson. The English phrase took 7 tokens, and its Russian version took 10. The reason lies in how the dictionary is built. Tokens appear for combinations that occur often in the collection of texts, and English usually dominates such collections. This is why English words are more likely to enter the dictionary whole, while words in other languages are assembled from parts. Languages with other scripts or many word forms end up with especially few whole words.

Long words show this most clearly. The OpenAI tokenizer writes “unpredictability” as 3 tokens and the Russian “непредсказуемость” as 6. Having many word forms also plays a part. In Russian, for example, the word for “meeting” appears as “встреча”, “встречи”, “встречу” and other forms. Each form on its own is less frequent, so it is less likely to enter the dictionary whole.

How large the gap is depends on the language and on the tokenizer. To measure it, we took the three fragments from Lesson 1 and translated them into five other languages. Then we counted the tokens in each version with each model’s own counter. The figure in brackets is the ratio to the English text in the same model:

| Language | GPT-6 Astra | Claude Opus 5.5 | Kimi K3.1 Pro |
|---|---|---|---|
| English | 166 tokens | 218 tokens | 167 tokens |
| Spanish | 169 (1.02×) | 332 (1.52×) | 236 (1.41×) |
| German | 184 (1.11×) | 439 (2.01×) | 280 (1.68×) |
| Russian | 181 (1.09×) | 310 (1.42×) | 292 (1.75×) |
| Chinese | 190 (1.14×) | 282 (1.29×) | 141 (0.84×) |
| Hindi | 232 (1.40×) | 541 (2.48×) | 457 (2.74×) |

The ratio depends on how much room each language got in each model’s dictionary. The OpenAI tokenizer treats the languages most evenly: none of them takes more than 1.4 times as many tokens as English. The dictionaries of Claude and Kimi are built differently, and their gaps are larger. For Hindi, the gap reaches 2.5 times in Claude and 2.7 in Kimi.

Kimi even needs fewer tokens for the Chinese text than for the English one, because its dictionary gives Chinese a lot of room. Even English varies: Claude counts 218 tokens where the other two count about 166. So the token count depends on the model, whatever the language.

This means the ratio between languages is not a fixed number. It depends on both the language and the model. If cost or usage limits depend on the token count, count the tokens in your own texts. Some model developers publish pages where you can paste a text and see how it splits into tokens. As a rough guide, all of Lesson 1 of our course, about 2,600 words, takes about 3,100 tokens in the OpenAI tokenizer. That is roughly 1.2 tokens per word.

> **Did you know?** Not long ago, text in many languages took far more tokens in OpenAI models. Before GPT-4o arrived in 2024, they used the GPT-4 tokenizer, whose dictionary of about a hundred thousand entries gave other languages little room. On the three fragments from the table above, it counted 166 tokens for English, 195 for Spanish and 236 for German. Chinese took 288, Russian 323 and Hindi 779.
>
> So Hindi took 4.7 times as many tokens as English, and Russian almost twice as many. When the dictionary doubled, the gap shrank: Hindi now takes 1.4 times as many tokens as English, and Russian about 1.1 times.

> **For developers.** For OpenAI models, you can count tokens with the `tiktoken` library:
>
> ```python
> import tiktoken
>
> enc = tiktoken.get_encoding("o200k_base")
> tokens = enc.encode("Let's reschedule the meeting for tomorrow")
> print(len(tokens), [enc.decode([t]) for t in tokens])
> ```
>
> Other developers have their own tokenizers, and their APIs usually offer separate methods for counting tokens.

## Why models find it hard to count letters

Let’s go back to the fact that the model receives a sequence of numbers. This leads to another quirk that seems strange at first. Tasks that are easy for a person, such as counting the letters in a word, can be hard for the model.

Take the question “How many letters ‘r’ are in the word ‘strawberry’?” A person sees the whole word and counts the letters. The model receives three tokens, “st”, “raw” and “berry”, and each of them is just a number in the dictionary. To answer, it needs to know that “st” has no “r”, “raw” has one and “berry” has two. The model only learns such facts indirectly: from texts where words were spelled out letter by letter, split into syllables or hyphenated. This is usually enough, but not always, and then the model confidently gives the wrong number.

For the same reason, models struggle with tasks where individual characters matter. Think of reversing a word, writing a headline of exactly 40 characters, or finding all words in a text that begin with a given letter. It does not help that the same word can look quite different to the model. With a space before it, “tomorrow” is one token, and “Tomorrow” at the start of a sentence is also one. But “TOMORROW” is four: T|OM|OR|ROW.

Numbers cause a similar problem. The OpenAI tokenizer splits them into groups of three digits, starting from the left. So the model sees 12345678 as 123|456|78 and 1234567 as 123|456|7. As a result, the same place values in different numbers land in different tokens. The ones digit of the first number is in the token “78”, and the ones digit of the second is in “7”.

When people add numbers in a column, they line the digits up by place value. The model has to guess which digits share a place. This is one reason it makes mistakes in calculations with long numbers.

Modern models handle such tasks much better than earlier ones, especially when they reason before answering or can run a program to do the counting. We will cover this in Lessons 5 and 6. Still, the cause of the difficulty has not gone away. So if the exact character count really matters, check the result with the character counter in your editor. A headline with a length limit is one example.

## What the limits restrict

The last topic in this lesson is the limits that are measured in tokens. Sooner or later, everyone who works with a model runs into them. They are much easier to understand once you know what a token is.

The most important of these limits is the **context window**. It is the largest number of tokens the model can take into account at one time. The window must hold the message, any documents attached to it and the answer itself. In a chat, it must also hold the earlier conversation.

In current models, the window ranges from several hundred thousand to a million tokens. A window of 200,000 tokens could hold more than sixty lessons like our first one. That is a lot, but it is not unlimited. A large report, several contracts or a long conversation can fill it completely. In Lesson 3 we will look in detail at why the whole conversation goes into the window and what happens when it fills up.

The second limit concerns the length of the answer. Besides the window, a model usually has a cap on how many tokens it can write at one time. If the answer reaches this cap, the text breaks off, even in the middle of a thought. When that happens, it is usually enough to ask the model to continue. As we saw in Lesson 1, the model continues text. Everything written so far is the start of a text, and the model adds the next tokens to it.

Finally, cost is also counted in tokens. When you work through the API, you pay for every token, and tokens come in two kinds. **Input tokens** are everything the model receives: the message, attached documents, instructions and, in a chat, the earlier conversation. **Output tokens** are what the model writes. That means the answer and, in models that reason before answering, the draft of the reasoning too, even if the user never sees it. Price lists and bills use these same names.

Output tokens cost noticeably more than input tokens, and the reason lies in how the model works. It can process all the text it receives at once, in a single pass. But as we saw in Lesson 1, it writes the answer one token at a time. Each new token takes a separate step of computation. With leading models, an output token usually costs five times as much as an input token. For example, with Claude Opus 5.5 a million input tokens cost $4, and a million output tokens cost $20.

Long answers, long reasoning (more on this in Lesson 6) and large documents attached to every request therefore raise the final bill noticeably.

If you use a chat app on a subscription, you do not pay for each token separately. But a subscription has limits too. Most often, the limit is on the amount of work. More and more services measure it by how many tokens the model had to process and write, not by the number of messages. Each service has its own rules, and they change often. Here is how they looked on November 18, 2026.

| Service | What is counted | Time windows |
|---|---|---|
| ChatGPT | no limit on ordinary conversation; files, images, the most powerful model and Codex are limited | for Codex, 5 hours and a week, depending on the plan |
| Claude | the total amount of work in the chat and in Claude Code | 5 hours and a week |
| Kimi | credits that are deducted per token | a month; Kimi Code also has a 5-hour window |

Where a service counts the amount of work, the more tokens your requests and answers take, the faster you use up the limit. So everything in this lesson applies to subscriptions as well. Long conversations, large documents and long reasoning use up the limit faster. With some models, text in certain languages does too. In Lesson 3 we will look at why a long conversation is especially costly and how prompt caching cuts part of that cost.

## What follows from this

Let’s sum up. The model works with tokens. These are pieces of text from a dictionary built in advance, and the model receives them as numbers. The dictionary is built from the combinations that occur most often in a large collection of texts. This is why frequent words appear in it whole, while rare words are assembled from parts. Three practical consequences follow from this.

First, the model does not see letters directly. Counting letters, keeping a text to an exact length and calculating with long numbers all depend on individual characters. The model does such tasks less well than you might expect from a program that writes so well. It is worth checking its results.

Second, the model measures the size of a text in tokens. The same text takes a different number of tokens in different languages and in different models. If money or usage limits depend on this, count the tokens in your own texts. Use the tokenizer of the model you work with.

Third, both the context window and the length of an answer have limits. If an answer breaks off, you can ask the model to continue. Too much material, however, will not fit into the context window. Many services also measure subscription usage in tokens, so long requests and answers use up the limits faster. In the next lesson, we will see how these limits show up in an ordinary chat.

## Self-check questions

**1. The model was asked to write a slogan of no more than 40 characters, but it wrote 47. Why?**

<details><summary>Answer</summary>

The model sees text as tokens and cannot see how many characters each token contains. So it can only estimate the length. It is worth checking the result with the character counter in your editor.

</details>

**2. A colleague says: “The model reads text letter by letter, only very fast.” What is wrong here?**

<details><summary>Answer</summary>

The model receives tokens, not letters. A frequent word can be a single token, while a rare one splits into several parts. The model knows only indirectly which letters make up a token.

</details>

**3. Why does the same text often take more tokens in other languages than in English?**

<details><summary>Answer</summary>

The tokenizer’s dictionary is built from frequent combinations. English usually dominates the text collections used for training, so other languages have fewer whole words in the dictionary. How large the gap is depends on the language and the model. On our three fragments, it ranges from slightly fewer tokens than English to about 2.7 times more.

</details>

**4. The model was writing a long report, and the answer stopped in the middle of a sentence. What should you do?**

<details><summary>Answer</summary>

Most likely, the answer hit the length cap. Ask the model to continue. It will add the ending to what it has already written.

</details>

**5. A company wants to translate its requests to the model into English to pay less. How can it tell whether this will save money?**

<details><summary>Answer</summary>

It should count how many tokens its usual requests take in both languages, using the model it works with. On our fragments, the difference ranges from slightly fewer tokens than English to about 2.7 times more. It depends on the language and the model. Besides, the translation itself costs time or money. And the answer is usually still needed in the original language.

</details>

**6. Without using a program, the model added two long numbers and got one digit wrong. Why does this happen?**

<details><summary>Answer</summary>

The model sees numbers as tokens too. The OpenAI tokenizer, for example, splits them into groups of three digits. So the same digit places in different numbers end up in different tokens. If the exact result matters, it is better to work it out with a program or a calculator.

</details>

---

Next lesson: [Lesson 3. Chat and memory: what the model remembers from a conversation](https://promt.engineer/en/lessons/3/index.md)
