What Is Tokenization and How Does It Work?

When you send text to a language model, it does not read letters or words the way you do. It reads tokens. Before any reasoning happens, your text is chopped into these small pieces, each mapped to a number, and it is those numbers that flow through the model. Tokenization is that first, invisible step, and it quietly shapes cost, context limits, and even how well a model handles certain inputs.

It is easy to ignore until it surprises you: a short-looking prompt that costs more than expected, a word that confuses the model, or a context window that fills faster than the word count suggests. All of these trace back to how the text was tokenized.

This guide explains what tokenization is, how it works, the main approaches, and why it matters in practice.

What is tokenization?

Tokenization is the process of converting raw text into a sequence of tokens that a model can process. A token is a unit of text, often a subword such as a common word fragment, that exists in the model's vocabulary. Each token has an integer ID, and the model works with those IDs rather than the original characters.

Think of a tokenizer like splitting a sentence into Lego bricks from a fixed set. Common words are single bricks, while rare words are built from several smaller bricks. The model has seen every brick before, even if it has never seen your exact sentence, so it can always assemble meaning from the pieces.

The key point is that the model's world is made of tokens, not words. A word like unbelievable might be one token or three, depending on the tokenizer, and that choice affects everything downstream.

How does tokenization work?

Text passes through the tokenizer before it ever reaches the model's layers.

• Normalize the text. The tokenizer may lowercase, strip, or standardize characters so inputs are consistent.
• Split into tokens. Using a learned vocabulary, it breaks the text into subword tokens, keeping common pieces whole.
• Map to IDs. Each token is replaced by its integer ID from the vocabulary table.
• Embed the IDs. The model turns each ID into a vector, the numeric representation it actually computes on.
• Decode on the way out. Generated token IDs are mapped back to text to produce the readable response.

Most modern models use subword methods like Byte-Pair Encoding or WordPiece, which learn a vocabulary from data. These balance having a small vocabulary against keeping common words as single tokens, so text stays compact.

Word vs. subword vs. character tokenization

There are three broad strategies. Word-level tokenization treats each word as a token, which is intuitive but struggles with rare or unseen words and needs a huge vocabulary. Character-level tokenization uses single characters, which handles anything but makes sequences very long. Subword tokenization sits in between: common words stay whole while rare ones split into pieces, which keeps sequences short and handles new words gracefully.

Subword tokenization is why almost every modern LLM works the way it does. It is the practical middle ground that keeps vocabularies manageable while still covering any input, including typos, code, and other languages.

Why does tokenization matter?

• Cost. Most APIs bill per token, so tokenization directly determines what a request costs.
• Context limits. The context window is measured in tokens, so tokenization decides how much fits.
• Performance. Fewer tokens for the same meaning means faster, cheaper processing.
• Language fairness. Some languages tokenize into far more tokens than English, raising their cost for the same text.
• Edge cases. How numbers, code, and symbols are split can affect how well a model handles them.

What are the pitfalls?

• Token count surprises. Text can use more tokens than its word count suggests, especially code or non-English text.
• Model-specific counts. The same text tokenizes differently across models, so counts are not portable.
• Awkward splits. Rare words, long numbers, or unusual symbols can split in ways that trip a model up.
• Hidden cost drivers. Whitespace, formatting, and repeated structure all consume tokens you might overlook.

When should you think about tokenization?

You should pay attention to tokenization whenever cost, context limits, or multilingual support matter. If you are trimming prompts to fit a window, estimating API spend, or working in a language other than English, the token count is the number that actually governs your constraints.

For casual, short prompts it rarely matters. But as soon as you build a product on top of a model, understanding tokens is essential to controlling cost and staying within limits.

Best practices for working with tokens

• Measure token counts with the target model's own tokenizer, not a word-count estimate.
• Trim redundant whitespace, boilerplate, and formatting that quietly consume tokens.
• Budget prompts in tokens, leaving room for the response within the context window.
• Watch non-English and code-heavy inputs, which often use more tokens than they appear to.
• Cache stable prefixes so you do not pay to re-tokenize and reprocess the same text.
• Track token usage in production to catch cost and context issues before they grow.

Conclusion

Tokenization is the quiet first step that turns your text into the tokens a model actually reads. It decides how much your input costs, how much fits in the context window, and how gracefully a model handles rare words and other languages. Understanding it is the difference between guessing at cost and limits and controlling them deliberately.

TrueFoundry's AI Gateway gives teams visibility into token usage across every model and provider, alongside caching, routing, and budgets in one layer, so the token counts that drive cost are easy to see and manage. Tokens are the unit everything in an LLM system is measured in, and watching them is basic hygiene for running one.

‍