What Is a Context Window and How Does It Work?

Every language model has a hard limit on how much text it can look at in one go. Send it a short question and it has plenty of room; paste in a long report and a chat history, and at some point you hit a wall. That wall is the context window, and understanding it explains a lot of odd model behavior, from forgotten instructions to truncated answers.

The context window is not storage the model keeps between calls. It is a working budget for a single request, shared by everything you send in and everything the model writes out. When the budget is spent, something has to give.

This guide explains what a context window is, how it works, why it is limited, and how to work within it.

What is a context window?

A context window is the maximum number of tokens a model can process in a single request. Tokens are the chunks of text a model reads, roughly a few characters each, so a window of 128,000 tokens is on the order of 100,000 words. Everything in the request shares that budget: the system prompt, the user's messages, any retrieved documents, and the model's own response as it is generated.

Think of it as the model's desk. Only so many pages fit on the desk at once. The model can read and cross-reference anything on the desk, but a page pushed off the edge is gone from view. A bigger desk holds more pages, but it also takes more room and costs more to maintain.

Crucially, the model has no memory of what fell off the desk. If an earlier instruction scrolled out of the window, the model does not remember it unless you put it back in.

How does a context window work?

Text does not enter a model as words. It is converted into tokens, and those tokens fill the window in order.

• Tokenize the input. The system prompt, history, and documents are split into tokens and counted against the window.
• Fill the budget. Those input tokens occupy part of the window, leaving the rest for the response.
• Attend across tokens. The model's attention lets each token reference every other token in the window, which is how it uses context.
• Generate within the remainder. The model produces output tokens, and each one also counts against the same budget.
• Hit the limit. If input plus output would exceed the window, the request is rejected or the oldest tokens are dropped.

Because input and output share one budget, a very long prompt leaves little room for a long answer. Managing what goes into the window is as important as the size of the window itself.

Why are context windows limited?

The limit comes from how attention works. In a standard transformer, every token attends to every other token, so the compute and memory grow roughly with the square of the window length. Doubling the context can quadruple the cost, which is why larger windows need more GPU memory and run more slowly and expensively.

Model makers push windows larger with architectural tricks and more hardware, and windows have grown from a few thousand tokens to hundreds of thousands. But bigger is not free, and a long window that is stuffed with irrelevant text can even make answers worse, not better.

How do you work within a context window?

• Trim the input. Send only the text that matters, since every token spends part of the budget.
• Summarize history. Replace long past conversation with a short running summary to save room.
• Use retrieval. Fetch only the relevant passages of a large corpus instead of pasting the whole thing.
• Order carefully. Put the most important instructions where the model is most likely to use them.
• Leave room to answer. Reserve enough of the budget for the response you actually want.

What are the limits and pitfalls?

• Lost in the middle. Models often use information at the start and end of a long window better than the middle.
• Cost and latency. Filling a large window is slower and more expensive, even when the extra text does not help.
• No memory between calls. The window resets each request, so persistence must come from your own system.
• Bigger is not always better. An overstuffed window can dilute the signal and lower answer quality.

When does the context window matter most?

It matters whenever you work with long inputs or long conversations: analyzing documents, summarizing transcripts, coding across many files, or holding a long multi-turn chat. In these cases the window size and how you manage it directly shape what the model can do.

For short, single-turn questions the window is rarely a constraint. The challenge appears as inputs grow, which is when retrieval and summarization become essential rather than optional.

Best practices for managing context

• Count tokens, not words, when you estimate whether something fits.
• Retrieve relevant chunks instead of pasting entire documents into every call.
• Summarize or truncate old turns before they push important context out of the window.
• Keep critical instructions near the start or end, where models attend most reliably.
• Always reserve enough budget for the model to produce a complete answer.
• Use prompt caching for a stable, repeated prefix so you do not pay to reprocess it each call.

Conclusion

A context window is the token budget a model can see at once, shared by your input and its output. It exists because attention grows expensive as the window grows, so there is always a limit, and once text falls outside it the model simply cannot use it. Working well with a model is largely about putting the right tokens in that budget and leaving room for the answer.

TrueFoundry's AI Gateway helps teams manage context in practice, with prompt caching for repeated prefixes and observability into token usage across every model, all in one layer with routing and rate limits. Whatever the window size, spending it wisely is what makes a model useful on real, long inputs.

‍