The context window

The context window is the fixed amount of text a language model can look at in one go. It holds your question, any instructions, any files or facts you supply, and the answer being written, all at the same time. The number sounds like storage, but it behaves like a budget. The real skill is not filling it. It is deciding what deserves a place in it.

What the window actually is

A large language model does not read the way a person does. It cannot pause, walk away, and come back to where it left off. For a single request it sees one continuous block of text and nothing else. That block is the context window. Everything the model will use to answer has to be inside it: the instructions you gave, the conversation so far, any documents you pasted, and the response as it is generated word by word. There is no separate memory it consults on the side. If a fact is not in the window, for the purpose of that request the fact does not exist.

The size of the window is measured in tokens, not words. A token is a chunk of text the model treats as one unit. Some tokens are whole words, some are parts of words, some are single characters or punctuation. As a rough guide, one token is about four characters of English, so a hundred tokens is roughly seventy five words. A window described as holding two hundred thousand tokens can therefore hold something on the order of a hundred and fifty thousand words at once, which is a large book. That sounds like plenty until you try to fit a real software project into it.

The important word is fixed. The window has a hard edge. When the text you send crosses that edge, it does not spill over into a second page. It is refused, or the oldest part is quietly dropped to make room. Either way, the model never sees more than the window holds in a single pass. This single constraint shapes almost everything about how these systems are used well.

Everything about your case has to fit

A model arrives at your request already knowing a great deal in general. It has learned the patterns of many languages, the shape of common code, the outline of countless topics. What it does not know is anything specific to you. It has never seen your codebase. It does not know that your project renames a field in a way that breaks three other services. It does not know what you decided in yesterday’s discussion. All of that specific, local knowledge has to be delivered through the window, because the window is the only channel into a single request.

Think of the model as a brilliant contractor who has built a thousand houses but has never seen yours. They can do expert work, but only on the room you actually show them. If you want them to fix the kitchen, you have to bring them into the kitchen. Describing a cupboard down the hall is not the same as opening it. The context window is the set of rooms you choose to walk the contractor through before asking them to work. Whatever you leave out, they will guess at, and their guess will be a confident average of the thousand other houses rather than the truth about yours.

This is why the same model can feel brilliant on one task and useless on another. The difference is often not the model at all. It is whether the few facts that mattered for the task made it into the window. Get the right rooms in front of the contractor and the work is excellent. Show them the wrong rooms and the work is confidently wrong.

A budget, not a container

Here is the shift in thinking that separates people who get good results from people who struggle. A container is something you try to fill. A budget is something you try to spend well. The context window looks like a container, a big empty box you can pour text into, and that appearance is a trap. Treating it as a container leads to a natural but harmful instinct: if some context helps, more context must help more, so dump everything in and let the model sort it out.

The reality is closer to a budget. Every token you spend on one thing is a token you did not spend on another, and every token you add competes for the model’s attention with every other token. A window packed with material the model does not need for this task is not neutral. It actively makes the important material harder to find. The goal is not a full window. The goal is a window that contains exactly what the task requires and as little else as possible.

The context window looks like a box to fill, but it behaves like a budget to spend. Every token you add competes with every other token for the model’s attention.

Why whole files waste the budget

The most common way to waste the budget is to paste in whole files. It feels safe, even thorough. The reasoning goes: I am not sure which part matters, so I will include all of it and cover my bases. But consider what a file actually contains. A single source file might be eight hundred lines. The change you care about touches one function of thirty lines. The other seven hundred and seventy lines are imports, unrelated helpers, boilerplate, comments, and code paths that have nothing to do with your task.

When you paste the whole file, you spend most of your budget on those seven hundred and seventy irrelevant lines. Worse, you make the model’s job harder. It now has to locate the thirty lines that matter inside a wall of text that all looks plausibly relevant, and every unrelated function is a small invitation to wander off toward the wrong problem. Multiply this across five or ten files and the window fills with noise long before it fills with signal.

The better move is to bring in the thirty lines that matter, plus a precise note about how they connect to the rest: this function is called from these two places, it returns a shape the frontend depends on, it was changed alongside this other file the last several times. That connected slice is a fraction of the tokens and far more useful than the raw file, because it is the relationships that carry the meaning, and relationships are exactly what a raw file dump buries.

Context rot: reliability drops before the limit

There is a second reason not to fill the window, and it surprises people. A model does not use the whole window equally well. As the amount of text in the window grows, the model’s ability to reliably use any particular fact inside it tends to degrade, and this degradation begins well before you reach the stated maximum. A window advertised as holding two hundred thousand tokens does not mean the model attends to the hundred and ninety thousandth token as sharply as the first. This effect is sometimes called context rot, or long context degradation.

The practical shape of it looks like this. Put a single important instruction near the top of a short prompt and the model follows it well. Bury that same instruction in the middle of a very long prompt, surrounded by pages of other material, and the model may skim past it. A fact stated once, far back, competing with thousands of other tokens, is easy to lose. The limit printed on the box is the point past which text is refused. It is not a promise that everything up to that point is used with equal care.

This changes how you should read the big numbers vendors advertise. A larger window is genuinely useful, because it raises the ceiling on what can fit. But it is not permission to stop choosing. The reliable working zone is smaller than the maximum, and the way to stay inside it is the same discipline as before: put in what matters, leave out what does not, and keep the important instructions where they will be seen.

Reliability fades long before the advertised limit. A bigger window raises the ceiling. It does not remove the need to choose.

A worked example with numbers

Make it concrete with a made-up but realistic budget. Say the window holds two hundred thousand tokens. A task lands: fix a bug in one function. The function itself is thirty lines, maybe four hundred tokens. The two places that call it, another few hundred. A short note on the data shape it returns, a hundred. The instruction describing the bug, fifty. Add generous room for the model’s reply. The whole thing that actually matters fits comfortably in a couple of thousand tokens, one percent of the window.

Now watch what happens under the fill-it instinct. Rather than the one function, you paste the whole file it lives in: eight hundred lines, perhaps eleven thousand tokens, and most of it unrelated. To be safe you also paste four neighboring files, because the bug touches that corner of the code. Suddenly you are at fifty or sixty thousand tokens, thirty times the lean version, and the four hundred tokens that hold the answer are now a needle in a large stack of hay. You have paid more, you have waited longer, and you have made the model’s job harder, all while the useful content stayed the same four hundred tokens it always was.

The lesson is not that big windows are bad. It is that the size of the window and the size of what a task needs are almost never the same number, and the gap between them is space you should leave empty on purpose. An empty window is not wasted. It is room the model does not have to search through.

The window fills as a conversation grows

One more thing surprises people who move from single questions to longer sessions. In a back and forth conversation, or an agent working through a multi-step task, the window does not reset between turns. Each new message is answered with the whole conversation so far replayed into the window: your first request, the model’s first reply, your follow-up, its next reply, and so on. The history accumulates, and it accumulates in tokens you pay for and that crowd the budget.

This means a long session slowly eats its own room. Twenty turns in, a large share of the window may be taken up by earlier exchanges that no longer matter to the current step, pushing toward the zone where reliability fades. It is why long agent runs need active management of what stays in the window and what gets summarized or dropped, and why the ability to keep durable knowledge outside the window, to be pulled back in only when needed, becomes so valuable. The window is not just a size. It is a resource that gets spent over the life of a task.

The cost angle

There is also a plain economic reason to be careful. Using a model is usually priced by the token, counting both the tokens you send in and the tokens it generates. Every token in the window is a token you pay for, on every single request. If you routinely stuff whole files into the window, you pay for all of that text every time, including the large majority of it the model never needed.

For a single question the cost is small enough to ignore. For a coding agent that makes hundreds of requests while working through a task, each one carrying a bloated window, the cost adds up fast, and so does the time, because larger inputs take longer to process. A lean window is cheaper and quicker on top of being more accurate. The incentives point the same way the quality does: spend the budget on signal, not on noise.

Deciding what goes in is the real job

Put the pieces together and a clear conclusion falls out. The model’s general ability is largely fixed for a given request. The window is fixed. Reliability fades toward the edges, and every token has a price. The one lever you actually control is the choice of what to place in the window. That choice is where most of the quality of an answer is won or lost.

Said plainly, the hard part of working with these models is not phrasing the question. It is selecting the information. For a small task you can do this selection by hand, pasting the one function that matters. For anything large, and certainly for a real codebase, you cannot do it by hand, because you would first have to know which thirty lines out of a million are the relevant ones. Answering that question, reliably and automatically, is the whole game.

To choose well you have to understand the units the window is measured in, which is the story of tokens and tokenization. You have to understand why the model attends unevenly across the window, which comes from the transformer and attention. And you have to understand the systems built specifically to decide what enters the window on your behalf, which is the work of the context layer and, one step upstream of it, retrieval-augmented generation. The window sets the constraint. Those ideas are how the constraint gets managed.