What Is a Context Window? Why It Decides How Well AI Works for You

If an AI assistant was sharp at ten o'clock and had lost the brief by three, the model did not get worse in between. What changed was what it could see. This article explains the context window: what it is, how a model reads it, why long sessions go off, and what you can do about it. No code is needed to follow it.

The short version: a model answers only from the text in front of it at that moment, and most of that text is yours to choose. The quotations come from my book The Context Window.

The desk, not the mind

The context window is the stretch of text a language model can see at the moment it answers you. Think of it as a desk. The model works from what is on the desk and nothing else.

The model does not know what you know. It knows what you showed it.

Three plain facts follow from this.

First, the model is stateless. Each call takes a block of text in and produces a block of text out, and when it is finished nothing survives. The next call starts exactly as clean as the first. Anything the assistant seems to remember, some piece of software arranged for it to be told again.

Second, a chat is a document. It feels like a conversation with turns, but underneath it is a transcript that grows by one entry each turn and is read in full, from the top, every time the model replies. The model is not continuing a thought it had a moment ago. It is reading the minutes.

Third, everything is counted in tokens. Models do not read words. They read fragments of text: often a whole short word, sometimes part of a longer one, sometimes a single punctuation mark. Every limit and every bill is counted in them, and the model's reply has to fit in the same window as your question. If your prompt uses most of the room, the answer gets cut off.

So when a product says its assistant "has memory", it means something specific. Somewhere outside the model there is a store of notes. On each new call, some of those notes are fetched and placed on the desk next to your request. The model did not remember. It was reminded.

Fitting in is not the same as being noticed

Windows have grown from a few pages to something like a shelf of novels. It is tempting to treat that as the end of the problem. If everything fits, why choose?

Because there is a difference between something being in the window and something being noticed. The first is a matter of capacity. The second is a matter of attention: when the model produced its answer, how much weight did that material actually carry? Capacity has grown a great deal. Attention has not grown in step.

Position matters too. Researchers studying long inputs found that models use information well when it sits near the very beginning or the very end, and less well when the same information sits in the middle. The effect has varied across models and has narrowed as they have improved, but it is steady enough to plan around. It is known as "lost in the middle".

The most troublesome material is not the obviously irrelevant kind. A cooking recipe in a tax question is easy for a model to ignore. The risk is the near miss: last year's version of the procedure, the refund policy for a different product line, the function with the same name in another module. These look right enough to attract attention and are wrong enough to mislead.

Why long chats go off

Every long session gets heavier. Each turn adds a message, a reply and perhaps several tool results, and all of it stays in the window to be read again on the next turn. By the later turns the model is carrying a large history, much of it finished business. Replies get slower, costs rise and quality often drifts down.

Practitioners call the general effect context rot. As the context grows, performance on the task gradually degrades, even when everything needed is still present. It is not a sudden failure at the edge of the window. It is a slow decline that begins well before the limit.

Two specific versions are worth knowing by name.

Context poisoning is when an error enters the history and is then treated as fact. A made-up function name or a misread requirement is read again on every later turn, and later steps are built on it. The session looks coherent while being wrong at the root.

The echo is the model imitating itself. Every reply becomes part of the context for the next, so an early habit, framing or flawed approach tends to continue. The model is not being stubborn. It is being consistent with the document it is reading, and that document is increasingly its own earlier work.

This is why correcting a long chat often fails. The correction is one more entry, and the mistake is still on the page above it.

When the conversation has gone wrong, more conversation is rarely the fix. A clean page usually is.

The practical move is a handover note. Before you end a session, write down the goal, where things stand, what was decided and what comes next. Start a fresh chat and paste that in. A fresh window with a short brief often does better than a long one that has been patched.

A bigger window is not the fix

Every increase in window size has been met with the same prediction: now that everything fits, selection will stop mattering. Just put it all in. Windows have grown enormously and the prediction has not come true.

A bigger room does not make a tidier desk. It makes a bigger mess possible.

Nearly every beginner's instinct is additive. The answer was weak, so add more instructions. The model missed something, so add more documents. It forgot a rule, so repeat the rule in capitals. Each addition feels responsible. Together they bury the request, and the model's attention gets spread thinner with each layer. The more useful skill is subtraction: removing the stale, the duplicated and the near miss until what remains is what the task needs.

When an AI system gives poor results, the reflex is to blame the model and try a bigger or newer one. Sometimes that helps. More often the problem lies in the context, and changing the model only changes which context failures you see.

Habits that help today

None of these need special tools.

Order the page. Put long, stable material first: background, reference documents, standing instructions. Put the specific question last, so the model arrives at the task with the task still fresh.

Brief a clever stranger. Imagine someone capable and widely read who knows nothing about your organisation, your earlier decisions or your private shorthand. Assume intelligence. Do not assume knowledge.

Give the reason with the rule. "Never use bullet points" gets applied literally. "Write in flowing prose, because these replies are read aloud and lists sound robotic" costs a few more words and lets the model work out what you meant in cases you did not list.

Show an example. "Friendly" means a hundred different things. A sample reply means one.

Ask for quotes first. When the answer must come from a long document, ask the model to find and quote the relevant passages, then answer from those quotes. It has to locate its evidence before it gives an opinion.

Read what the model read. When an answer is wrong, read the actual text the model received, top to bottom, as if you knew nothing else. The evidence is usually there.

What to take from this

• The context window is everything the model can see when it answers. It has no other knowledge of you or your work.
• The model keeps nothing between calls. A chat is a transcript that is read again in full each turn.
• Fitting in the window is not the same as being attended to. Material in the middle of a long input is used least.
• Long sessions degrade well before the window is full. Start again with a short handover note.
• Adding more is the usual instinct and often the wrong one. Remove what the task does not need.
• Put sources first and the question last, give reasons with rules, and show an example.

If you want the full version, The Context Window: What a model can see, and how to choose it covers system prompts, memory, retrieval, tool results, compaction, caching and multi-agent work, in 100 short chapters with a labelled diagram in each. It is a 314-page PDF and costs £9 on Gumroad: https://matsiems.com/books/the-context-window.