You paste a 60-page contract into a chatbot, ask about the termination clause and get a sensible answer. An hour and many messages later, you ask about a payment term from page 3 and the reply contradicts what the contract says. Nothing crashed. The tool simply reached the edge of what it can keep in view, or it was still holding the text but no longer using it well. Both problems come down to the context window.

What the context window holds

Anthropic’s documentation on context windows defines the term as all the text a language model can reference when it generates a response, including the response itself. It calls the window the model’s “working memory” and separates it from the much larger body of data the model was trained on. Google’s long context guide uses the same comparison: the window works like short-term memory, with a limit on how much it can hold at once.

The practical point is that a model does not remember your conversation the way a person does. Each time you send a message, the application sends the model a bundle of text, and the model answers from that bundle. According to Anthropic’s documentation, the bundle includes:

  • the system instructions the application adds before your first message;
  • every earlier message and reply in the conversation;
  • files, images and tool results that were added along the way;
  • the descriptions of any tools the model can call;
  • the answer being written, including any visible or hidden reasoning steps.

All of that competes for the same space. A long document you attached early on still takes up room in every later turn.

Tokens, not words

Window sizes are measured in tokens, the units a model splits text into before processing it. OpenAI’s help article What are tokens and how to count them gives rules of thumb for English: one token is about four characters or three quarters of a word, so 100 tokens come to roughly 75 words, and about 1,500 words take around 2,048 tokens. The article also notes that tokenization varies by model, so these are estimates rather than exact counts. Code, tables and languages other than English can split into more tokens per word.

Window sizes have grown quickly. Google’s guide says earlier generative models handled about 8,000 tokens at a time, and it gives a sense of scale for a window of 1 million tokens: about 50,000 lines of code or eight average-length English novels. Anthropic’s documentation lists windows of 200,000 tokens and 1 million tokens for different Claude models. Sizes change with each model release, so check the provider’s current documentation for the tool you use.

What happens when the window fills up

There is no single behavior. It depends on how the application is built.

When developers call a model directly, Anthropic’s documentation describes the standard case: each turn adds the new message and reply to the window, and earlier turns are kept in full. A request that does not fit is a problem the developer has to handle, for example by trimming or summarizing. The same page says chat interfaces such as claude.ai can instead manage the window on a rolling “first in, first out” basis, which means the oldest material drops out to make room. Anthropic also describes server-side compaction, where earlier parts of a long conversation are condensed.

OpenAI’s token article lists the usual options when a limit is reached: shorten the prompt, break large text into smaller chunks, or summarize the input before sending it.

For a reader, the consequence is the same in each case. In a long chat, the contract you pasted at the start may have been dropped or replaced by a summary that leaves out the clause you now need. The tool will often answer anyway, which is how a context problem turns into a confident wrong answer. Our explainer on why AI hallucinates covers the guessing behavior behind that.

A bigger window is not the same as better recall

Fitting a document into the window does not guarantee the model will use every part of it. Researchers from Stanford, UC Berkeley and Samaya AI tested this in Lost in the Middle: How Language Models Use Long Contexts. They placed the passage that answered a question at different positions among other documents and measured accuracy. Performance was usually highest when the relevant passage came at the beginning or end of the input and dropped significantly when it sat in the middle, a U-shaped pattern that held even for models built for long contexts. They also found that extended-context versions of a model often performed no better than the standard versions on these tasks.

That study tested models from 2023, and newer models may behave differently. The providers themselves still describe the same direction of effect. Anthropic’s documentation says accuracy and recall degrade as the token count grows, which it calls context rot, and that more context is not automatically better. Its engineering post on context engineering for AI agents says some models degrade more gently than others but that the effect appears across all of them, and describes the result as a gradient rather than a hard cliff.

How to work with the limit

These habits follow from the sources above. None of them is a guarantee, but each reduces the chance that the answer comes from text the model no longer has in view.

  1. Start a new conversation for a new task. Old messages from an unrelated question still take up space and can pull answers off course.
  2. Put the question next to the material. Given the U-shaped results, the start and end of the input are the safest positions. Paste the relevant section and ask your question directly after it, instead of relying on a document attached many turns earlier.
  3. Send the part you need. If only two clauses matter, paste those two clauses. Cutting the input is the first remedy OpenAI lists for token limits.
  4. Ask for the quote. Have the tool quote the passage behind each claim, then find that passage yourself. If it cannot quote it, the text may have fallen out of view.
  5. Re-supply facts after a long session. When a conversation has run on, restate the key figures or re-paste the source rather than assuming the tool still has them.
  6. Be careful with agents. An AI agent that reads email, files or web pages fills its window with whatever it reads. Our guide to giving an AI agent access to your inbox covers why approvals matter when the tool’s view of the situation is incomplete.

If you run models on your own computer, the window also has a hardware cost: a longer context needs more memory. Our guide to running AI on your laptop covers what to check first.