Automation and AI

Context Window

Also called context length

The amount of text an AI model can hold in mind at once, including your instructions, the conversation and attached files.

Quick facts: Context Window

Category
Automation and AI
Also called
context length
Level
Advanced
Affects
Answer quality, cost per run, response speed
Where to see it
Model provider dashboards, chat assistants, assistant builders
In this article4
  1. What the context window holds
  2. Why the context window matters
  3. Where the context window goes wrong
  4. How to act on it

What the context window holds

Everything the model is looking at when it writes a reply sits in one space: your standing instructions, the conversation so far, any document you pasted or attached, anything a tool fetched on your behalf, and the answer being produced. It is measured in tokens, which are chunks of words rather than words themselves, so a long document takes up more room than its page count suggests.

The window is a hard edge, not a soft one. Once it is full, something has to leave. Chat tools handle that by dropping or summarising the oldest turns, which is why a long conversation slowly forgets the rules you set at the beginning while remembering the last thing you said perfectly well.

There is a quieter effect too. Even inside the limit, recall is not uniform. Material at the start and the end of a very long input is used more reliably than material buried in the middle, so where you put an instruction changes how well it is followed.

Why the context window matters

It explains most of the odd behaviour people blame on the model being unreliable. An assistant that ignored your tone-of-voice rules after a long session did not decide to; the rules fell out of the window. A summary that missed a clause in the middle of a contract did not skim it lazily; it was competing with everything else you pasted.

It also has a price. Longer inputs cost more to process and take longer to answer, and a workflow that pastes the same large brief into every run pays that cost on every run. Deciding what the model actually needs to see is a cost decision as much as a quality one.

Where the context window goes wrong

The commonest error is treating the window as memory. It is not. Close the chat and nothing is retained unless the product deliberately stores it, and the next session starts blank however much you explained last time.

The second is stuffing. Pasting an entire site, a year of transcripts or every product sheet into one prompt feels thorough, but it dilutes the part that mattered and makes the answer vaguer. Selecting the relevant passages beats sending everything.

How to act on it

Put the instruction you care about most where it will be read: at the end of a long input, restated briefly, rather than buried at the top under context. Keep a written brief you can paste fresh into a new session instead of relying on a thread that has drifted, and start a new session whenever the assistant begins contradicting itself.

When the source material is genuinely large, retrieval is the answer rather than a bigger window. Retrieval-augmented generation fetches only the passages that relate to the question, which is cheaper, faster and usually more accurate. Understanding how the underlying model reads its input also makes prompt structure feel less like superstition and more like layout.

Do and do not

Do

  • Restate the key instruction near the end of long inputs
  • Retrieve relevant passages instead of pasting whole documents
  • Start a fresh session when a thread has drifted

Do not

  • Paste an entire website into a single prompt
  • Assume the model remembers previous conversations
  • Buy the largest window before testing a smaller one

Questions people ask about this

Does a bigger context window always give better answers?

No. A larger window lets you send more material, but it does not make the model weigh that material better, and long inputs cost more and take longer to process. Answers often improve when you send less and choose it more carefully. Test the smaller, cheaper option on your real task before paying for capacity you may not need.

Why does the assistant forget my instructions in a long chat?

Because the oldest part of the conversation is dropped or compressed once the window fills, and your opening instructions are usually the oldest thing in it. The fix is not to repeat yourself endlessly but to keep the brief in a file, start a fresh session when a thread has drifted, and restate the essential rules near your latest message.

Is the context window the same as the model's training data?

No, and confusing the two causes real mistakes. Training data is what the model learned from before it was released and cannot be changed by you. The context window is the temporary space holding what you supply right now. Anything you need the model to be accurate about should be supplied in the window, not assumed from training.

Related terms

Found this useful?

Share it, or ask an AI to summarise it

Back to the glossary

Knowing the term is the easy part

Applying it to your own site and budget is the work. Book a call and I will tell you what actually applies to you.