The context window is not memory

Context windows have grown roughly three orders of magnitude in a few years, and each expansion arrives with the suggestion that retrieval is now unnecessary: just put everything in the prompt. It is an appealing idea and it is wrong in a specific, load-bearing way.

A context window is working memory. It is scoped to a single turn, it is paid for on every call, it degrades in the middle, and it is gone when the conversation ends. Calling it memory confuses a buffer with a system. What people actually want when they say memory is something else entirely: a body of knowledge that survives the session, accumulates from experience, and is retrieved selectively when relevant.

Working memory is what the model is holding. Context is what the system knows. Only one of them persists.

The window degrades before it is full

The clearest way to see that the window is a buffer, not memory, is to measure what happens inside it as it fills. The popular test, needle in a haystack, is close to solved and close to useless. In the RULER benchmark (Hsieh et al., 2024), models score nearly perfect on the vanilla needle test and then fall apart on tasks that ask them to actually use the whole window. Of the seventeen models RULER evaluated, all claiming a context of 32K tokens or more, only half held satisfactory performance at 32K.

The gap between the advertised window and the usable one is large and specific. RULER defines a model's effective length as the longest input at which it stays above a fixed quality bar, the score Llama2-7B posts at 4K, which is 85.6%. Measured that way, GPT-4's effective context was 64K against a claimed 128K. Yi-34B's was 32K against a claimed 200K. Several models trained for a million-token window, GradientAI's Llama-3-70B among them, held only 16K, and one, LWM-7B, never cleared the bar at all, falling short at 4K despite a million-token claim.

Size is only half of it. Position matters too. In "Lost in the Middle" (Liu et al., 2024), the same fact was placed at different depths inside a 20-document context and the model was asked to use it. Accuracy traced a U: highest when the fact sat at the very start or the very end, worst in the middle. For GPT-3.5-Turbo the best-to-worst gap ran to more than 20 points, and in the worst case, with the answer buried among distractors mid-context, the model did worse than closed book, that is, worse than being handed no documents at all.

A window you cannot use to the edges is not the size on the label. In Liu et al. (2024), oracle accuracy for GPT-3.5-Turbo, given only the one relevant document, was 88.3%. The same model, handed the same fact buried mid-context, dropped below its own 56.1% closed-book score. Past a point, adding tokens to the prompt bought negative value.

None of this makes long context useless. It makes long context a buffer with a quality gradient, which is a different object than a store you can trust to hold a fact and return it unchanged three sessions later. The advertised number is a ceiling on what fits, not a floor on what works.

Three things that are all called context

Part of the confusion is that one word is doing three jobs.

• The window: the tokens in front of the model right now. Finite, expensive, ephemeral.
• The corpus: the documents, records, and connected systems the agent can reach. Large, durable, mostly irrelevant at any given moment.
• The accumulated record: what this agent has learned from doing the work. Small, high value, and the piece almost nobody builds.

Most stacks build the first two and skip the third. That is why agents that have run a task four hundred times approach the four hundred and first exactly as they approached the first. They have access to everything and have learned nothing.

Retrieval is a judgment problem, not a search problem

Once the corpus is bigger than the window, something has to choose what goes in. That choice is the actual product. Semantic similarity is a reasonable first pass and a poor final answer: the most similar document is often not the most useful one, and the most useful one is frequently the record of what happened last time someone tried this.

Good selection weighs recency, provenance, and outcome, not just closeness. It should be able to prefer a short note that says "this approach failed for this reason" over five pages of documentation that are topically adjacent and practically useless. That preference has to be learned from results, which means it depends on measurement, which means context and proof are the same problem wearing different clothes.

The part that compounds

The reason to care about the third category, the accumulated record, is that it is the only part of the system that gets better on its own. Models improve on someone else's schedule. Prompts improve when a person edits them. But a corpus of what worked, what failed, and under which conditions grows every time the system runs, and it belongs to whoever operated it.

This is also the honest answer to the question of what a company owns when it builds on someone else's model. Not the weights. Not the architecture. What it owns is the accumulated context: the record of its own work, structured well enough to be retrieved. That is the asset, and it is worth being deliberate about where it lives.

How we think about it

We treat context as infrastructure rather than prompt engineering. Memory is a stored, queryable thing with an owner and a scope, not a blob concatenated at the top of a request. Agents write to it deliberately, including recording their own evaluations of how a task went. Retrieval is a first-class operation you can inspect, which means when an agent does something surprising you can ask what it was looking at when it decided.

The window will keep growing, and that is genuinely useful. It just answers a different question than the one most people are asking.