Short-term vs long-term memory in RAG: your system has RAM. Does it have a disk?

🧠 Your RAG system has RAM. Does it have a disk?

The industry found its phrase this week: context engineering — Tobi Lütke framing it as providing all the context for a task to be plausibly solvable, Andrej Karpathy as the delicate art of filling the context window with just the right information for the next step. The analogy travelling with it is the useful part: the model is the CPU, the context window is RAM, and we are the operating system deciding what gets loaded.

Every operating systems student knows what comes next in that analogy, and it is the part getting less attention: RAM is not storage. A system with only a context window has working memory and no memory. Having built RAG systems on both sides of this line, the short-term/long-term distinction is, I would argue, the single most consequential design decision in the category right now:

Short-term memory is the context window — the current conversation, this turn's retrieved chunks, tool outputs. It is fast, expensive, and gone when the session ends. And it is smaller than it looks: windows have grown enormously, but models demonstrably degrade as you fill them — retrieval quality decays, the middle gets ignored. A bigger window is a bigger desk, not a better filing system. Stuffing it is not remembering; it is hoarding.

Long-term memory is everything that survives the session — and it is not one thing but three, each wanting different storage:

Semantic memory: what the system knows — the document index. Classic RAG. Most teams stop here.
Entity memory: stable facts worth persisting — this account's structure, this user's role and preferences, decisions already made. Structured, small, high-value.
Episodic memory: what has happened — past interactions, distilled. Not transcripts; summaries. Storing raw history and re-retrieving it is how systems get slower and dumber simultaneously.

Why the distinction matters more than it looks:

1. The write path is the neglected half of the design. Everyone engineers retrieval — what to load into RAM. Almost nobody engineers persistence: what deserves to be written to disk, when to distil, what to forget. A system that remembers everything drowns in itself; one that remembers nothing makes every conversation a first date. Deciding what is worth remembering is a product decision wearing an engineering costume.

2. Long-term memory is a data asset, and inherits every obligation of one. The moment your system persists things about users and accounts, you have created a store containing preferences, possibly PII, possibly things people told the system in passing. It needs what any such store needs: access control, retention policy, and a working answer to "forget that." Teams that discover this after launch discover it in a compliance review.

3. Memory is where injection goes to become permanent. A poisoned document influencing one answer is an incident; a poisoned instruction written into long-term memory recurs in every future session. Whatever inspects your inputs must also inspect your writes — the disk deserves more suspicion than the RAM, because it outlives the request.

4. Trust is asymmetric. A system that forgets something important is mildly annoying. A system that remembers something the user did not expect it to keep is unsettling in a way that kills adoption. Being visibly selective about memory — and letting users see and edit what is held — is not a nice-to-have; it is what makes persistent memory socially acceptable inside an enterprise.

The one-line summary for leadership reviews: context engineering decides what the system is thinking about right now; memory architecture decides what kind of colleague it is over time. The first is this week's skill. The second is next year's differentiator.

A question for any team building agents or assistants today: can you state, in one page, what your system remembers, where that lives, how long it is kept, and who can make it forget? If that page does not exist, your memory architecture does — you just have not met it yet.

Reading: https://simonwillison.net/2025/jun/27/context-engineering/