AI Memory: Learning from Chatgpt's Memory
Using ChatGPT's newly published 'dreaming' system as a reference to reverse-engineer how memory in AI systems actually works and the challenges in building memory systems.
If you have interacted with any AI assistant and have follow up interactions in a different session, you’d see it behave in one of the 2 ways:
It can either have a “50 First Dates” moment, and forget you ever had a conversation with it.
It remembers what you talked about, and the responses feels customised to you.
The behaviour difference is based on if the AI assistant has “memory” available or not.
What is Memory?
Memory is an overloaded term - I've noticed people often collapse several genuinely different mechanisms under this one umbrella word. Before going further, it's worth separating out the classifications:
Parametric Memory: This is the memory that is baked into the model at its training time, encoded in its weights. This memory comes from the corpus of training text and the model weights. It's frozen the moment training ends and it's identical for every user talking to it.
Context Window/Short Term Memory: This is the memory, that the model receives as part of the current conversation the user is having. This state is short lived, as the current conversation state isn’t persisted across sessions. So a preference you remember in this session is not something that will be leveraged in the next one.
Long Term Memory: This is the piece we’ll be diving into in this blog. This is an external, mutable, persistent store, which can store facts/preferences associated with the user.
Why do we need an external memory?
Inference is stateless by construction — the same weights, run on the same input tokens, produce the same output every time, with nothing carried over. That holds no matter how large you make the context window: a bigger window buys you more short-term memory within a single session, not persistence across sessions.
An external memory system has two critical jobs:
Identify what is worth storing. Not every sentence in the conversation can be used to extract meaningful & relevant information.
Injecting the relevant stored preferences into the prompt such that the LLM can provide more personalised responses.
How Memory has evolved?
This blog from OpenAI talks about how their Memory Architecture has evolved and this is a common evolution trend across the industry.
Explicit Fact Saving: Most initial versions of memory implementations was simple. If you the user asks the agent to “remember” something, the agent will store this into its memory as is and inject it into your prompt for future conversations.
Dreaming": In case of “Dreaming”, there is a background process that reads conversation history of the user and curates memories automatically, without you asking it to remember anything.
Memory Architecture
A lot of people I’ve talked to look at Memory as a storage problem.
The challenge with Memory is converting an ever-growing, contradiction-riddled stream of raw conversation into a compact, current state that's cheap to inject into prompt, and doesn't degrade the agent's own performance in the process.
Remember: A bad memory could be worse than no memory.
Architecture Choice 1: Brute Force
The brute force approach to memory is to keep every message a user has ever sent, stored in the external memory and at inference time stuff the whole history into the context window.
As with all brute force solutions, this doesn’t scale. If a user sends 20 messages each day, with each message having 30 tokens on avg, we’re looking at ~220,000 tokens after a year, just from “memory”. Also, this kind of system would also have a huge impact on cost, which makes it a non-starter at scale.
Architecture Choice 2: Top K Memories
In this approach, we still store every message from the user into our external memory(Vector Store in this case), but at inference time, we fetch the top-k most similar ones to the user query, with vector search.
This prevents bloating up context window, keeps the cost down by injecting only relevant memories, if found.
However, the problem in this approach is in ensuring quality of the memories being retrieved. You could have stale conversations that might be fetched as relevant memories.
Architecture Choice 3: Extract, Score and Consolidate
This is a common architectural pattern that I see today across all memory systems. The generation of memories happens across 2 steps:
Extract: Every conversational turn is a candidate source of facts/preferences. An extraction step (an LLM call) proposes short, atomic statements — "user is vegetarian," "user likes photography" — rather than storing the raw turn.
This is exactly the ADD/UPDATE/DELETE/NOOP pattern Mem0 uses in production: a new candidate fact is compared against similar existing memories, and the model decides whether it's genuinely new, an update to something existing, a contradiction that should delete the old fact, or a no-op duplicate.
Consolidate/Dream: This is a background job that re-derives the memory state from the accumulated raw facts. It groups facts by topic/entity, keeps the freshest statement, merges near-duplicates, and expires anything that's become time-bound and past its expiry.
This step is what ensures that your memory is up-to date, because it looks at all facts/preferences for a particular topic, unlike consolidate which operates at a conversation turn level and never has a holistic picture.
The other change in this approach is how the retrieval happens:
Scored Retrieval: At query time you don't want "all memories similar to this prompt". You want some that are similar and relevant to the user query. We decide this with the help of a retrieval score:
The retrieval score is a weighted combination of:
Recency (exponential decay since last access)
Relevance (embedding cosine similarity to the query)
Importance (a self-assessed weight, usually itself produced by an LLM call at write time).
Challenges with Memory
Bad Memory/Non Relevant Memory: Just because you have generated memory from conversations, doesn’t automatically imply it will improve the user experience. Injecting memory into prompt can have the following side-effects if not handled correctly:
Lost in the Middle: Injecting a large amount of memories in the context could lead to degraded performance of the agent, as the LLM tends to focus on items sitting at the beginning/end of the context window. Memory is generally in the middle in case you have a long system prompt.
Context Poisoning: A stale/outdated/wrong memory could be saved and referenced/injected in the context, leading to bad responses.
Context Confusion: While your memory might be correct, it could be that it’s irrelevant to the task at hand. This could also lead to degraded response quality.
Context Conflict: Your conversation history might be in direct conflict with the stored memory(even if the memory is correct), leading to a poorer response from the LLM.
Consolidation Latency: Since dreaming runs as a periodic offline job, a change/correction in memory will not be consolidated till the next run of Consolidation.
Memory Deletion Challenges: Since memories have been generated as a consolidation over multiple conversation turns, deleting a memory is no longer just a matter of deleting the memory itself, but also, all the original source conversations that was used in creating that memory.
Summary
Hope you found this blog useful. We’ve dived into some of the methods of memory generation, either through explicit instructions or “dreaming”. We looked at how you can build your own memory system, and what are the caveats that you need to focus on.
👉 Connect with me here: Pratik Pandey on LinkedIn
References: