|
[Webinar] How to stop babysitting your agents (Sponsored)
 |
Agents can generate code. Getting it right for your system, team conventions, and past decisions is the hard part. You end up wasting time and tokens in the correction loops.
More MCPs, rules, and bigger context windows give agents access to information, but not understanding. The teams pulling ahead have a context layer to give agents exactly what they need for the task at hand.
Join us for a FREE webinar on Sep 23 to see:
Where teams get stuck on the AI maturity curve and why common fixes fall short
How a context layer solves for quality, efficiency, and cost
Live demo: the same coding task with and without a context layer
If you want to maximize the value you get from AI agents, this one is worth your time.
An LLM can analyze a 100-page document, follow a complicated programming discussion, and refer back to something discussed several messages ago. However, as soon as we open a new chat where the earlier conversation is not present, it immediately forgets everything. It appears that LLMs have the memory of a goldfish.
This observation is indeed true. LLMs usually have no personal or persistent memory of previous interactions. So, how is it able to refer to what we’ve said in the past?
In truth, an LLM doesn’t remember a conversation like human beings do. However, it receives information about the conversation so far with each new message that we send to the LLM. All of this is handled by the application that is built around the model and not by the model itself. For example, a chat application might store messages, maintain summaries of earlier discussions, retrieve relevant memories, and maintain a user profile. It can then place some of that information in front of the model as needed. From the user’s perspective, the model appears to remember. But technically, the surrounding application is doing most of the remembering.
This difference between the model and the application around the model is the key to understanding LLM memory.
This design has important consequences. As a conversation grows, the application must keep processing more text, thereby increasing cost and latency. Eventually, the conversation ends up turning too large to fit inside the model’s context window. At that point, older information must be removed, summarized, or stored somewhere else.
In this article, we will learn how LLMs handle memory so that they are useful to end users in performing complex tasks that require conversation and holding context.
 |
What Does “memory” Mean for an LLM?
In the context of LLMs, the word “memory” is used for several different things that should not be confused. Let’s look at each type in detail.
Trained Memory
During training, an LLM learns patterns from enormous amounts of data. These patterns are encoded in billions of numerical values called parameters or weights.
This is why a model can explain JavaScript, recognize a common historical event, or write an email without receiving that knowledge in the current prompt.
But this is not personal memory. If a user tells the model, “My preferred programming language is TypeScript,” a normal API response does not rewrite the model’s weights. The base model does not permanently learn that fact from the conversation.
Working Memory
The model’s temporary working memory is its context window. This contains everything the model can consider while generating its current response.
 |
It can include:
System and developer instructions
The current user message
Previous messages in the conversation
Retrieved documents
Tool descriptions and tool results
Saved user preferences
Summaries of older conversations