AI Agents, today, have pretty limited context windows; even when they include a million tokens or more, they feel very forgetful after a compaction. They also don’t continuously learn from experience in mainstream harnesses, and generally have to re-obtain context every time you open a new session. As amazing as they are, this all makes for a fairly janky experience.

While Engram is not the only or maybe even the best system out there, it embodies the ideas laid out here and might be worth a look. Full disclosure: it’s my project, and I use it as a daily driver, so I am about as biased as academic integrity will allow.

The Baseline: Static, Homogeneous Memory

I think static memory for agents is what we are most familiar with. When you ask an agent to remember something, typical baseline behavior is to save it in a markdown file somewhere that will get read (or at least noticed) each new session.

How well these files “stick” varies pretty wildly based on the harness being used, and this shows up pretty consistently after a compaction. My experience with Claude, for example, is that CLAUDE.md is read at the beginning of each session, including after compactions, and carries a lot of behavioral weight. Other files are consulted but carry less behavioral weight or don’t survive compaction.

Some files are read in full every session, others are only read “on demand”. Shifting memory around to keep from burning a pile of tokens in your context window at every restart and after every compaction is something of an art (bias: an argument for using a memory tool that transparently handles that).

By themselves, the standard files don’t do a great job of distinguishing between things like short-term, long-term, and invariant memory. It’s all treated as somewhat invariant by default. Agents see something written down, and that’s just the way of things. Change it up in the session, and it only lasts for that session unless you memorialize it, and then that is the new way of things.

It’s all fairly uniform and not very dynamic. It is, as some researchers have pointed out, more like memo than memory.

A Side Quest through Human Memory Layers

Humans have layered memory. The most familiar layers are “working”, “short-term”, and “long-term”, and definitions can overlap. There is also a common informal notion of “core memories”, experiences that affect how we view the world and react to various situations. These are not the only ways to slice up the idea of memory, but they are very useful, and getting an agent to understand the difference can fundamentally change its effectiveness and the amount of context needed to perform tasks.

Duration

Viewed through my agentic lens, the concepts of long-term, short-term, and working memory are not as much about duration as they are about purpose. Duration is part of it, of course: working memory doesn’t stick around very long at all, short-term memory sticks around just long enough that you remember forgetting something, and long-term memory is supposed to be approximately forever, though it’s never there during finals week.

Though duration differences are real, duration itself seems like it could be an evolutionary accident of sorts, or maybe just a necessary optimization for our biological limitations. Ultimately what matters is that we can file things away, retrieve them when needed, learn from them, and get enhanced function by doing so.

Context

Memory is contextual to some extent, across multiple systems. If you have ever smelled something and it has triggered a memory, that’s one thing we could call “contextual”: the memory surfaces with a particular stimulus.

Humans navigate this pretty effortlessly, for the most part. We remember different rules for different situations, like parties. We remember different behaviors on different days, like trash day. We remember our anniversary every day except on the actual anniversary. You know, contextual.

Context is pretty multi-granular, too. Time of day can form a context, as can day of week or time of year. Context is something we’ll want to keep in mind for agents.

Utility

For agents to be useful, we don’t need to really mirror human experience perfectly, but it is helpful to be inspired by it; it’s a pretty well evolved system. As inspiration, we can focus on a rough interpretation of different memory layers and context triggers. I’m not an expert in human memory, except my own, I think. I can’t always remember. What I am becoming rather expert at, however, is knowing whether agent memory is working for me, so we’ll view everything through that lens from here on out.

Human memory is interesting and serves as the seed idea, but that’s as far as it takes us, here.

Useful Layers and Contexts

I’ve converged on the following mapping in my agentic memory experience, obtained while building and constantly using Engram:

• Invariant Memory: personality, global preferences, persistent motivating philosophy and principles. Importantly, includes meta information on how to handle memory.
• Long-Term Memory: context-specific things that evolve over time, but usually not drastically and not necessarily often (e.g., project design decisions), includes skills.
• Short-Term Memory: to-do lists, knowledge transfer items, things that can identify their own retirement triggers.
• Working Memory: what’s in the session context.

Working memory is good, we have a handle on that. The rest are pretty poorly represented by default in all the agents I routinely work with.

Contexts are quite a bit simpler:

• Global Context: sits in my home directory, available all the time from anywhere I work.
• Local Context: sits in a folder, usually a git repository, only drawn on in that context.

There are likely other ways we could think of dividing contexts up, but this serves quite well for now.

Let’s dig into some of these layers individually.

Invariant Memory

Personality is a canonical example of something people mean when they talk about the impact of core memories. Our personalities are, of course, a blend of inherited and acquired traits, and at least some of the acquired parts come from core experiences that shape us. Sometimes memories are so deep that we aren’t always aware of them, and therapists are typically trained to help connect them to problems we want to solve. Day to day, core memories are usually invisible.

“Core” is not really a memory layer, but an informal abstract term, so for agents I call it “invariant” memory.

I give my coding agents personality, and that goes into the compaction-surviving area of whatever agent I’m using. There is a whole “context canary” idea that makes it more useful than just “my agent knows I like Star Wars references,” though I would argue that having a little joy in your work is part of being effective at it.

The idea of a context canary is just something that you notice before things start to go off the rails. If personality is read into the session at the beginning, and you are able to rely on your evolved ability to catch personality shifts, then personality becomes a good indicator of session context dilution. In other words, if your agent starts to feel off, it’s time to reset: things are about to get wild.

Invariant memory, since agents are still used mostly as individual aids, can also include personal preferences. These are what I consider to be somewhat invariant. “I don’t like history lessons,” “I prefer stories over bullets,” “Please for the love of all that is holy don’t say the words ’load bearing’.” Stuff like that.

These memories are always there in the background, but not explicitly brought up, usually. They just affect behavior and give you a sense of familiarity with every agent interaction, regardless of context.

Crucially, invariant memory includes information about how to manipulate memory over time, which is addressed in its own section below.

Long-Term Memory

Long-term memories are durable, but under pressure to change all the time as circumstances change. A real example for me is my daily work routine. That is something that sticks around for a while, but it is always evolving. The primary principle here is that it stays put unless I explicitly want to change it.

I usually use these for project-level concerns. Design decisions and motivating principles go here. Correcting agents at persistently confusing areas is done here. Frankly, anything that isn’t invariant or short-term goes here. Long-term memory is the default place to put things.

Perhaps most importantly, long-term memory is where skills go.

Skills

Long-term memory can feel something like a default grab-bag, and it’s probably the closest analogy to the traditional markdown-file approach that most agent harnesses use: stuff goes there by default.

But skills are a special and important case. Harness peddlers would have you believe that skills are some special, magical thing, but really they are just a combination of prompts, tools, and sometimes some otherwise-unavailable harness behavior like spawning subagents that you can talk to mid-stream.

That last part is tough to replicate across agent types because that’s where companies try to get market differentiation nailed down in the user experience, but you can do a lot without it, and often you can trigger those internal behaviors by just asking for them, so a well-crafted prompt is often plenty for our needs.

Let’s get specific. A skill is a set of behaviors triggered on demand. You can think of it as a behavioral template of sorts: if activation A, then behavior B, adjusted for context C. Before AI, this would have been a pretty difficult thing to achieve in a general way, involving a long list of templates and vocabulary that can meaningfully fit into slots. Basically a huge mad lib. That’s how Google Assistant worked forever. People thought, at first, that it was smarter than that, but a long list of templates was a core feature. I worked with it when working at Google and experienced it firsthand. Don’t work at a sausage factory if you like sausage.

Concretely, then, a skill memory consists of three basic components:

• A trigger: when does this skill apply?
• A behavior: what is it intended to do?
• A toolset: what must it use to do it?

It’s just information. If you tell an agent “when asked to cut a release, check for an updated CHANGELOG, ask the user whether they want to use SemVer semantics or violate backward compatibility this time, and use git tags in accordance with historical behavior in this repo”, you just defined a skill.

Skills are often better when they are more detailed than that example, but they are really just prompts with these essential elements.

When you understand that about skills, you can see how they are really just lightly structured long-term memories, and that understanding gives you the power to make your own without being beholden to anyone else’s opinionated tool. You should still use mine, of course, because it’s great, but I’m also going to show you how to make your own. Knowledge is power, and I’m only biased because I’m happy. You can be happy with something else if you want.

Short-Term Memory

Short-term memory is one of the more frustrating human experiences, because most of us want everything to be long-term memory, and it just doesn’t always transfer when we want it to, particularly if we are sleep-deprived.

In any case, short-term memory in an agent context is, as per the theme here, not so much about duration as it is about purpose. Why do we need short-term memories if we already have long-term? Isn’t it just a label?

It is, but it’s a very useful label. I use short-term memory as a way of keeping track of present concerns and agent handoffs. Both of these share a common principle: if you can easily identify a retirement trigger, it’s a short-term memory.

Days-long to-do lists are a great example of this. I am in the middle of a big code change, and I need to track the steps to take to get through it. When I am done with those steps, I don’t need to remember them anymore.

You can use all kinds of tools to accomplish the to-do list idea, and a very common and useful tool is your ticket system, if you use one. Trello, Linear, Jira, etc. can be used to track a “definition of done” for a project, and that is usually a glorified checklist. It is by nature short-term: tickets are made, work is done, and tickets are marked complete and taken off your radar.

If you could put it into a ticket and then forget all about it when the ticket is completed, it’s a short-term memory.

Short-term memory is also great for handing work off between agents. I like to have a single session per work item, and I usually have 10 or more of them open at any given time (incidentally, it’s amazing how evergreen tools like ssh, tmux, and vim have remained for code work, even in the new agentic world). I frequently need to branch off into new work while continuing what I’m doing. No problem. Store a short-term memory with needed context to branch out, start a new session in a new worktree, and read it there.

People have started referring to this as forking a session, inspired by forking Unix child processes, and that’s an apt metaphor. Some harnesses support something like this directly, but none of them fit my desired workflow. All of them create way more context than I typically need, too, so the fork burns a new pile of tokens as soon as you take a new turn in the new session.

I prefer just to summarize things into short-term memory, start up a new tmux session, and load that minimal context to start work there.

True Memory, or Just a Memo?

I’ve been talking about all of these memory layers as though we were really discussing memory. But memory is not just storage. Going back to human inspiration, memory is actually a creative, productive act every time we recall something. We don’t store and then merely retrieve, we also recreate. That explains a lot of differences you might have with how you or your friend remember something. Different key elements got stored, and your recreation of the details is going to be unreliable by default. It will be reasonably reliable, hopefully, but not perfectly reliable. Very few people have perfect recall of anything.

What makes something memory instead of a memo? There has been an argument made that it’s learning that makes something a true memory system, not just recording and retrieval. Along with those ideas, suggestions have been made that we might need more ML in our memory systems, that we need to be careful if we do that because it’s yet another opaque thing with unknown hallucination and security implications, but that we’re stuck with significant limitations until then.

I think there is something to the idea that memory needs to have a learning component to be more useful, but I also think that it’s much less complicated than adjusting weights over time and hoping that you didn’t create an exploit. I think it’s actually very simple: useful agent memory uses selective, human-guided consolidation and pruning.

Let me give you an example.

We use Linear for our ticketing at work. What I really mean, of course, is that agents use Linear for ticketing. I hardly ever interact with it directly anymore.

For a while there was a strong preference for us to make our own titles and our own introductory paragraphs in every ticket, human-generated, because the agents were producing jargon-filled opaque stuff in there that only agents could really understand, and that was a problem for many reasons. But it was also a problem for people to have to go and edit those default agent-produced things, because it was opaque to the person in the thick of it, too. My solution: instruct agents on what a better title/description looks like and see if we can uplift everyone in the company by making their agents smarter about this one thing.

That worked at first. Things got better, and at the very least, easier for a human to reword for clarity. I had a rule like this:

Linear ticket titles should have a succinct problem statement focused on “what”. Introductory body should have what, why, and how in 3 sentences or fewer.

At the time, that fit our pattern very well. Tickets were mostly problems to be tackled. But tickets aren’t only used for that. They are also used for defining something that must be done. One could argue that this is isomorphic to a problem to be solved: the problem is just “this thing isn’t done”, but that’s super awkward for someone to parse. It’s better to phrase true prescriptions as imperatives.

And that was the issue. Problem statements are often declarative statements (“just sayin’, the database is broken” if you’re passive-aggressive) and describe undesirable state, while work statements are often imperative. An additional rule was formed:

Linear work tickets are to be phrased as imperatives, focused on “what to do”. The introductory body should refer to what “done” looks like, briefly, but be sure not to bury the lede. Favor clarity over brevity.

My agents got all kinds of confused about this. To me it was pretty clear, and I could hold the cases in my mind pretty well. When this badly worded stuff resulted in apparent contradiction, I refined it. The above rules grew to twice their size. Then three times. Then ten times, each successive refinement getting more lawyerly.

This was not memory, this was litigation.

Ultimately, I stepped back and realized that what I wanted was a ticket title that told me what need it was satisfying. That would allow me, at a glance, to do rough prioritization without digging into every ticket’s description. It’s not perfect, but it’s better, and it’s shorter than the spiral we were in. Hundreds of words and two competing memories were immediately consolidated and pruned to this:

A ticket exists to serve a need. Title it after that need, because the need is what survives everything else being re-decided.

If the approach changed, would this ticket still exist?

If it would, a need outlived the approach, and the title has to name that need. If it would not, the approach was the need and naming it is right.

The agent was no longer confused. We got there through a process of refinement, starting with an initial idea about evocative prioritization titles and converging on a principle that, even though it used plenty of abstract language, still made for better behavior.

Well, that’s neat, but what just happened?

• The agent got confused, and when prompted, showed me the contradiction.
• I looked it over and realized there was a simplifying principle.
• We consolidated the memories into one simpler memory and pruned the rest.

When looking to make agentic memory useful, and to incorporate learning, we can distill the above into this:

Useful agent memory combines storage, context-sensitive retrieval, consolidation, and pruning, all in service of taking better future actions.

In humans this happens with deeply embedded mechanisms that we don’t have full control over. These mechanisms actually span all of our layers of consciousness and instinct, so we have some control over them, but there are fundamental lizard-brain parts of these mechanisms that we do not get direct access to.

With agents, it’s different. Currently I don’t know of any generalized in-built mechanism for running a self-evaluative feedback loop in popular harnesses. And frankly, I’m not sure you could do that meaningfully, because the utility of agent memory is not a single function, it’s highly context-dependent, and there is no context more important than the needs of the person the agent is working with.

How do we get that consolidation feedback loop working, then? We use agent memory coupled with periodic human judgment: it’s a partnership. We’re taking advantage of the fact that the overall human-agent system has the capabilities that the agent itself might lack. The agent has a really great memory, the human has really great intuitive ideas about how to manipulate it. That’s the reason we don’t need some specially tuned machine learning system layered over memos. We just need to leverage what we have: memos and human judgment.

Importantly, and elegantly: the memory learning procedure is itself a memory, a stored skill. The memory system applies to itself.

My favorite memory system is precisely that kind of self-referential:

• Trigger: a long-term memory is growing over time due to perceived contradictions.
• Behavior: pause, work with the human to distill it into a unifying principle. Suggest candidates if obvious.
• Toolset: the memory tool itself.

This works astoundingly well, and makes it feel like the agent is learning over time. In fact, the agent’s memory system is learning over time, because its memories are changing over time. What’s different is the trajectory of those changes: when consolidation succeeds, the utility trajectory tends upward. Can it make mistakes and overgeneralize? Sure. And so can you, dear human. What do you do when you are operating on a too-general principle? You add an exception and move on. The same mechanism works for agents.

Agent memory improves through an ongoing consolidation process in partnership with a human.

I’m not under any illusions that this is the only way to do it, nor that there won’t be any future innovations that render this idea obsolete, but it certainly works, and it is certainly easy to do, and it does not require complicated machinery. I’m quite happy with it.

So, what makes the difference between memo and memory? A feedback loop that generalizes what it knows. The feedback loop invokes human help to generalize by directing pruning and consolidation. And memory is all you need because this is literally just a skill in memory.

How to Make Your Own

This isn’t like cryptography, where the answer is simply, “Don’t.” It is nice to just be able to use a tool for this, and obviously I have my own favorite. Let’s say you wanted to build your own, using just markdown. You could think about memories as things stored in a special directory called, let’s say, .memory. We’ll go with that.

Context Sensitivity

Invariant memories are global-only. To bootstrap this, I like to do the following:

• Tell the agent about yourself, things you like, things that you find funny, how you like to work, etc.
• Ask the agent to refine its understanding of you in a few turns.
• Have the agent pick a name for itself based on how you want to work with it.
• Store that in, say, .memory/invariant.md or better yet, in whatever the main agent settings file is. That way it gets strong compaction-surviving behavioral weight.

Short-term and long-term memory, including skills, can be in any context, global or local.

Index vs. Bodies

You don’t want your agent pulling everything in all the time. With even a modest amount of useful memory, that can be tens of thousands of tokens in your preciously small context window, every single session.

One way to handle this is to make use of hierarchy in the file system. Consider this:

• .memory/
  • invariant.md
  • long.md
  • short.md
  • long.d/
  • short.d/

In, say, long.md, you would have small entries: a short slug and a one-line description. For skills, you would also have the trigger here. In long.d/, which is a folder, you would have a filename that matches the slug, and a longer detailed description.

Let’s say you want a code review skill separate from the provider-built one. You’d store that in long-term memory.

In long.md:

# my-code-review

title: how to run a code review as a merge gate
trigger: run when ready to move a pull request from draft to ready-for-review

And then in a file named long.d/my-code-review.md:

Use the repository tool run-code-review.sh. Watch its output and the time. It
should never take longer than ten minutes. Ask the user if they want to
continue if it does. You will see a heartbeat on stderr every 30 seconds if
things are working properly. Note that the user might need to go to another
terminal to log in to another agent provider depending on their preferred
harness.

When the agent loads long-term memories on startup, it only pulls in the index, then it knows when to expand it based on the title and (if present) the trigger.

General Agent Instructions

Once you feel good about your file and directory layout, it’s all just agent instructions after that. Here there is some real art, so you would need to tweak this as you find it not working for you.

You want to inject this system into the beginning of a session. That means putting instructions somewhere that will survive a compaction, if you can.

Different providers call their global settings files different things, and I usually have to put instructions into all of them if I want to have all of my agent types share memory. I don’t remember details like this, so I ask my agent to store the text where it can find it.

These prompts can get a bit on the long side, so I’ll just give a gist here:

• Directory Specification: tell the agent about your directory structure. Specify that invariants are always global, but that all other memories favor local first and fall back to global.
• Memory Action Triggers: give the agent as many pointers as you like to get it to trigger memory actions on its own. Minimally, tell it what “store this in long-term memory”, “save this in short-term”, etc., mean to you. Indicate that you want to override its normal memory system in favor of this one.
• Short-Term vs. Long-Term: Explain how to differentiate short-term memory: if you can identify a trigger that would cause the memory to be deleted (a definition of done, as one example), it’s short-term. Otherwise it’s long-term.
• Learning Over Time: tell it about principles of learning memory, things like how to notice, reconcile, and consolidate conflicting memories in partnership with you. Usually this involves asking questions, then rewriting the index and combining or deleting files.

And that is basically it. Memory is a series of prompts and a structured mechanism for getting at them, and this includes skills. You can even do things like instruct your agent to notice when it is doing the same thing across sessions and to write a new tool for that repetitive work (note that I’ve had very mixed results with this, but it can work to an extent). Now your memory contains tools, as well.

Finally, if you don’t want to be just managing prompts and fiddling with files all the time, and you especially don’t want your agent to rely on complicated instructions, you can build all of this into a piece of software that your agent helps you write. Then memory really is just a tool with operations like memory listing, injection, search, file manipulation, etc. That helps quite a bit with local memory, because without a tool, you might get weary of repeated repository-specific setup.

That’s basically what Engram is: a tool to do all of this. It uses SQLite instead of a collection of markdown files, and it knows about different agent idiosyncrasies and file locations and limitations, but it all boils down to what’s laid out in this section.

Memory is All You Need

Skills, personality, preferences, project design, current state, and learning can all be captured in memory that evolves over time. When we step back and remember that agents are typically working with a human, and we leverage that as a partnership, we can get a learning memory system without having to update model weights. And the system applies to itself, as well, making this entire process just another skill sitting in memory.

Memory really is all you need.