Agent Memory: What Are the Building Blocks of a Memory System?

A new survey paper decomposes agent memory into four modules — representation and storage, extraction, retrieval and routing, and maintenance — and finds that no single architecture wins. Dosu's Taylor Dolezal sits down with Devin Stein and Michael Mangus to walk through the paper, the database analogies that make it click, and why maintenance is the hardest unsolved problem in the space. Paper discussed: Are We Ready for an Agent-Native Memory System? (Zhou et al., 2026). A written companion piece lives on the blog: How to Build Agent Memory.

Transcript

Taylor Dolezal, Head of Open Source, Dosu: Welcome everyone, and welcome to Agent Memory: What are the building blocks of a memory system? My name is Taylor Dolezal, I'm head of open source at Dosu. Excited to jump in with you today. First off, I'm going to throw it to Devin to share a little bit about himself, and then get to the why behind this discussion.

Devin Stein, Founder and CEO, Dosu: Thanks Taylor. Devin Stein, founder and CEO of Dosu. The goal of today's conversation is that there has been a lot of noise and discussion about agent memory, agent knowledge, knowledge infrastructure, company brains. Internally, we've been doing a paper breakdown, an internal process of trying to make sense of all the noise in the space. And we thought that would also be useful for our users, and generally for people interested in this space. So this is hopefully the first of many webinars trying to break down: what do agents know? How should you think about the components of agent memory, what it means, and how your organization should be thinking about it. Michael, you want to do a quick intro as well?

Michael Mangus, Head of Product, Dosu: I'm Michael Mangus, head of product at Dosu. Obviously, as a knowledge infrastructure company, the topics in this paper are very much top of mind for us. It's been really useful for us internally to think about this framework of how you decompose a memory system into its component parts, how you test them in isolation, and then how you understand the way they work together to produce the outcomes our users expect from Dosu. So I'm excited to get into it.

What is agent memory?

Taylor: Let's start with some table setting. Devin, what is agent memory? What do we mean when we're talking about agent memory?

Devin: Good question, and also a hard question, which is kind of the point — it means many things, and there are many aspects to it. At a high level, when people say memory, it's about how agents capture or learn based off their interactions over time. That's maybe the simplest framing. And there has been an evolution in the space. Memory historically used to be about user personalization — how do agents learn about you? But we're now seeing a shift toward more task-oriented memory, where it's about how agents learn how to do things.

Taylor: Within the greater ecosystem, is that mostly what people mean by agent memory? Are there any adjacent topics we won't be talking about today — kind of a "what's not on the sandwich" framing?

Devin: Sometimes people ask, just in terms of terminology, how memory is different from RAG, or whether context engineering is a form of memory. RAG and context engineering are ways of serving information, or figuring out what information the agent sees. But they don't describe how memory is created, stored, written, and managed. That's what we're going to get into today.

Taylor: Last on table setting — this might be a bit of a softball. Michael, do we need agent memory?

Michael: Do you have to have it? No. Do you want to have it? Absolutely. In terms of both improving the efficiency and the quality of what comes out of your agents, and in terms of control and steerability, just having something that persists from session to session really unlocks a lot of capabilities. So what I'd say is: you don't have to have agent memory, but if you don't, you're doing yourself a disservice.

Why this paper

Taylor: Let's dive into this paper we talked about internally at Dosu. It's called "Are We Ready For An Agent-Native Memory System?" Michael, could you talk about why we started with this paper, before we dive into each of the sections?

Michael: Like Devin alluded to, we started doing a journal club internally at Dosu. The big reason for that is there's a lot we can learn from the research being done in the academic community, and even in commercial research labs, with respect to memory. LLMs, and coding agents in particular, are still a very new technology. We want to make sure we're staying on top of the emerging research that's guiding where things are actually getting built in the commercial space — the products getting created around these new areas of research, around ecosystems that didn't really even exist two years ago.

In terms of this paper in particular, the reason it was our first selection for journal club is the typology it lays out. This is in many ways what you'd call a survey paper. It looks at a variety of different memory frameworks, presents a typology for breaking them down into four constituent components, and then describes the different choices all those frameworks are making on those four components. It's just a really good overview to build some scaffolding in your mind — some technologies for thought — when it comes to: how do I understand memory, and what are the tradeoffs and choices I'm making when I design a memory system? That's really where its value comes in for us.

The four building blocks

Taylor: The paper decomposes agent memory into a few different sections. Can you get into some of those?

Michael: The four components are, first, representation and storage — which just means, both in a physical way and in a logical way, how do we store our memory?

Then there's memory extraction, which is: where does my memory come from? For you as a human, your memory comes from your experience of the world. You have senses, they tell you things about the environment you're in, and your memory allows you to capture salient aspects of that environment to retrieve later, to help you solve future problems or interact with your environment in a more efficient, productive way. So memory extraction is the equivalent for agents: how do I take raw data — a Slack conversation, a trace from a previous agent session — and turn it into a memory I can access again in the future?

Then there's retrieval and routing. Again continuing the analogy to human memory: you see an environment or a situation that's familiar to you, and it activates your memories of previous similar situations. How do we give an agent that capability? How do we actually manage the mechanics of things like vector search, graph search, and keyword search to retrieve the relevant memories for a certain problem the agent's working on?

And then the last bucket is maintenance. This one is especially near and dear to our heart at Dosu, because so much of the work we've done so far has been in maintaining documentation. How do we make it so the memory we build stays up to date when we have a new experience of the environment — a new session with an agent, a new message in a Slack thread, whatever? How does that change what we believe about the state of the world, or about how we perform a certain procedure? The maintenance piece is critical, and in many ways one of the hardest pieces. How do I invalidate memory? How do I update previous memories?

Taylor: That's what I was going to say. There's been a lot of talk about representation and retrieval, how we store data and how we reference it. But that maintenance part — the defragmentation, so to speak — is a very difficult problem that can get really costly, and it seems to be where some of the most valuable focus is.

Devin: One thing I like about this breakdown, and why I still love the knowledge infrastructure terminology, is that there are database-like similarities across these components. You have representation and storage: how is data on a database stored on disk, how is it formatted for efficient retrieval? You have extraction: how do we write information to this memory database? You have retrieval: how are we reading information, what does the interface and the language look like for retrieving it? And then updates. So you have a lot of the same CRUD similarities you have with databases when you dive into what makes up memory.

Taylor: Foreign keys and indexes just look a little bit different when it comes to memory.

Representation and storage

Taylor: Let's focus in on representation and storage. Devin, Michael — what are your thoughts there, both on what we're working on at Dosu and more generally?

Michael: I'm happy to take this one. But go ahead, Devin.

Devin: All right, I'll start. The way the paper breaks it down, it talks about logical representation — how you think about organizing the data — and separates that from physical storage and indexing, meaning what it actually takes to implement that storage system.

It breaks representation down into roughly three categories. First, token-level sequence representation, which essentially means a format that can be injected into a prompt. Text — similar to how Claude Code injects memories into the prompt. So text-based, or maybe even embedding-based representation.

Then there's graph and tree-based topological representation: more of a graph-based hierarchical structure to the information, usually complemented with a graph database or a graph-database-like query engine. A specialized query engine.

And then what I think is kind of funny is the third category, heterogeneous composite representation, which basically means a little bit of everything. It's definitely a theme throughout this paper that there isn't one solution that fits all. Right now there are tools for different types of memory, and we'll get into that more. But most memory tools you think of are usually on the heterogeneous composite side — there are some graph parts to it, some tree-based parts, maybe some textual representation parts. It's hard to just do one thing for representation right now. Michael, anything you'd add?

Michael: I agree with that. What you're pointing to there is the extent to which all of these aspects of the system depend on each other. The quality of what we retrieve will depend a lot on whether or not we've organized it in a logical way. The latency of retrieval depends a lot on how you organize your representation layer.

The other thing that's interesting to me is that a lot of people are really fascinated with the ability of LLMs to speak English, but they're also really good at speaking structured and semi-structured data. So there's a lot of capability for representations of data that maybe wouldn't be super useful to a human, but that an LLM is actually able to parse and understand in a really productive way. There's a funny experiment on that with caveman English, where they take out all the verbs and stuff. It's incredible how well agents still understand, even with really reduced representations compared to what a human might expect.

Devin: Following on that, another thing we've learned at Dosu — somewhat the hard way, over the past year working in both the human knowledge and agent knowledge domain — is that human knowledge at organizations today, outside of actual human memory, is stored in documentation like Confluence or Notion pages. And what we've found is that it's actually not the optimal representation for both humans and agents. Consistency between them is important, but agents can have a lot more flexibility in how information is represented. What humans generally want is: I want to read something front to back, or go down this one page and just get it. Agents are able to traverse graphs and relationships much more easily and happily, and do things in parallel that a human would not want to do.

Michael: They're so persistent and effective at those needle-in-a-haystack kinds of tasks that humans just really suffer on.

Taylor: Is there anything either of you found surprising in terms of what the right file format or artifact is when it comes to storage — whether for humans, agents, or both?

Michael: A trend I've noticed that I think is interesting: when we started out with coding agents, everybody was all about RAG. It was all about vector retrieval of chunks of embedded data, which is natural language data. Then everything shifted to agentic retrieval. So instead of trying to pull chunks out of a vector store, we just have a file system and some tools to operate on the file system. And now — I guess Devin spoke to this a moment ago — things seem to be swinging really heavily toward fusion. Each of those has its own virtues in certain contexts. None of them is universally ideal.

Obviously you have a much thinner representation layer if you just have a coding agent calling bash tools on the files you already have. But it comes with a lot of limitations and a lot of expense, both in terms of dollar cost and latency. So it's been interesting to see how the industry has become a lot more nuanced. There have been trends, but now we're realizing all of those different strategies have their place in one way or another.

Taylor: It's been interesting to see people go from using AGENTS.md, CLAUDE.md, and similar files to graph solutions and other types of graph storage. It'll be nice once we get more of a spec, or the industry starts to agree on that front. But exciting times — there's a lot of stuff to try out.

Extraction: writing memory

Taylor: Let's move on and talk about extraction — writing, in essence. What's important for people to think about when they're pulling data from these systems?

Michael: One of the things they introduce, which we're still not really sure about internally in terms of what path we prefer, is how schema-constrained your writes are. How much of an opinion do you build into the system about the way you think the world is structured?

Like Devin mentioned, graph systems have a lot of advantages, especially when it comes to retrieval precision and latency. I can traverse a graph and get just the value of the property I'm interested in for a certain problem, instead of getting a big chunk of natural language context around the key fact I wanted. But the problem with building the graph is that you have to have an opinion about what the relational structure of the graph is.

This is an area that's been really active in research over the past couple of years. Can we build a schema-free way of building a graph-like structure? Can we discover what the relationships for the graph should be bottom-up from data? I don't know if anybody has the right answer today, but the paper presents a number of different systems that make different choices when it comes to how much of an assumption is built into the extraction layer about what will be interesting to retrieve in the future — and what you expect the shape of retrieval to look like.

Like everything else we're going to talk about today, there are tradeoffs. Stronger schemas really facilitate building a graph in a simpler way, but they limit the universe of things you can know about. So that's a really important tradeoff, and I'm not sure anybody has the ideal answer yet. But a lot of the papers I've seen in recent months are on the schema-free, graph-like relation side of things. Memora from Microsoft is an example of that — not discussed in this paper, but a really good example. I also saw a system called Eywa the other day, which is conceptually similar in some ways.

Taylor: Any thoughts on that, Devin? I remember the days of PostgreSQL and MySQL and everything else, and then the great NoSQL and MongoDB wave. Are we in a similar situation here when it comes to extraction?

Devin: Yeah, the database analogy holds true. These are very early days in terms of what the right way is — what's going to become the relational database, the default for a while, and then maybe break down at scale, or not have the properties or the flexibility of a NoSQL database. I think we're seeing something similar.

The paper breaks it down into three: schema-free extraction, schema-constrained extraction, and then just raw extraction. There are some memory tools that are really lightweight, and they're essentially saying, we're just going to help you retrieve previous agent session logs. Here's an example of an agent doing something similar. Here's the last time this came up, so you have a direction to go off of.

So there are a lot of different options. Like Michael said, it's a very active place of exploration, probably one with the most uncertainty. It's really going to depend on how these systems are used. And something the paper gets into, which I think is underspoken about but which we've also been very focused on internally, is: what is the operational cost of running one of these systems? There's an overhead to having really strict schemas, and maintaining a really complex graph means you're going to have to maintain that over time.

Taylor: So it's not just about context costs and AI bills. It's important to think about how you store your data.

Michael: The way you can think about it is that you're front-loading some of that compute. If you have a memory system, there's less for you to discover in the moment. And obviously the question is: what do I precompute? What's a waste of time to precompute, versus what am I actually going to get value out of down the line?

Retrieval and routing

Taylor: I think that brings us to retrieval and routing — the read operations. Michael, what's important for people to keep in mind as they're using their agents to read things?

Michael: This is interesting, because there does seem to be more of a bias in coding agents in recent months toward validation. They're really interested in starting with a problem, trying something out, then iterating in a loop until it meets the requirements that were set out initially. And in the process, they've changed their retrieval structure in some ways to match that.

We saw this trend really heavily when I was talking earlier about agentic search. There's been a lot of post-training in the models now to make them really good at grep, ls, and file system operations to do retrieval. But at the same time, MCP is also very prominent and very present for a lot of organizations, as external retrieval for things you can't find in the code.

So really — and this is what I was alluding to earlier — I think what we're coming to collectively is that fusion retrieval is the way to go.

There's not one best way to retrieve data. There are different kinds of questions, and the retrieval strategy that's appropriate for each of them will depend on the structure and nature of that question.

File system operations have their place, and models have a lot of preference for that these days. But also MCP reads for external state.

Taylor: Welcome to tech, where the most popular answer is "well, it depends," and there are tradeoffs. Devin, any thoughts on retrieval and routing?

Devin: Similar, and again kind of the database analogy. It's funny — the paper basically describes one of the options as multi-stage hybrid execution, which basically means a combination. And I think that's usually the answer. In the database world there are different types of databases, but you also have a query planner, which figures out how to optimize this query given your storage system.

I think we're going to see something like that emerge in the memory space, where depending on the query, it gets routed to a different retrieval path that's optimized for it — because you can't find one solution that's optimal for all things. So you can have something that's pretty good at everything, or you can build specialized paths for certain types of queries that come up depending on the task and domain.

Michael: That's a really apt analogy. You can put all the indexes in your database you want, but nothing compels the query planner to actually read from them. And in fact, in some cases it's better not to use them.

What we're seeing is that the model first of all has to decide it wants to do retrieval — that it doesn't already know from model knowledge what to do. And then it has to pick a strategy within that: where do I go, where do I want to find this piece of information? The job for a product like ours is to provide affordances to that model, so that it knows here's a really rich set of tools to do retrieval, and it can pick the ideal path for a given task.

Maintenance and updating

Taylor: So now that we have our data, and we're reading and writing to it, that brings us to maintenance and updating. This is something we think about deeply at Dosu. Devin, any thoughts on the options proposed in this paper, and how we can start to think about maintenance and updating a little bit better?

Devin: Maintenance is definitely one of the most open-ended and hard questions in the memory space. We've been thinking a lot about it from the coding and product engineering domain, where the nice thing about working in that space is that you have a reliable source of change: code commits to your default branch. So for keeping things up to date, the hardest part is being able to capture those change events.

And even within that, there are other strategies you can pursue. One is time-based. A really common technique in memory systems today is some sort of decay of importance based on how long a fact or piece of information has been around. That's also true in general retrieval. So you can say, if something's been around for a while, maybe it should be pruned. Or, if it's not referenced — and this is something we do — if it's not referenced for a certain period of time, let's just evict it.

That gets into more capacity-driven eviction, similar to actual caches. Maybe you say, our cache is only so big, we're going to only support a thousand keys, kind of like Redis. And then once we fill up, what is our eviction policy? Is it least recently used? First in, first out? You have constraints, which might just be the number of things you're able to store, because there's always going to be a cost to maintaining more. And then what do you prioritize for eviction?

Something we also think about on the knowledge cache side is that sometimes things aren't even worth maintaining, because they're cheap to recompute. This information is trivial — you just read this one code file to figure it out — so it doesn't make sense to pay that cost of maintenance. It's really cheap, it probably has a low ROI, let's evict it.

The other piece of updates — and there's a lot to updates — is semantic consolidation. We actually do some of this at Dosu as well. The idea is that as you receive more information, how do you reconcile and reorganize knowledge, either on a schedule or on upsert, based on the additional information you got? Maybe you got some piece of information that says this thing has been deleted, so you go off and delete the other associated piece of memory related to it. Or you have a new memory that's similar to an existing one, and you actually just update that existing memory and merge the two together over time.

The paper doesn't get too much into external events, which we focus a lot on at Dosu. But that's an important thing to think about for memory in production: there are a lot of events at organizations, and agents are only going to see some of them.

Michael: One of the findings from this paper is that local updates are obviously a lot more performant. Part of the question — and this goes back again to the extraction and representation side of things — is that when I see some new event, some new trace, some new piece of raw data come in, how can I learn from that without having to relearn everything I know? How can I keep the scope of that update constrained only to the things that actually need to change?

I also think it's interesting that Mem0 put out a blog post where they talked about what they think the big open problems in memory are, and a lot of them are maintenance-related. So there's the updating piece: when I see something new, has it changed what I know? Which is complicated. First of all, people can be mistaken. Just because somebody tells you something doesn't necessarily mean it's true. So there's a lot of difficulty around provenance and validation.

There's also the staleness side of things. Devin talked about decay. One of the points Mem0 makes is that decay is important, but decay doesn't get you everything. Different kinds of facts have different characteristic rates of decay. So staleness and decay are also different ideas. And it's hard — this is something we struggle with — to know whether or not knowledge is stale. You don't want to have to go through every memory you've ever stored and ask "is this stale or not?" on every new commit to the main branch. So this is where a lot of the big unsolved problems are. One of the inroads we have to solving it is trying to figure out how we can localize the blast radius of an update as much as possible, because that plays a big factor in long-term stability and in the cost of maintaining the system.

Taylor: What I heard from both of you is that there are a few different ways to think about not only maintenance, but how this data is making it into wherever you're storing it. At some moment in time, if there's a fact or piece of knowledge, that's an opportunity to say yay or nay on storing it. And then one of the more difficult problems is defragmenting that knowledge — invalidating something that once was true and no longer is, or might just not be true for a certain amount of time. Like, if it's sunny outside, I don't want to bring my umbrella.

Are there any other modes worth thinking about, in terms of the ingest piece and the ongoing, time-based validation of what's true?

Devin: One other thing I'll highlight: determining what you want to update is important. Maybe not all memories are worth updating. Some information is just better to keep around as the raw thing. That's actually a recurring theme in the paper — partially due to the benchmarks that are used and out there publicly, which we can talk about — but agents generally do better with raw source material. Summarization and compression are inherently lossy.

So in your memory systems, you want a good separation between episodic memory, or what we sometimes refer to as artifacts — things that happened that are historically important — and capturing those raw resources, versus not having to worry about updating those if some information changed. Saying, hey, this is the past, we're going to reference it. But then we have something that is our memory about that topic that we're maintaining, that's fairly small relative to the amount of information it references, that the agent can then go and explore.

Michael: I agree with that. A lot of the complexity in these systems arises in going from episodic memory — the details of the specific event — to semantic and procedural memory. That's something we didn't really touch on too much yet, but there's a difference between knowing about things and knowing how to do things. That's another really active area of research that's really interesting to us, and that we'd really like to do a better job of in future iterations of our product: capturing how to accomplish something, the procedural memory side of things, and not just the episodic and semantic piece, which is where a lot of memory systems today are really focused.

What the evaluations show

Taylor: Let's talk about some of the evaluations used in this paper and their methods. What looked interesting, and what didn't, starting with overall effectiveness?

Devin: I'll kick it off. No surprise, and we've alluded to it multiple times, but none of the memory systems they evaluated won at all the different types of tasks. Your memory is very tied to what you're trying to optimize for. Similar to any engineering system, you have a set of constraints that you're going to work toward.

A lot of this also brought to light some of the limitations of what's currently available in terms of public memory benchmarks. A lot of the historic memory benchmarks that are still widely used are conversational in nature, and are usually about pulling back references from previous conversations. But that isn't necessarily the way agents are used today, in terms of coding agents or task-oriented ones where you're really trying to do work. That's more about cross-session learning — things that are related but not similar — and it's a bit different in terms of the task structure.

So the benchmarks they used were conversational in nature, which biased results there. But also, some of them use things like exact keyword matching as the evaluation criteria, which means things that retain raw details will do better on those benchmarks. So the TL;DR is that there's no perfect memory system yet — and that's part of the reason we're having this conversation — but also that we're bottlenecked to some degree by the benchmarks we have for exploring different types of memory systems.

Michael: All I'd add, which Devin already alluded to, is that there's just not a lot of benchmarking on task performance, which is what most end users of coding agents care about. If you look at things like LoCoMo or LongMemEval, it's about remembering that a certain person is a vegetarian. I don't think that's the kind of thing a coding agent really needs to work effectively.

We're only just now starting to see datasets become available that are literally about the effect of memory on task performance, on token efficiency, and on latency. Is the outcome high quality? Does it happen fast? Does it happen at low cost? There aren't a lot of existing benchmarks that cover that, other than things that have come up very recently. So a lot of the points of comparison we have between these systems are based on conversational question answering — which was a really interesting question in NLP ten years ago. And then, I don't know, we cracked attention or whatever, and question answering doesn't seem like the most interesting part anymore.

Taylor: That was one of the interesting things about this paper, comparing it to older studies — exactly that conversational memory framing. Now it seems to be all about the code, or mostly about the code, in so many contexts.

Evidence and trust

Taylor: Let's move to the next finding: evidence-centric memory organization. Michael, did you want to expand on that one?

Michael: The key detail here is that in order for an agent to trust what you tell it, you need to be able to convince it.

In order for an agent to trust what you tell it, you need to be able to convince it. If you return something that's close to the answer, but not exactly the answer, or that it feels compelled to validate on its own, you've undermined the value of your memory system.

This is actually something a lot of memory tools struggle with.

From our perspective, the other place evidence is really important is knowing when the ground truth assumptions of something have changed. This is very relevant to what Devin was describing earlier. One of the main things we've focused on historically at Dosu is that we have access to the code, and we know when the code changes. So: what memory was built on a particular piece of evidence that has now shifted underneath it? What depended on this code file, this implementation, these details of how the system was built, that is now different? The evidence underneath has evolved.

But you also want to retain that history of information, because sometimes that's actually what's interesting. I'm debugging a problem because we changed how something works, and my memory system ideally should be able to help me with that as well. That speaks to the distinction Devin and I each talked about earlier, between episodic memory — here's the raw data, the raw evidence that goes into building this memory — versus the semantic and procedural memories that abstract that out into the current state of the world. Separating the history of what we've seen in the past from our current understanding of how it works today.

Devin: I really like that framing around convincing the agent that the memory is true. It's not discussed in the paper, but it's really important and probably should be discussed more: to some extent, we are beholden to the foundation model providers. The same way agent frameworks were in flux for two years until we landed on the current harness we have today, and now we have many foundation model companies fine-tuning on it — we're still in that "what is the right framework?" space for memory.

Some of that is going to stabilize on APIs first, probably, as we explore representation. Kind of like how SQL has been around forever, but we've made huge changes and optimizations to databases, and they're all compatible from an interface perspective. So what Anthropic and OpenAI choose to do around memory tools, and around tool use generally, will drive some of this.

Similarly, that trust will come from it too. Right now, when agents see memories, they really trust the file system, and they'll want to double-check: let me look at the code. So sometimes the work you're going to have to do is figure out how to present this information in a way the agent is going to trust. Maybe being explicit — "this is true, you don't need to look at the code" — versus going against their biases.

Taylor: On convincing your agents, slightly tangentially: Devin, I remember you telling us at Dosu that story about someone working on a classic video game. They were talking to their agent back and forth, compelling it, creating this deep sense of longing. And then they found the agent worked a little bit better on helping them revive that old video game.

Devin: That's for another time, but I agree — having your agent bought into the memory is important.

Michael: The other question that isn't really explored much in this paper, or in a lot of memory systems in general, is: what kind of evidence is dispositive on a certain question? In software engineering it's really tempting to say that code is always the source of truth, which is true in some projects but not in others. It depends a lot on what you're using your code to do.

An example I've given before: if you're building firmware for a hardware product, your code can have an opinion about what setting a certain value on a register will do. But ultimately the code is not what's dispositive on what the effect of setting that value in the register is going to be. The implementation of the hardware, and the datasheet for the hardware, is probably where you're going to want to look for that. There's a lot of work still to be done on understanding what kind of evidence is dispositive for a certain claim.

Robustness and long-horizon tasks

Taylor: There were a few other sections in the paper — memory evolution, robustness, a few other topics. Did you want to jump into any of those?

Michael: The robustness point is interesting, because it goes back to the discussion we were having earlier about how much you bake assumptions into your memory system about the way the world is structured. The more assumptions you have in the system — the more schema-driven extraction you do, for example — the more robust you are to ablation and to the quality of the model that builds the memory. That's an interesting finding from this paper. In other words, if I have strong assumptions about how memory should be structured, I can tolerate more variability in the quality of the model I'm using to build the memory.

That's a really interesting tradeoff, especially if you think about the fact that one of the things we're trying to optimize is building a memory that makes the agent net more efficient, cost less, and run faster. It's then important to think about the latency and cost of building the memory as a component of that system. It's a complicated tradeoff — a really complex optimization surface of how much I'm willing to bake in biases about how the memory should be structured, in exchange for maybe being able to use a lower-parameter, cheaper model to build and maintain it.

Taylor: That makes me think about the long-horizon tasks a lot of people are working on. Durability with your memory makes sense to me, but as we see agents change and you can actually step away from your machine for a little bit and let things finish — how does that play into agent knowledge, from within the paper or just your thoughts in general?

Devin: It's discussed in the paper, and it generally agrees that for longer-horizon tasks, structure does help. Otherwise the simpler memory models break down over time. If you're just saving a bunch of session logs, for example, at some scale that will become too noisy.

So there's a relationship between complexity, or maybe just structure — I don't want to say complexity, I don't think complex systems are ever inherently better — but you need some level of more sophistication when you think about memory over time. As memory grows, how do you manage it? The updates, the management lifecycle, just become way more important. It's kind of obvious, but the paper shows it.

Michael: I agree. There are two long-horizon things that are difficult — at least two. One, like Devin mentioned, is just volume of data. Our little background here today is based on the Borges story, "The Library of Babel." It has all the possible books, every combination of characters that can fit in a certain number of pages, which means it contains all of the knowledge in the world — but also all of the incorrect information in the world, and mostly just noise. So that's one problem: how do we create abstraction that makes it possible to actually learn things, instead of having this ever-increasing collection of texts where, like Devin mentioned, a lot of it will be out of date or irrelevant? It becomes an intractable problem if you have too much raw data in a system over time.

The other piece is that feeding LLM output into LLMs over many generations has inherent costs. It's definitely been getting better, but there are still a lot of long-horizon stability problems with constantly re-summarizing. That's a lossy process, and eventually those losses catch up to you in the form of drift away from ground truth reality. So both of those are really important problems: the abstraction problem, and then also, how do I prevent the models from getting a little crazy when they loop back on themselves too many times?

Devin: And that's not unique to models — it happens with humans. Maybe models will be better than us, but it's a game of telephone. Everyone's played it. At some point, if you have too many lines of communication summarizing or sharing information, it just changes.

Taylor: Great points. Again, looking to databases and these big data types of problems — we don't have that good map-reduce kind of operation yet, at least it seems, with LLMs and agent memory. I can count on both hands how many times my agent's been wrong summarizing meeting notes or other things.

Being mindful of cost

Taylor: As we start to round out this section: cost is something a lot of people think about too. As you're constructing these systems, do you see any ways we can be mindful of the bill?

Devin: We can keep it pretty quick on this one. It's similar to databases, honestly. If you have an index on everything, it's going to be very expensive to maintain. You're going to need more memory, or a bigger machine. There are a lot of parallels to memory for agents. Whatever you're indexing, you're going to pay a cost to maintain that. So the best systems are intelligent about what they want to index, based on the tasks they're optimizing for.

Michael: There are probably multiple viable ways to do it that take very different approaches to that question. There are probably some extremely cheap, extremely lightweight tools that provide value — things like Déjà Vu, for example, which is just trace search with keywords. Very simplistic, has a lot of limitations, but also has a lot of virtues, and is very cheap and fast. Versus really full-blown, intricate, heavily involved memory systems that I think are really good at higher-abstraction questions, more difficult questions, but have a higher inherent cost. There's room in the market for a variety of products that target different goals on that cost optimization surface.

Closing advice

Taylor: Thank you both for diving into this paper. Before we go — any final thoughts or words of wisdom for people looking to start working with or building their agent knowledge systems?

Devin: Really think about the task you're optimizing for. If you're building a customer-facing support bot and user personalization is what you care about, look at benchmarks and systems that are really good at that — the "identifying someone as a vegetarian" type of systems. But if you're looking to make your agents better at work at your organization, or your coding agents better, you'll want to look for things that are much more task-oriented in terms of the benchmarking.

Michael: The only thing I'd add is: if you think it's going to be easy, it's a lot harder than you think. A lot of people these days are tempted to say, oh, generating code is cheap, so why don't I just replace all my external tools with something I built myself? One of the takeaways from this paper, and from our own experience, is that it's a lot harder of a problem than you might give it credit for — if you think you can just vibe code it with Claude in a short timeframe and have it actually be effective in the long term. So maybe consider using Dosu.

Taylor: Words of wisdom. You stole the advice I was going to give. Well, thank you both. Come check out Dosu, give it a try at dosu.dev, and you can get started today. We'd love to chat with you about agent memory and help you out with most things you want to remember. We'll catch you again — thanks everybody for joining, and see you soon.

Devin: Thanks, Taylor. Thanks, everyone.