Agent Memory: What Information Is Worth Remembering?

Episode 3 of Dosu's agent memory series works through the write side of memory: an agent can save just about anything, so how do you decide what's actually worth storing? Taylor Dolezal sits down with Michael Mangus and Devin Stein to cover memory as a cache, the tests Dosu runs before writing knowledge, how much schema to impose up front, filtering early versus late, and what changes when memory belongs to an engineering organization rather than an individual. The series works through Are We Ready for an Agent-Native Memory System? (Zhou et al., 2026); see episode 1 on the four modules and episode 2 on storage. This transcript has been edited for length and clarity.

Transcript

00:00 · Introduction

Taylor Dolezal, Head of Open Source, Dosu: Hello and welcome, everyone. My name is Taylor Dolezal, I'm Head of Open Source here at Dosu, and I'm joined by Michael Mangus, our Head of Product, and Devin Stein, our founder and CEO. We're continuing our series on agent memory. I'll post the paper we've been discussing in the chat, so check the comments and you'll find it there.

Today's title is "Agent Memory: What information is worth remembering?" An agent can save just about everything — the hard part is deciding when to store something as knowledge. This week we're focused on write operations; next time we'll get into reads.

00:48 · What's worth caching

Taylor: Welcome, guys. My first question is for you, Michael: what's one piece of functional knowledge that's worth caching, either because it's frequently used or because it's expensive to recompute?

Michael Mangus, Head of Product, Dosu: I think that's really the most important question, because with caching you have an induction problem — you don't know what will or won't be used in the future. So, as you're implying with your question, you have to make a calculation between the cost to write something, how frequently you expect to retrieve it, and how much effort you save when you do retrieve it.

In terms of practical examples, the thing I've found most effective personally so far tends to be troubleshooting. You have a situation where you think something should work. Out of the box, the model agrees with you — "yeah, that should work" — but in reality it doesn't. You have to go through this interactive debugging back-and-forth with the model, trying to explain to it why the thing you expect to be happening isn't actually happening.

Those tend to be the cases where there's a lot of back and forth, a lot of theories that get thrown away or backtracked out of. That's a really good example of something that's inherently hard to reconstruct. A lot of the time you need the human there providing the interface to try things out, especially if you're working with an agent that can't directly manipulate the system you're debugging — you're the intermediary, the go-between. And often the model has the wrong default assumptions, just like you did as a human.

So I think that's a really valuable case, and one I've seen come up quite a lot: remembering how to debug or troubleshoot something without going down the incorrect paths again.

02:32 · What's cheap to recompute

Taylor: That's helpful. It's the same question I ask myself about a meeting — what's meaningful here, what do I write down, what are my to-dos?

Devin, are there examples of things that might be cheaper to recompute than to store in long-term memory?

Devin Stein, Founder and CEO, Dosu: For what's cheap to recompute: information that's available from a single file or a single location, or that's already well documented, doesn't necessarily make sense to store as a separate memory or as knowledge.

Here's a fun example from something I did recently. One thing agents are really good at is reverse engineering systems. That normally takes a lot of human time, and it actually takes a lot of agent time too. What's interesting is how much undocumented or learned knowledge comes up in those sessions, because when you're trying to understand a system there's nothing telling you how it works. You have to probe it, test things out, experiment, and see what works and what doesn't.

That's an example where capturing knowledge and memories is really valuable, because it's expensive and hard to figure out what's true in those systems. When you don't know what the answer is, the learnings along the way are often what you want to capture.

04:02 · The full cost of a write

Taylor: I've heard the same thing about people learning: when it feels a little more difficult, that's useful learning, and it helps that fact stick in your head. It's interesting applied to agents — capturing the friction it takes to get somewhere.

We've talked a bit about knowledge being similar to a cache: cache hits, cache misses. Michael, when it comes to memory, why maintain it at all? We've talked before about how we don't know what we don't know, or what we'll need in the future. Is there anything worth expanding on there?

Michael: The basic calculation we make is: what's the cost to write this memory and maintain it over time, versus what I expect its utility to be in the long run? If the balance comes out in favor of it being cheaper to write and maintain than to recompute, then obviously you have something that makes sense to put into your cache.

What you have to consider is all the different costs that go into storing that memory. There's embedding the memory itself. If you're going to enrich it with entity extraction or relationship graph concepts, those take computational time and effort. Storage has a cost, even though these days that tends to be very small — but it also comes with things like precision penalties on retrieval. The more you have in your index, the more likely it is that you get some spurious result. So you have to take a full accounting of what the cost of saving the memory is.

Then on the read side, like I mentioned, it's hard to predict, but there are two ways you can get value out of a read. One is that the information is really hard to find — it requires a particular human's perspective on a decision that was going to be made, or, as Devin was mentioning, searching through a lot of different files or across different systems. And then, especially when many people frequently need to access that information, you get a big multiplier on the value of reading it back out of the cache.

So the obvious case to cache something is when it's really hard to know that information and people need to access it a lot. But the reality of most systems is that you have to make a much more subtle calculation than that. The balance sheet doesn't usually shift so clearly in one direction, so you have to be really thoughtful about what the cost of a write actually is.

06:41 · When is a memory worth maintaining?

Taylor: That really brings us to the core of our conversation today: what counts as memory? Coding agent sessions aren't always going to be source material or automatic memories. Figuring out what we want to write is what decides what future agent sessions inherit, so that we don't start from square one.

Devin, I'd like to start with you. If an agent can reread code, run commands, and go look at the source itself — when is a memory worth maintaining?

Devin: We touched on this, but it's really about how difficult it was to get that piece of information. If it was easy, if it was trivial — read the README — then it doesn't make sense to maintain it. But if it took trial and error, if the agent made mistakes, then that information is worth capturing.

There's also interesting research related to this: similar to humans, different knowledge is valuable to different types of agents. What's hard for Claude Opus may be different from what's hard for Codex, for example. What's difficult can vary from model to model. But fundamentally that's what we're trying to figure out — what's hard to relearn, what mistakes can we prevent in the future, what information is coming from external systems that the agent might not have access to, like humans.

08:25 · Scoring difficulty with agent session data

Taylor: Michael, when you're using something like Claude Code or Codex, you have session data, you have your logs. Is there anything meaningful about knowing how that's stored as memory, or how to store it as memory?

Michael: Like Devin mentioned, we need some way of operationalizing the idea of difficulty to reconstruct the memory. There are some cases where that's obvious — it's a human decision, so you need to talk to a human to get it. But there are other cases where it's less obvious.

Having session data from your agent lets you give a very literal score: how many tool calls did it take to produce this context, and how much wall clock time did that take the agent? That's really nice, because it gives you an objective measurement of difficulty, at least for the things the agent is capable of retrieving itself. You can assign a very literal score from agent session data to help you decide what the cost of reconstructing it is.

09:24 · The tests we run before writing knowledge

Taylor: Expanding on that score — I know we've talked internally about tests to run before writing a memory: scope, and a few other things to consider. I'd love to jump into what some of those tests are that we've enumerated so far.

Devin: We're actively iterating on what that rubric looks like, but roughly, right now:

What's the future value of this memory? Did this information help prevent a mistake, or would it have caused the agent to do something different, to take a different action?

Then there's authority. This is true both for agent sessions and when you think about enterprise knowledge: where the information is coming from, or where agents are reading it from, matters. Some people don't know what they're talking about but say things anyway; other people actually built the system and have a very deep understanding. And sometimes even the people who built the system make mistakes. So authority is really important to consider.

Scope is another big one, and probably the hardest when you think about knowledge at enterprises and knowledge in production. Is it true for this session? Is it true for this task or branch we're working on? Is it true within the context of this repo, this product, or the entire org? You can have the same information differ between teams — that's really common at companies.

And then, what's the cost of being wrong? Sometimes the most valuable things have an equivalent downside if they're incorrect.

Finally there's the recomputation cost, which is more measurable, as Michael already touched on. Especially with agent sessions, we know roughly how much time and how many tokens it took to compute that. So: is it worth saving?

Michael: Just to add to that — a lot of memory systems so far have been really focused on understanding individual people's preferences. I think some of that is a residue of what we talked about maybe two weeks ago, in terms of how benchmarking has historically been tied to conversational question answering. That was one of the most interesting problems in the 2010s: how do I remember what a person said their preference was several turns back in a conversation?

Because those are the benchmarks that are out there, those are the things people have built to. And it sidesteps a lot of the problems Devin was just describing, right? Because I'm the authority on what I want, and it's scoped just to me, because it's only supposed to remember my memories.

What we're seeing now is that people are interested in bringing memory away from just the individual and their preferences, and toward what's true for an engineering team — both in terms of policy and in terms of the systems that have been implemented. A lot of those problems become much harder to solve. So what you're seeing is that some of the preliminary memory systems that were really tailored to those individual preference use cases don't scale well to engineering teams or organization-wide memory, because they struggle with the categories Devin was just outlining.

12:52 · How much schema is too much?

Taylor: When thinking about how to write knowledge — another "it depends" answer — how do we define a taxonomy or a schema? I like what you've been saying on that front, Michael. I know Mem0 has some default categories, among other approaches to look at. But do you think we might be getting ahead of ourselves by trying to figure out the taxonomy or tagging of memories before we've seen what those memories might actually constitute?

Michael: There's a lot to say about that. Schemas can be really powerful insofar as they give you a baked-in structure for your memory, which can make it a little bit easier to write and maintain in certain circumstances, because there's just less to understand at write time if you already know the categories you're sorting things into. The problem is that it biases your perception.

Taylor, you mentioned analogies to human memory earlier, and that's a perspective I bring to a lot of this just because of my own background. Selective attention is a big part of memory for humans. There's a very famous experiment in psychology: you have a video of people passing a basketball back and forth, and you ask participants to count the number of passes. About a minute and a half into the video, a guy in a gorilla suit walks through the background. The real experiment is asking people afterward, "did you notice the guy in the gorilla suit?" People are so fixated on counting the passes that about half of them didn't notice him.

One of the things that can happen if you have a really strong schema constraint on how you write memory is that you become attentive only to the things outlined in your schema. Then, in the future, if you need to retrieve information from that episode that isn't encoded in your schema, I think you're going to struggle to do that.

I saw an interesting paper, I think it came out in July, on file system-based memory for LLM agents. One of the things they find is that raw episodic content works really well with big and powerful models, because those models have an understanding of the world, of how things are organized, and of abstractions and categorization that they can apply on top of that raw data at inference time. Whereas if you have a smaller and less powerful model, it benefits more from having structure built into the memory.

At the same time, there's a lot of cost to that structure, in terms of maintenance, the complexity of writes, and nuance being lost. So, like you mentioned, Taylor, there are trade-offs there. I don't know if anybody has the right answer. I think the best systems today do a good job of balancing having some opinions about how to organize data, while at the same time not being too top-down in the categories they impose on how they attend to input.

The other consideration is what you're able to restore as you maintain that memory over time. We've talked before about how one of the costs of maintenance is the long-term instability of memory. If you don't have the raw underlying data to go back to, you have inherently lossy summarization carried out over many generations, and certain information will just be totally lost to you. So there's a lot of complexity in deciding the right amount of organization to build into a memory system.

Devin: The gorilla analogy is amazing — I'm definitely going to use that. On small versus big models, and small models liking schemas: I think there's an interesting parallel to prompting. As models have gotten quote-unquote smarter, you actually want to do less with your prompting. You want to be less restrictive, so you can lean on the model's fine-tuning and intuition to get more out of it.

So as models get more and more capable, I think we'll want less and less schema up front, because models will start to just know. We already see this with Claude — Claude is clearly ahead of Codex on memory. Claude has a much better intuition about what it should save.

Michael: I agree with that. A final thought there is that the scope of your memory will also have a big impact on what kinds of schemas you'd even want to maintain. The history of knowledge bases is full of failed ontology projects that never really took off because they tried to capture the entire world in one schema. Specialization of memory tools can have a lot of value in just narrowing the scope of the things you need to be able to represent in your schema. So part of the effect Devin's describing, in terms of the quality you get out of these models, is also their specialization for the kinds of tasks they're best at.

Taylor: That brings to mind something Boris from Anthropic, one of the creators of Claude Code, has talked about with Opus 5, Fable, and some of these newer models: deleting your AGENTS.md every six months, I believe it was. Because the models have so much context, they might not need as much steering as they continue to get more parameters, get smarter, and as the harness improves.

Maintenance is hard, I guess, is the takeaway. But — just use Dosu. We can help with that.

18:24 · Filtering early vs. filtering late

Taylor: When we think about artifacts, knowledge, and then filtering early or filtering late on what we might want to store: are there good use cases for filtering knowledge early versus filtering late? Devin, I'd love to turn this to you.

Devin: I think this is really related to what we were talking about with where you want to be on the schema-versus-no-schema spectrum, just from a slightly different angle.

What we've seen is that agents typically do really well when they have raw source material. But there's also a lot of information generally. It's almost like the point of memory is: how do you take everything from the ingest and make it into something more searchable, easier to explore, so agents get a good starting point?

So at one extreme, you could say every session history is your memory. And yes, that is true — those are useful artifacts for exploring what has happened. But it's also very inefficient, and part of the promise of these knowledge systems is to improve token efficiency. Like humans: we're not relearning the same thing every time. You remember where to look in your previous work to find information.

I see it similarly with agents. You want to be pretty generous in terms of having access to raw source material, but then you need a summarization or semantic layer on top of that to help organize the raw information it's coming from.

Taylor: And to underline your earlier point, Devin — if you're changing out models, these things might be different, engine to engine almost, when you're thinking about it that way.

Michael: It also interacts with the question we started off with, Taylor, which is: how do I decide what makes sense to put into a cache? Part of what we talked about there is that the cost of writes is a big consideration. If you do very minimal filtering and you're more of a maximalist on ingest, sometimes that can reduce the cost of your writes. You're moving some of that cost more toward the retrieval side when you filter late, because now you're paying that filtering cost at the end. Again, it depends, but in a lot of cases I think that can make sense as a trade-off in terms of where you want to spend that computation — because in exchange you can have a wider variety of things that make economic sense, so to speak, to put into your cache.

21:06 · Making memory trustworthy, and the golden age of provenance

Taylor: That makes sense to me. I did see a comment in the chat from Carl Cardenas: "It's hard to delete my markdown files. I put so much work into them." I'd say hang in there, Carl. It's tough to keep this up to date.

When we're thinking about artifacts and what's canonical and what isn't — how can we think about making memory or knowledge trustworthy? Do either of you have tips on that, or things you've observed?

Devin: On trust, we talked a little bit about that before, in terms of how getting models to trust memories is sometimes difficult. It varies from harness to harness too, Claude versus Codex. At the end of the day, models do like to look at the source material. Making sure they have a path to seeing supporting evidence for any information you surface is very important. You can also use prompting within AGENTS.md to help guide or correct this behavior to some extent.

But at the end of the day, you can only summarize or abstract away so much — from a model, or from anyone. Sometimes you do want to check things out for yourself, and there are inevitably going to be missing pieces of information that come up in the summarization or information extraction process that require referencing the source material.

Michael: I would just add that I think we live in a golden age of provenance when it comes to software engineering. It used to be that I would go off and write my code, and the only time I would actually have to explain what I was thinking, or why I did something a certain way, or what I was trying to accomplish with a certain choice, was in the PR description — which I think people didn't always put that much effort into either.

Now, because so much work is done through coding agents, I have no choice but to write down what I'm expecting. And that gets to your comment earlier, Taylor, about the value of session data from coding agents — the actual trace from the agent run. You know exactly what the person asked for, where things backtracked, what decisions were made, what files were retrieved by the agent as "here's the template or the pattern or the analogy that I want to follow."

In a lot of ways, the transition in how people write software is really amenable to having a really rich set of provenance for all of the things that end up in your codebase now. I think we're going to see a lot of that reflected in what version control means in the future. What we have with agent traces is like the most descriptive commit messages ever available. And there's a lot we can do with that provenance information when we build systems like what we're building at Dosu.

24:00 · From personal memory to organizational knowledge

Taylor: I've seen a lot of commentary across X, Reddit and other places on the internet where people who have been coding for twenty, thirty or more years talk about stepping away from working on this more manually and starting to think a little bit more like an architect, or a stakeholder, or a product manager — which I feel relates really well to what you just said.

Going a little bit deeper on scoping and trying to frame things appropriately within your sessions, I see at least two ways to think about this: whether it's personal memory, single-player mode, where I'm working on a project; and then how that relates to organizational memory, where we're the greater sum of our parts working on something maybe more ambitious. Most memory systems start with personal preferences. What changes when the memory belongs to an engineering organization?

Devin: Going from personal memory to organizational knowledge, it's really a matter of scope. That's what we mentioned before as the gates, or the things to think about, when writing knowledge. Scope is a really important one with knowledge in production.

Historically, around user personalization, it's just the user — it doesn't matter. But when you go beyond user preferences, you need to understand where a piece of information fits into the broader org. Maybe it's a habit or an opinion that's only held by this individual, so it should belong in user memory. Or we've seen this action across four people on the team, so it's actually a shared workflow and it should be team knowledge. Or maybe it's specific to a repository you're working in, or even something more granular, like a branch within that repository.

I think branching is an interesting piece of scoping that's unique to coding agents, but all products in the next decade are going to look something like coding agents, or have a coding agent component in the system. So I think the notion of a branch, or a session, as an isolated piece of knowledge that hasn't been committed back to your broader knowledge base is going to be a recurring theme.

I had a fun experience over the weekend. I was working on a side project where I had a bunch of parallel coding agents going, and Claude used its default memory to write a bunch of memories that ended up confusing my agent, because they were on different branches and hadn't been committed yet. So it was looking for information, or thinking about things, that didn't actually exist yet. I think that's a really good example of why scoping to something like a working product is really important.

And then beyond that, you can think about the organization as comprised of products, teams, domains — there are lots of different ways to think about organizing information within the company.

27:11 · Who owns institutional memory?

Taylor: That brings me to something we can expand on a little more before we wrap up. If LLMs and agents are writing nearly all of the code within an organization, is there any worry about frontier model providers determining what's worth remembering in your organization? There's a lot of IP there, a lot of knowledge. Do you have any thoughts on that one, Michael?

Michael: What I worry about the most is that we are losing a concept of institutional or organizational memory, because there is less active human attention to the outputs that are actually making up our systems. So what I'm worried about is not so much that a third party will own that, but that nobody will own it.

That's where I do think tools like what we're building at Dosu will play a major role in the future of engineering teams. We've talked about it before in terms of what it would take to have an infinite bus factor on your team — to have nothing that lives entirely in one person's head. Because of what I was describing earlier, about how we now have much richer information about what somebody was thinking when they worked on a task, we do have a lot more capability to fill in those gaps in memory.

But the hard part right now is that, from the perspective of somebody who has historically really felt like I was on top of how things work in a system, it can be really hard to keep track of that these days, just with the pace of development. I think you really need tools that can serve that function of organizational memory, so that for any given system, decision, or feature there is some record to go back to about how we got to where we are — so that we don't repeat the mistakes of the past.

Taylor: Great point. Another trend and topic I hear a lot about is over-reliance on tools. Even GPS, right? People don't need to remember how to get everywhere within their city, especially if it's sprawling. Is that good? Is that bad? It depends.

Michael: And there's so much research about that also, Taylor, in the context of things like self-driving cars, or automation in aircraft. How do you maintain situational awareness of a system that mostly runs itself? It's obviously a major safety concern in those industries, and maybe even in software to some degree — I think it could be a cause for concern when it comes to things like security. So I think that's actually still a very active area of research that nobody has a great answer to yet: how do you stay engaged with a system that only really requires your intervention a very small percentage of the time?

Taylor: I love it. Thinking about memory between provenance, scope, and permission — and then, what are the mental muscles you're using, or not using, with your agents? How do you trust the outputs? What makes sense to still keep in your head? And how do you avoid trusting them too much?

Michael: It's like the GPS takes you into the lake, right?

30:19 · Closing thoughts

Taylor: Well, with that, I think let's close out. Devin, I'd love to start with you: any words of wisdom wrapping up how we think about agent memory and writes?

Devin: For words of wisdom, maybe I'll just share one topic we didn't touch on that I think is worth bringing up: how do you write knowledge? There are a lot of different approaches being explored.

One is more active — agents explicitly calling a tool to add a memory or write knowledge, like writing notes in the case of Dosu. And then there is more and more happening in passive or background knowledge collection, where you're doing analysis of agent sessions, of code review comments, of Slack conversations, and trying to identify important information that might be useful in future retrieval.

So it's not just about what information is worth remembering, but how you go about it — which I think is also an interesting question that no one has figured out yet.

Michael: I definitely agree with that. I think that's one of the big open questions. And a related issue that Devin alluded to earlier is: how do I make sure that the information written by model A is still useful to model B, or to the next-generation model A prime? I think there are still a lot of open questions in terms of the variation between models and what they want to write, and then what they want to read back later.

Taylor: Well, everyone, thank you for joining us this week. Thank you, Michael, and thank you, Devin, for your time and your insights. Hopefully that helps as you're thinking about knowledge, agents, and writing.

Next week we're going to tackle read. I'll go ahead and share our Luma link in the chat — please join us, we'd love to hear from you and continue the discussion. And if you're scratching your head about memory, maybe you should check out Dosu. Maybe we can help out with what you're working on.

Thanks, everybody. See you next time.