Agent Memory: What Happens When It's Wrong?

Episode 5 closes Dosu's agent memory series with maintenance. What does it take to keep memory useful once the codebase, the team, and the organization move on without it. Taylor Dolezal, Devin Stein and Michael Mangus cover why time, scale and contradictions make maintenance the hardest module, what gets lost each time a memory is re-summarized, and the primitives a team can reach for, like time decay, TTL eviction, scheduled "dreaming," update-on-insert, reacting to external events like merged PRs, and self-correction. They close on the two questions every memory system eventually faces on how to measure whether maintenance is working when today's benchmarks don't, and how much an organization is willing to trust AI to manage the problems AI creates. The series works through Are We Ready for an Agent-Native Memory System? (Zhou et al., 2026); episode 1 covered the four modules, episode 2 storage, episode 3 the write side, and episode 4 retrieval. This transcript has been edited for length and clarity.

Transcript

00:00 · Introduction

Taylor: Your agent recalls something about how something works, and then your team makes changes to the codebase. How does your agent figure out what happened? That's one of the things we're going to be talking about today as we wrap up our agent memory series: what's required to keep memory useful over time, and how to evaluate whether our efforts are working or not.

Devin, jumping right into what makes maintenance difficult as it pertains to agent memory — can you talk about how we keep agent memory up to date?

00:36 · Why maintenance is the hard part

Devin: Maintaining memory is arguably the hardest part of memory, and it boils down to time. As an organization, time moves forward and things change. Scale as well: as the size of the memory grows, there are more places for contradictions to appear and for overlaps to happen within memory, and there's just general entropy within an organization. As things change, your memory has to keep up. So: time, volume or scale, and then the contradictions that appear as part of that.

Taylor: What can go wrong if you don't continue to update your memory? Are there specific failure modes you've seen? Many of you out there have seen knowledge cutoffs and working with bounded knowledge. Any thoughts on the failure modes if you don't have something in place for maintenance?

Devin: If you don't maintain memory, it's going to be stale. And stale memory, if it's incorrect or misleading, is arguably worse than no memory. You lose a lot of the value if it's not correct. So maintenance is critical to actually putting agent memory and knowledge into production.

02:02 · What gets lost when memory is rewritten

Taylor: Shifting gears to what changes as memory gets rewritten. There's a decision we need to make about new data updating current memories. Michael, what are some of the things that can get lost as an agent keeps summarizing and updating the same memory? Is that the right approach?

Michael: This is interesting because LLMs and humans have a similar behavior here. Something some people don't know about your own memory is that you don't have an exact copy in your mind of what you experienced when you think back on it. There are schemas you apply to how the world works, and whenever you remember something, it becomes malleable. You recall it through those filters, and you have to re-encode it and store it back to remember it next time, and it'll change. So, ironically, the more often you remember something, the less faithful your memory of it probably is.

LLMs have essentially the same behavior. We talked in previous episodes about the risks of iterative summarization. If what you're asking an LLM to do is "here's a summary of what we knew so far, here's a new event, update this summary," the problem is that the summary is in and of itself a lossy recollection of what you had seen before, and we know that over time that leads to a loss of nuance. Maybe there's some small detail that wasn't attended to in your initial summary that you would realize is important when you start seeing it come up again and again in future episodes you're trying to consolidate into that memory.

One of the things we do at Dosu that I think is really helpful is we try to avoid that iterative summarization and keep around the actual raw inputs from all the individual episodes we've recorded over the history of building the memory. So when we re-summarize, it's not seeing a summary and a new event; it's seeing all of the events, and it has the ability to identify patterns that only emerge after some number of individual episodes have been recorded. That's a really important insight to me in terms of memory maintenance: if you want the most faithful memory, you need to keep that exact history as much as possible.

That's also one of the findings in the paper that kicked off this series, Are We Ready for an Agent-Native Memory System? One of the things they determined is that the systems that keep raw history around tend to be the ones with better stability in the long run, if I'm remembering correctly. So it's definitely something we've noticed and incorporated into our product.

04:51 · Storing memory so it can be updated

Taylor: We've talked in the past about how to read memory and how to write it. With maintenance in mind, is there anything we should be doing when building an agent memory system that's going to help with updates? We've talked about taxonomies and why they may or may not be the best approach, but I want to dig into what makes storing things easier as it pertains to maintenance.

Michael: This is another finding in the survey paper we've been talking about: systems that bake in more assumptions about the structure of memory tend to be more cost-efficient on update. We've addressed this in previous episodes as well. The stronger an opinion you have in the schema, the typology for how you store memories, the easier it is to identify what needs to change — especially if you have a graph-style structure. It gives you more constraints to operate under. Although, as we talked about, what you're trading off is that you then have to force things into that typology to be able to learn about them, which has its own disadvantages.

I think what a lot of people have come to is that you need multi-factor memory: different ways of storing and encoding the memory. What you struggle with for update is the same thing you struggle with for retrieval — find me the previous memories that are relevant to this new information. So if you have strong retrieval, you can do more effective traversal of the memories you have stored across all those different representations to do your updates in a cost-efficient way. It's something we've struggled with a lot. Devin had a good LinkedIn post earlier in the week showing just how much cost savings we still have possible in our knowledge update pipeline. So cost trade-offs are really at the heart of it, and I think a lot of people struggle with them.

Devin: I'll add to that. If you think about the challenges of maintaining memory, one of them is scale, size, volume. Related to the conversations we've had around retrieval, scoping is basically a way of partitioning your knowledge, and that's also a way of making a large memory database smaller. When you're trying to update, you're updating a smaller subsection — maybe it's a team-specific memory or a user-specific memory — and you're not always looking at the global level. Partitioning knowledge is very important for how you store it for updates as well.

07:44 · Primitives for keeping memory fresh

Taylor: There are a few ways to try to keep memory current. Devin, let's say a team is building an agent memory solution in-house. What are some primitives, or ways to think about this, that teams can use for ongoing memory maintenance to keep those memories fresh?

Devin: Great question. I'll start from the most simple and move to more complex.

At the most basic, memory maintenance is around decay: time-based decay. You make an assumption that the organization is moving relatively fast, so you give more weight to something that was said today than something said three months ago, and gradually, organically, older memories hold less meaning during retrieval. Similar to that, on the cache analogy we like to use at Dosu for a knowledge cache, you can decide that you're only going to maintain memories that are actively being used. You have a time-to-live for memories, where if a memory isn't being used or accessed, it gets evicted from your knowledge — say, any memory not accessed in ninety days. The benefit of these approaches is that they're really cheap. The cost of this maintenance strategy is very low; it's purely computational and doesn't require an LLM in the loop. But it's not perfect. There are things that don't come up for ninety days and then come up a year and a half later that are really important, and you want to remember them.

Another popular approach to maintaining memories, and also creating them, is dreaming — that's the term that's been used. Like a person, it happens on a scheduled basis: maybe once a day you run a job that reviews all your memories and does a consolidation and reassessment of their importance. If there are duplicate memories, maybe you merge them. If there are conflicting ones, maybe you surface that or choose the more recent one. You're trying to prune and manage memory on a schedule. Unlike the more algorithmic approaches, this is actually pretty expensive, especially at scale. If you're dreaming over everything, it can get very expensive. Some strategies we've seen are to dream only on recent memories or only on older memories — some partitioning of what you review on a schedule.

Then there are more real-time approaches. One is updating on insert. Every time there's a new memory, that's an event, and it gives you an opportunity to reassess related memories. If we form a new memory about how we do knowledge reviews at Dosu, we can look at what existing memories we have related to knowledge review. Are they conflicting? If so, maybe evict the older ones, or rewrite them given the new information. That's a simple type of event: a new memory has formed.

The reality, though, is that one of the challenges of deploying agent memory at scale in teams is that there's a lot happening outside of the agent. Coding is one of the unique places right now where knowledge work actually gets done end to end by agents, but that's not true across the entire organization. So the other form is ingesting external events and having triggers: this PR merged, like we do at Dosu — how do we update knowledge and memories accordingly? A conversation was resolved in Slack or in a support thread — what does this mean for memory? A ticket was closed in Jira. Reacting to external events is really the only way to keep things up to date in real time at organizations.

The last one, which I think is interesting, is self-correction mechanisms. If you hold the belief that memory is never going to be perfect — there will always be some mistakes — then the most important thing is to build a system that's self-correcting. When issues arise, the agent is able to correct them in real time, and so over time, if a memory is frequently used, it'll be correct.

13:00 · Relatedness, and what age doesn't tell you

Michael: I want to pick up on the relatedness idea in Devin's answer, because, as he said, age and importance are not necessarily correlated. One of the nice things about graph representations, in addition to the maintenance cost benefits I talked about earlier, is that they build in this concept of relatedness, and that can give you other measures of importance — graph centrality, power-style measures — to help you understand which nodes are significant on dimensions other than their age. That's another example of where the organization of your memory can go a long way to answering questions of importance and relatedness that other ways of storing don't represent.

13:49 · Invalidating memories: the cost calculation

Taylor: You've both covered some of this around deleting, versioning, and the cost of maintaining memory. Devin, as that corpus of knowledge changes within your team or organization, how do we invalidate those memories? Our favorite database analogy: is it a soft delete or a hard delete? Are there things you still want to keep around to compare later?

Devin: We've touched on this in previous webinars. Using the database analogy, maintaining memories is like maintaining an index in a database: you're doing it to make reads optimized, to be able to retrieve that information, but you're paying a price for it on writes. It's expensive to maintain over time, so there's really an ROI calculation.

We've talked before about necessary knowledge versus functional knowledge. Some knowledge came out of a human and can't be recomputed. But other memories are learned through agents doing work — maybe spinning its wheels trying to figure out how to run a dev container. Those memories can be recomputed; it just takes tokens to do it. For those, there's a cost calculation: how much am I spending to maintain this memory? How much did it cost to create it? And what value am I getting out of it — how frequently is it retrieved, and how many tokens does it save downstream?

15:36 · Measuring whether maintenance works

Taylor: That brings us to one of my favorite questions in any technical meeting: how do we measure success? How do we know maintenance is helping? Michael, do any of today's benchmarks speak to maintenance, or where do they fall short? Have you seen anything compelling?

Michael: I think this is a major shortcoming when it comes to agent memory, because most benchmarks, especially the ones you see in academic work, don't measure coding agents on realistic workloads or over long time horizons. They'll report conversational recall percent, or the ratio of tokens stored in memory versus the input and call that a token saving, or they'll compare a full-context baseline with the entire history in the context window every turn against their memory system. I don't think any of those tell you about the real-world outcomes or the business outcomes you probably want if you're adopting agent memory.

There are some interesting projects. SWE-ContextBench is one whose data I've been using a little in some of our internal evaluations at Dosu; it tries to measure cross-task transfer over time — a collection of coding agent traces with shared themes and shared work across the different tasks that you can evaluate on. But longitudinal research is expensive, and it's really hard in a fast-moving market where the agent harnesses and the model revisions are changing underneath you. Academic work needs experimental control to test a theory, but commercial applications need, first and foremost, ecological validity: an experimental setup that's realistic to what will happen for companies adopting the system. I don't know if anybody has a good way to measure that right now.

I mentioned that memory systems sometimes report token savings in terms of how much the memory compresses versus the raw input. But the effective token saving you get depends on how an agent uses that information, and maintenance is a big factor there: staleness can easily send an agent down the wrong path and consume a lot more tokens by forcing it to verify and determine for itself that what was in the memory was incorrect. Those are things you don't typically see in benchmarks, but they're all over real-world applications.

I read a report from SAP on the internal memory system they've been building for their organization — it came out a couple of months ago. It's interesting because they have measures of memory throughput and scale, how many memories they're writing. But they're not reporting — and they explicitly say this — the relevance or downstream utility of the system. It doesn't measure any tangible business outcomes. I think a lot of people are in that boat. We can look at our memory system and say, well, it is storing memories. But to what extent are agents using and getting value out of those memories? To what extent are those memories shifting over time and creating counterproductive recall?

So memory, in a lot of cases, still needs to prove that it has an impact over doing nothing. There are a lot of studies that seem to indicate that many of the memory systems out there actually have a net negative effect on performance. One of the things we try to do at Dosu is build real-world-accurate simulations, both for our own internal development and when we're doing POCs with customers, to make sure we're measuring the business outcomes people want from their coding agents, and not just navel-gazing on whether the memory itself is following the instructions we've given the system for how to build the memory.

19:45 · Reading the impact from agent sessions

Taylor: It's a lot to consider. Devin, anything to add? Are there other ways you think about maintenance and agent memory that Michael didn't cover?

Devin: From an impact perspective, the only thing I'd add is that another way we've been looking at it is qualitative feedback. There's a lot you can measure, but an amazing thing about agents and agent trajectories is that you can actually analyze what happened. We're not quite there yet at Dosu, but you could imagine forking sessions to see what would have happened — say agent memory is injected; fork the session, run them in parallel, and see how they differ.

Even outside of that, you can analyze sessions, look at the impact of memories, and get a pretty good read on whether it was helpful. Did the agent trust the memory? Was it correct? Was it harmful? We've been working on ways of productizing this — how to give that insight to users. It's useful product feedback for us, and it's also cool to see those wins in the product: as you're using your agent, it can surface a highlight like, I was able to do this because of this memory that was surfaced.

21:10 · Trusting AI to manage the problems of AI

Taylor: As we round out, one of the conversations we've had internally this week is about trust. With maintenance, you can be really diligent and inspect everything, which takes a lot of time and money that not a lot of people have. Michael, when it comes to trusting technologies, are there examples, historical or recent, where this has helped or hurt people and companies?

Michael: Part of the problem is that with coding agents there is so much volume, and one of the struggles of maintaining a memory system is the scaling problem. As it gets bigger and bigger, it's harder and harder for a human to introspect it. People aren't able to keep on top of manually reviewing it.

It reminds me of the traffic light. When cars were invented, traffic got a lot higher in volume and a lot more dangerous, so it created a problem. Initially we tried to solve that with human solutions, like police officers in the intersection directing traffic. And there was a lot of resistance to the automated traffic light when it came out. People thought no human was going to respect the traffic light — why should a traffic light tell me whether to stop or go? I can look at the intersection and judge whether it's safe. They didn't think other drivers would respect it either; they'd respect the police officer directing traffic a lot more. So we had a technology that created a problem, we developed a technology to address that problem, and what had to be overcome was not that the traffic lights didn't work, but that people didn't want to put their faith in the traffic light, and they didn't want to give up control.

I think we see that a lot with AI. Unfortunately, we're at a scale with AI where you have to trust AI to fix the problems from AI. I don't know if many people have a better answer than that right now, and it can be a difficult pill to swallow: accepting that this may be a system where human control is not something that can scale in the long run. Organizations have to decide how comfortable they are with that, and a lot of people are not totally comfortable with it yet.

What we try to do in our product is have more of a forgiveness-than-permission model. Rather than ask people to manually review, we'd rather make it easy to correct mistakes and fix the things you think are wrong, rather than putting all the burden on humans to review at a scale where it's just not feasible. The agent traces are produced faster than you could ever process and consume the memories that come out of them. So I think that's a big open question for a lot of organizations: how much faith are they willing to put in AI to manage the problems of AI?

24:10 · Closing

Taylor: I like that. Any closing thoughts, Devin, as we round down?

Devin: I think Michael said it well. We're getting to the stage where we're trusting agents to write code and build products for us, and I think knowledge is following the same path: we're going to have to trust agents to manage and build their own knowledge for your organization.

Taylor: It reminds me of what I was taught growing up: inspect what you expect — which is very difficult to say five times fast. It's been fun talking with all of you about agent memory and everything there is to consider. Next up, we're going to dig into measurement, benchmarking and evals, and we'll be sharing that out with you all. Devin, Michael, thank you so much for talking through all of this with me and with our community. Thank you all for watching and joining. You can find all of the series transcripts and everything else at dosu.dev/webinars, and we'll see you next time to talk a little more about agents. Have a good one. Thanks, everybody.