Agent Memory: How Does It Find Anything?

Episode 4 of Dosu's agent memory series turns to the read side: why agents need retrieval at all, what coding agent harnesses actually want to see in their context window, and how search has evolved from grep to embeddings to agentic and graph-based retrieval. Taylor Dolezal and Michael Mangus cover precision and the cost of giving agents "homework," what blurs when agents build their own tools to get at information, how orchestrators and sub-agents trade specificity for abstraction, and why selectivity matters more than volume. The series works through Are We Ready for an Agent-Native Memory System? (Zhou et al., 2026); episode 1 covered the four modules, episode 2 storage, and episode 3 the write side. This transcript has been edited for length and clarity.

Transcript

00:00 · Introduction

Taylor Dolezal, Head of Open Source, Dosu: Hello and welcome, everyone. Welcome to another episode of Agent Memory. Today's title is "How does it find anything?" We're going to be talking about retrieval and routing as it pertains to agent memory — more so the read path. Why do agents need retrieval? What changes once they retrieve useful knowledge and information?

I'm your host, Taylor Dolezal, Head of Open Source here at Dosu, and I'm joined by Michael Mangus, Head of Product at Dosu.

00:33 · Why retrieve anything at all?

Taylor: Let's jump right into it, Michael. Why retrieve anything at all when it comes to agent memory? These agents are supposed to be super intelligent, right? They're the forefront of AGI. They should have all these things trained into their memory already. Why are we retrieving things?

Michael Mangus, Head of Product, Dosu: I should preface this by saying I don't believe we have anything close to AGI, but the key word in there is general. And I think that's actually the ultimate answer to your question. LLMs are trained to know about things that are generally true. They're supposed to build a model and understanding of the world that is broadly applicable. In fact, that's part of why they're so valuable — they're few-shot learners. You don't need to constantly retrain an LLM. It used to be that if you wanted quick, iterative learning in a machine learning system, you had to retrain nightly with new data. LLMs are great because you don't have to do that. You can feed the extra information through the context window, and the model can use its general understanding of semantic meaning to make sense of a new problem it hadn't previously seen.

What that means, though, is the training doesn't know anything about your specific time, your company, and so on. You're always going to need retrieval for information that's private. If I have an internal customer or company policy, that's not going to be in the training data. Similarly, things that change a lot — stock prices, the weather — are inevitably not going to be represented in the training data. The model has to use tools to bring those things into its context window.

So that's ultimately what it comes down to: things that are not publicly available, not knowable at training time, or just specific to you as an individual person or company. That's never going to be represented in the model itself. It has to come in through retrieval.

Taylor: That's helpful. We've talked in previous sessions about parametric memory, about ways to extend it through fine-tuning, and — like you just shared — about what's constantly changing. When does it make sense to just use tools for time, temperature, or other facts across an organization? And when does it make sense to recompute? I'm curious to see what develops on that front.

03:23 · What the harness wants to see

Taylor: Moving forward — when we try to find evidence, facts, artifacts, useful things to pair with our agent sessions and runs, do you have any thoughts about what kinds of information are useful for specific tasks? I know the answer is likely "it depends," but what can we consider as we're working with agents and trying to pull in the right artifacts for the task at hand?

Michael: That gets into a harder problem, which is: what does the agent harness want to see? Like we mentioned, LLMs are heavily influenced by — and valuable because of — how they can be steered through the context window. But you're not the only one contributing to that context window. You have an agent harness that's also adding its own prompting about what the goals of the agent are. And increasingly, models are being trained to follow the kinds of paths those harnesses want them to follow. That's a moving target. It's evolved a lot over the past year or so, as coding agents have really become capable of doing real engineering work.

What I see most today is that harnesses are very focused on validation. They don't just take your word for it. If they have a clone of the code, they're more inclined to actually execute code to test something rather than read documentation and trust what it says.

What I've noticed when I do internal evaluations on the memory systems we're building at Dosu is that, as a consequence, the most valuable memories for current harnesses seem to be steps to do a task — how to accomplish a certain thing. They're much more concerned with how something works than with what it means: a definition, a documentation page describing the thing. The agent would rather know how to use it and then use tools to directly manipulate the thing it's trying to learn about.

With code, that's a lot easier than it is with things that are more abstract or, let's say, physically embodied. We're lucky, working primarily with code, that agents are actually able to use tools to validate the information they retrieve. But it's tricky.

06:17 · Don't give agents homework

Michael: I think one of the hardest problems for memory is: under what circumstances will your memory convince your agent to trust it, versus simply giving your agent more work to do? One of the things you have to avoid in retrieval is putting information into the context window that causes the agent to take additional action that may be unproductive or unnecessary to complete the task. It really speaks to the value of precision in retrieval. You want to make sure you don't end up giving agents more homework, essentially.

Taylor: I like that. I pair it with when I used to play RPGs. It was really fun to work on the side quests, but if you do too many of those, you stray from the main story path — which is what we want our agents to focus on.

Michael: I want my agents doing a speedrun, not hundred-percent completion, personally.

Taylor: And in a video game you pay with time. With your agents, you're paying with time and money.

Michael: And complexity, and understanding. Part of the issue with LLMs is they're post-trained to be extremely accommodating and agreeable. If you suggest the possibility of something to an LLM, it will generally pick up on that thread and try to act on it. The consequence of too many side quests can be overcomplication. In fact, the number one thing I struggle with in trusting and using the output of coding agents is just getting them to do simple things. The more instructions you give them — whether through your own prompting or through retrieval in a memory system — the more things they're going to build, and the more complex a structure they're going to create.

Taylor: Anecdotally, I've seen that firsthand. When I tell my agent something that's just objectively a fact — time, date, place — a lot of the time the output comes back with, "I can't verify this, but sure," and then it carries out the task. But there have been other times where I've created a small tool, or introduced an MCP tool call, and I've had better success. That pairs with what you said about agents wanting to prove things. At least anecdotally, it makes more sense to give them the tools and get them into that workflow rather than force-feed them knowledge. I haven't seen much research on this yet, but I feel like that's going to be helpful knowledge to have over the next couple of months.

09:06 · Parametric memory and LoRA

Michael: I should mention, we started off by talking about when you need retrieval. There are some situations where it's unavoidable. But increasingly, when it comes to things like tuning for an individual engineering team's preferences or a specific company's policies, a lot of people are pursuing that from the perspective of parametric memory — where you change the underlying behavior of the model directly, instead of trying to prompt your way to the result you want.

In other words, instead of doing retrieval and adding information into the context window to steer the agent, you can use low-rank adaptation — basically attaching onto the attention mechanism of the model to change its output probabilities in a way that more closely aligns with your team's way of working, your engineering policies, and so on. There's really interesting work on how to bypass the context window entirely and just change the model itself.

Taylor: I like that pluggability with LoRA — you don't have to completely retrain the model and spend hundreds, thousands, hopefully not millions of dollars to get something useful.

10:24 · How search has evolved

Taylor: As we think about agents and searching — where is the memory stored? How do we retrieve it? Can you give me a bit of a deep dive into how search has evolved over time? Dating myself, I remember doing keyword searches on AOL and other early search engines. Fast forward to today and it looks completely different.

Michael: Right — in the library class in elementary school we learned all the operators. You don't see much of that these days.

I think there are many search strategies, and all of them have some legitimacy in terms of how things have unfolded over the time I've been working on systems like this.

There are plenty of systems that still just do substring matching, or substring matching with a little fuzziness. That can be very effective for certain kinds of problems. Essentially every time you see your agent call grep, you're doing a pattern match. If I know the exact name of a symbol in code that I want to find, grep is an excellent tool for that.

Keyword search can be more sophisticated than that. Often you'll process the terminology you're searching over — stem words, take the suffixes off, strip the derivational affixes — and so on. That still has value in modern search. Keyword search is very useful for certain kinds of problems.

These days, of course, there's a lot more focus on embeddings. Instead of looking for a specific word or fragment of a word, I convert a word into a location in a semantic space. When I query, I convert my query into a vector in that space, find the similarity between the document vector and the query vector, and that gives me my best search results. With LLM embeddings you get the added understanding of the comprehensive meaning of all the terms in your query together, so you can do much more natural-language queries. You can ask the things they taught us not to do as kids — type a whole sentence into the search engine — and it works.

Because LLMs came along with very high-quality embedding models that have a good understanding of fine senses of meaning, a lot of people reached for embeddings as the main search strategy early on in the development of coding agents. That's appropriate if you have fuzziness in what you're looking for, or if natural-language querying is important to your use case. A lot of that depends on whether there's a human on the other side or not. Agents love pattern matching because they're very good at writing regular expressions. Most humans are not great at writing a regular expression off the top of their head — they'd rather type a sentence.

So when we talk about agent memory in particular, the value of vector embeddings tends to be in document similarity. When we do memory consolidation, for example — embedding fragments of a trace and comparing them to previous memories we've built — that's inherently fuzzy. I wouldn't want to do that with a keyword match or an exact string match.

Agents calling tools also have the capacity to do agentic retrieval: building up a sequence of command-line operations to pattern match, list the contents of directories, and so on. That's really good when you have multi-hop problems. It's not about finding one document that's the best match for a query — it's about spanning a sequence of relationships. I don't know what I don't know yet. The first document I retrieve might point me to the name of a symbol I need to find, then I search for that, then I need to look at its callers or the functions it's calling, so I build more searches around it. Traditional search has really not been able to do that — following up with multiple searches in sequence. We talked about this in a previous session: there's been a lot of emphasis on shifting from vector embeddings toward agentic retrieval over the past year.

And we haven't even touched on graphs. If I know the name of a specific symbol, keyword or pattern-matching search can be great. But a graph built around all the symbols of my code can also be really useful for the kind of thing I was describing — callers of this, things this calls. Those structural relationships are easy to traverse in a graph in a way that would be hard to retrieve out of keyword search. You'd have to do multi-hop agent work, which is very expensive.

So there are many strategies, and I'm sure there are others I didn't describe. You have to ask yourself: what do I know? What is the query? What kind of results do I want? Do I want exactly my specific thing, or adjacent things too? How likely am I to find what I want in a single result? What's the probability I need to build on what I learn to actually produce usable information? Good agents use all of those strategies to some degree.

16:55 · "Tell me about this project" isn't a query

Taylor: I like the happy path you laid out. One thing you remind our engineering team of from time to time is to think about the failure modes too. In Dosu's early days, a lot of people testing Dosu would go to an "ask" endpoint and say, "tell me about this project." Those are the kinds of things to watch out for, because if your agent goes and looks up the entire history of a project, trying to do a deep-research task, when what you wanted was a sentence or two about what the project does —

Michael: That raises an interesting point. I said a second ago that you have to think about what you already know and how many steps you'll need to retrieve the useful information. A prompt like "tell me about this" — there's not really a query there. There's not a specific question. I think you could argue that's not even a search. You're giving an agent a command, not asking it a question.

The tools you need in your memory to answer "hey, tell me about this project in general" are much more about abstraction and conceptual understanding. That's somewhere I do think knowledge graph representations are very useful: you can imagine finding the most central nodes in the graph as a heuristic for the key concepts a person would need to understand to have a conversation about the project. Those things are relevant to retrieval, but it's debatable whether they're even in the same category as, "hey, I want to know about the implementation of the XYZ class."

19:16 · Permissions, sandboxes, and agents that build their own tools

Taylor: You talked about regular expressions, and a lot of this agentic search makes me think about metaphors. I keep going back to databases and the very old adage: sanitize your inputs. Make sure you're interacting with your database in a safe way — you're not putting in user-supplied input unknowingly, in a way that could drop tables. With agents it's a very different type of guardrail system, but still important to think about. How are these elements of your workflow stacking, and how are we using them?

Michael: Increasingly, with agentic workflows, it can be hard to separate the conventional permission idea of "I want to have access to read, but not to write." What I find lately — and I think you see this in some of the high-profile incidents from the more powerful, less constrained agents — is that you give it a task and the agent is going to find a way to complete the task. The task might be, "I want to retrieve some information." But if it's not able to retrieve it through the avenues it's trying, it will build itself the scaffolding. It'll write things, build utilities for itself, manipulate things such that it's able to get that information.

I see funny examples of that when I'm doing evaluation. This came up in the blog post I wrote a few weeks back about where your money goes when you use a coding agent — where the tokens go, how much is planning or communicating with the user, and so on. One thing I found interesting there: given an environment that was limited in terms of what the agent was able to do, it would find really interesting retrieval strategies, like using the GitHub command-line tools to get around the sandboxing we'd done to only show it one specific state of a repo at a certain commit. It would call the GitHub CLI to find the actual reference PR we were having it replay in our evaluation.

They're very clever about building themselves the tools they need to get the information they think they need. And it really blurs the line in how we've traditionally thought about permissions, because I don't think anybody wants to sit there and say yes to every tool call in that sequence.

Taylor: Well, no. When's the last time you said no to a tool call an agent prompted you for?

Michael: It's been weeks. I'm not sure I could name the last time. So you raise a good point — I'm not sure what it's really going to mean to build permission boundaries on this stuff in the long run.

Taylor: I don't think I've ever done this, but it's something I'm adding to my to-do list: check your temp directory. Some of these throwaway tools the agent is creating, the assets and artifacts — session logs are helpful, but there are a lot of tools that get created along the way.

Michael: I definitely purge that folder from time to time. It's a huge source. I keep finding, when I'm doing agent evaluations, just how tricky they are — how hard it can be to isolate the variables you're trying to measure. That temp directory is a huge source of leakage. There's so much stuff that ends up there that the agent rediscovers later, that I don't want it to have on the subsequent run. Especially when you're running things in parallel, it's very hard to sandbox.

Taylor: Craft agent memory and retrieval the way you expect, but also try to anticipate the unexpected.

Michael: Right. I could do more on environmental sandboxing there, but it's an open question. People also don't want to lock down their code-execution sandbox too hard, either — then you can't get much done.

23:28 · Orchestrators, sub-agents, and the cost of summarization

Taylor: I know we're a little over time, but I'd love to end on something we've talked about internally. When you're setting up multiple agents — an orchestrator and sub-agents — how do you think about context? You can have everything in one global context, which reminds me of global variables, not always something you want. When does it make sense to create boundaries around those things?

Michael: That's another good example of how building to agent behavior has been a moving target over the past year or so. Increasingly you do see agents create sub-agents for the sake of doing parallel work on difficult or long-running tasks. The main reason is to keep the orchestrator's context window a little clearer. It delegates to a sub-agent, the sub-agent has a specific goal, and it comes back and reports just the final outcome to the orchestrator instead of all the sausage-making along the way. That's good for avoiding context bloat or context rot at the orchestrator level. But it also means you have lossiness you have to account for in how that information gets passed back up. Sometimes you get verbatim information back, or the sub-agent writes a file into the file system as one of its outputs.

It gets to a broader problem we also have to think about a lot when we're building our memory system at Dosu: the cost of summarization versus its value. If you think about it, this is very related to how you manage retrieval as well. When it comes to keyword versus vector search, you can compress information — that's what happens when I build a semantic vector. If I have a very dense embedding, I've inevitably reduced the number of individual things I can know about in that vector space, but I've gained abstraction: the power of summarizing, finding synonyms and linkages, things that vary together.

The same is true when an agent reads from a file. You can summarize it and make it smaller. You'll gain some abstraction from that summarization, but you'll lose specificity. This is actually mentioned in the original launching point for this series of webinars, Are We Ready for an Agent-Native Memory System? One of the first things they talk about is this trade-off between abstraction and specificity. It also comes up a lot in the Memora paper from Microsoft Research, which I think we've also referenced. Inevitably, summarization comes at a cost — especially iterative summarization with an LLM, which is where I think a lot of long-term stability failures happen.

So it's not just about agents and sub-agents and what that handoff looks like. It's a more general problem: when you have raw information, do you want to deliver the raw information, or some abstraction on top of it? Again, it depends a lot on the nature of the question. If the question is "explain this codebase to me," abstraction is really good. If it's a question about a specific implementation of something — specificity. It's right there in the way I phrased it.

27:19 · Raw data versus abstraction

Taylor: That makes sense. If you had unlimited budget, you'd keep absolutely everything — back it up, store it, keep the ability to access transcripts, artifacts, whatever it is. But most teams don't have an unlimited budget. So figuring out the "well, it depends," the specific use case, is so important to do before you start writing the line.

Michael: I slightly disagree with part of that. Maybe it's true that we want to be maximalist about the raw data we retain. But there are some patterns that only emerge from a collection of raw information as an abstract unit — emergent things you would not learn from looking at individual specific events in isolation, from one single document or one single episodic memory from a past agent trajectory. There are emergent abstractions that require you to build a higher-level conceptual understanding. So it's never enough to only have the raw episodes, at least if you want to be able to answer questions at that higher level of abstraction — which oftentimes you do.

Taylor: Good point. There's the state and there's the runtime as well. These emergent properties we only get to see after we have that large corpus.

28:51 · Closing thoughts: be selective

Taylor: That brings us to time. I'd love to close with any words of wisdom. Michael — for new teams starting to work with agents and thinking about retrieval, is there any more global, important thing to consider first? We've had a lot of good nuggets, but is there anything you'd advise people to start thinking about as they work on this problem?

Michael: I'd say selectivity is very important. I talked about this earlier when I talked about the value of precision in search, and again just now when I talked about how there are certain things you can only learn by abstracting over individual instances of information. A lot of people have the inclination to throw in the kitchen sink — every document we've ever generated, every issue that's ever been opened, and so on. I think pruning it, abstracting it out — I know I talked about the Library of Babel two episodes ago. If you have all possible information, it's effectively equivalent to having no information.

So one of the hardest parts of retrieval is being judicious and selective about what you want an agent to be able to retrieve, and being extremely precise and incisive about the additional information you introduce into your agent sessions. That's something we focus on a lot in the product we're building at Dosu. We'd be happy to take care of it for you.

Taylor: With that — go check out Dosu, go try it. We'd love to hear your thoughts. If you aren't working in agent memory every day, I'm certain we can help with some of the workflows you're working on.

Thank you all for joining us today, and thank you, Michael, for your time. Next week we're going to look at Agent Memory: What happens when it's wrong? — some of the failure modes and the cases that might not be so glowing. We'll see you next week. Thanks, everybody.