How to Build Agent Memory: Search & Retrieval

This post is based on our Agent Memory: How Does It Find Anything? webinar. Watch the recording or read the full transcript there.

In the previous post, an agent spent forty minutes debugging a problem before finding the answer. We looked at how to capture that discovery so the next agent could avoid the same dead ends. But saving the answer only helps if another agent can find and use it when the problem comes up again.

That's less straightforward than it sounds. The next agent might describe the symptoms differently or have no idea which component is responsible. Even if it finds the right note, it might still repeat the investigation because the note doesn't give it enough evidence to act on.

This fourth post in our series on how to build agent memory follows that discovery through the read path. With the building blocks, storage, and knowledge capture covered, we'll look at how to find that saved knowledge and put it to work.

Start with what the agent needs to know

Models bring a lot of general knowledge to a task. They can explain a common error, recognize a programming pattern, and suggest ways to investigate a failure. But that doesn't tell them why your team kept an unusual workaround or what changed in your deployment yesterday.

Private information, recent changes, and local context have to come from somewhere. Retrieval gives the agent access to information beyond what it learned during training.

Before choosing a search tool, though, consider what the agent is trying to find.

“Where is this function defined?” has a fairly precise answer. “Have we seen this failure before?” requires recognizing a similar situation, potentially described in different terms. “Tell me about this project” asks for an overview, with very little guidance about what matters.

That last example came up in Dosu's early days. Someone would ask for a project overview, and an agent could interpret it as an invitation to investigate the entire project. A sentence or two about its purpose would have been enough.

These requests need different retrieval paths. An exact lookup needs a match. An investigation may need several searches. An overview needs a useful level of abstraction before the agent starts collecting details.

Match the search method to the question

In the first post, we compared retrieval routing to a database query planner. A database has several ways to find data and chooses a path based on the query. Agent memory has a similar problem.

The search methods available to us solve different parts of it.

If the agent knows the name of a function, grep or rg can be an excellent starting point. There's no need to translate an exact symbol into a semantic approximation. Keyword search also remains useful when the agent knows the vocabulary used in the relevant notes.

Embeddings help when the wording varies. The previous debugging session might have described a connection expiring, while the new agent is investigating requests failing after a period of inactivity. Semantic search can help connect those descriptions. Whether they refer to the same underlying problem still needs to be established.

A graph helps when the question is about relationships. If you have a representation of which functions call which other functions, finding a caller can be a traversal instead of another round of searching and interpreting code.

Agentic retrieval lets the agent build on what it learns. It might find a troubleshooting note through semantic search, pick up a function name, locate that function with a text search, then inspect its callers. The first result gives it the vocabulary for the next query.

So agentic retrieval can use all the other methods. Its value is in adapting the investigation as information arrives. Its cost is the work involved in each additional step.

Start with the tools your agent already has. Watch where it struggles. Repeated vocabulary mismatches suggest a different problem from repeatedly tracing the same relationships. Those failures tell you what to improve.

Retrieve information the agent can act on

Finding the relevant memory gets you part of the way there. The agent still has to use it.

The harness matters here. Its instructions and tools influence what the agent trusts, what it checks, and how it approaches a task. A coding agent with a local checkout can inspect an implementation or run a test to validate what it has read.

In the webinar, Michael described an observation from our internal evaluations at Dosu. Memories explaining how to accomplish a task have been especially useful for the harnesses we're testing. An agent may get more value from a procedure it can follow than a general description of what a subsystem does.

Consider a hypothetical note saying that a service uses a connection pool. That's relevant to a connection failure, but it leaves most of the investigation to the agent.

A more useful note might explain that failures after an idle period have previously come from reusing expired connections. It could describe how to reproduce the condition, which test exercises it, and where to find the original investigation. The agent now has a concrete hypothesis and a way to check whether it applies.

This connects to the questions from the knowledge capture post. Where is the memory true? What evidence supports it? Would knowing it change a future action?

Those details also matter when reading. They help the agent decide whether it has found guidance for this task or an account of something that happened elsewhere.

Don't give agents homework

Suppose the agent retrieves the connection troubleshooting note alongside several loosely related documents about network architecture, migration plans, and possible future improvements. Now it has more things to investigate. It may check whether the migration happened, explore an unrelated dependency, or expand a small fix into a larger change.

Each result can influence what the agent does next. Even accurate information can be a distraction when it doesn't help with the current task.

That's why precision matters. How much of what you return helps the agent complete the work? How much introduces another thread to follow?

The write path already makes one selection about what deserves to become memory. The read path makes another about what belongs in this session. A discovery can be worth keeping and still be irrelevant to most tasks.

This also changes how we should evaluate retrieval. Finding the note is a useful signal, but we need to look at the rest of the run. Did the agent avoid the old dead ends? Did it reach a correct result with less work? Or did retrieval add time and tool calls without changing the outcome?

Some validation is necessary. A previous failure doesn't prove that today's failure has the same cause. Good memory should help the agent focus that validation on the relevant uncertainty.

Choose the right level of detail

In the storage post, we separated historical artifacts from maintained knowledge. The artifacts preserve what happened. The knowledge layer gives the agent a concise account of what we currently believe.

Retrieval needs access to both, because different questions need different levels of detail.

For an overview of a codebase, a summary of its main components and how they fit together can be useful. For a question about a specific implementation, the agent may need the exact code, test output, or discussion behind a claim. A summary can omit the condition that makes an explanation applicable.

Sub-agent handoffs have the same tradeoff. An orchestrator can delegate an investigation and receive a short report, keeping all the intermediate searches out of its own context. But the report may leave out a detail the orchestrator needs. Returning a reference to the relevant source or an artifact from the investigation gives it a way to recover that detail.

Keeping raw material alone doesn't solve every problem either. Some patterns only emerge when you consider multiple events together. Several separate debugging sessions might reveal a recurring weakness in how a system handles retries. No individual session necessarily contains that broader conclusion.

Abstraction is useful work. The goal is to make that understanding available while preserving a route back to the evidence when the question demands specificity.

Follow one memory through a real task

The previous post suggested starting with your next difficult debugging session. The next step is to watch what happens when another agent needs that discovery.

Does it know to search? Can it find the memory using the symptoms it has? Does the result explain where it applies? Can the agent use it to make progress, and reach the source when it needs more detail?

You can learn a lot from one example. If the agent can't find the note, look at the search path. If it finds the note but repeats the investigation, look at what the note provides and how the harness responds. If it wanders into unrelated work, look at everything else you returned alongside it.

At Dosu, this is part of what we're building toward. Useful discoveries should help the next agent make progress without repeating the work that produced them.

Of course, finding and using a memory raises another question. What happens when that memory is wrong? That's where we'll go next in the series.