Stop Re-explaining Your Company to AI Agents

In our fireside chat with Pydantic, Taylor and I joined Laís Carvalho, Douwe Maan, and Michael Pfaffenberger to talk about where we think agentic development for enterprises is going. The full recording and transcript are on our site.

94% of the billions and billions of dollars spent on tokens today go towards re-reading the same context about companies! Every session, every agent, starting from scratch.

That's a lot of money spent on déjà vu. We are at rock bottom for token efficiency for internal agents, and enormous room to improve. I recently sat down with the Pydantic team to dig into how. (Watch the full conversation here, it's worth it!)

Short on time? Here's the recap.

The agent's main job is finding things

If you look at what internal agents spend their time on, the biggest use case is sifting through internal information: searching a codebase to debug something, answering a question from internal docs, digging through Slack. A lot of what people call memory is really about learning where to find things, so the next agent searching the same corner of the company gets there faster.

That's why we think of knowledge as a cache. Some context is available somewhere in the organization but expensive to reach. An agent might spend a lot of tokens mapping the API endpoints for a service or working out which services depend on it. That result is worth caching, and once you've cached it, you're responsible for keeping it current.

Where agents learn from

Human corrections are the obvious source of things worth remembering: someone tells the agent "remember this" or "do it this way next time." But a lot surfaces on its own inside agent sessions, even fully autonomous ones:

• Hard-won context. The agent spent a lot of tokens to find one small piece of information. Cache the information, or a faster path to it.
• Mistakes and backtracking. Agents are stubborn. Given a goal, they'll keep re-verifying until they get there. Where they went wrong and had to back up is worth learning from.
• Unproductive tool calls. Places where an agent used tools inefficiently or made the same mistake repeatedly.

One of our own examples at Dosu includes an agent that helps with issue triage. At first, it kept getting our database table names wrong every single time it touched the database. But after learning that our tables use singular names instead of plural ones, we saw a pretty big difference and improvement in its ability to triage issues.

From personal memory to team memory

Two years ago, agent memory mostly meant personalization: learning a user's preferences. Most out-of-the-box harnesses are still mostly there. At a company level, the more important thing is team knowledge: common workflows, things people look up all the time, and mistakes made in the codebase, in the context of everyone on the team.

That makes scoping one of the hardest and most important problems. Is a piece of information specific to one user? To a branch, where it's still in flight and not yet shared knowledge? To a repo, a team, or the whole organization? Deciding where a piece of information lives, and where it applies, matters as much as capturing it.

Agents don't trust every source equally

Douwe posed a question: if you cache knowledge, are you just building a second copy of the source of truth? Where knowledge lives turns out to matter more than I expected, because models trust some sources more than others.

Current frontier models are surprisingly skeptical of tool calls. Show an agent the same fact from an MCP tool call and from a Markdown file in the repo, and it's more likely to trust the Markdown file. Knowledge injected into the session through native hooks in harnesses like Claude Code and Codex is also trusted more than a tool call. So instead of telling an agent to call a tool, we've been moving toward taking the task, distilling the relevant knowledge in the background, and injecting it into the session. The agent keeps doing what it's trained to do, with better information along the way. If you tell an agent to call a tool, it will. Whether it trusts what comes back varies.

Measuring whether knowledge helps

Most memory benchmarks measure recall: did the agent retrieve the right piece of information? That made sense when agents were chatbots. When agents do work, finding the right information matters less than whether they used it and whether it changed the outcome.

We measure that at a few levels. One is knowledge reuse: how often agents are pulling from the knowledge layer, which tells you about efficiency. Another is analyzing the agent's own trace afterward: given a piece of knowledge that was injected, did the agent trust it, use it, and did it show up in the final result? A third, which we haven't done yet but I've seen other companies do well, is asking the agent at the end of a session whether a tool call or a piece of information was helpful. And there's always the vibe check: if I corrected my agent once, I expect it to do better next time.

Our evals come in layers. The full end-to-end benchmark simulates a company over time, because the order things happen in matters, and it's expensive and slow, so we use it sparingly. Below it are cheaper, more classic evals: whether we extract the right knowledge from a session, and whether our retrieval step returns only the chunks that are actually relevant.

Pydantic Logfire has been a big unlock here. Because we have decent production traffic, we can use agents to pull real examples out of Logfire and turn them into eval datasets, and sometimes shadow-launch a change to collect data before it ships. Before, we'd have hand-curated or synthetically generated a lot of that.

It also changed which models we use. Michael asked whether we'd tried System One models for extraction. We switched once they came out, and on our benchmarks recall held steady while P50 latency dropped by roughly half. Jev is still the best one in our benchmarks on both cost and latency.

Memory is model-specific

Someone asked how to keep agents reliable when a provider retires a model. Part of the answer is that which memories help, and how much, varies by model family. Something that's hard for Claude might be easy for GPT, and the other way around, much like what's hard for one person is easy for another. We use one consistent model for extracting memories, which gives us a stable view of what gets captured even as the agents consuming it change. For rolling out new models, the answer is evals, and especially defining what actually matters to you. Some of our customers test new models on how well they work like an engineer at their company: following its style preferences and reaching for internal tools a model wouldn't use on its own.

If you don't know where to start, just start simple

Douwe closed by asking for advice for teams building this themselves. Every company will end up needing a shared context layer of some kind. Otherwise you get a mess of separate connectors and one-off memory systems, where Claude knows some things about your business, your internal agents know others, and your code review agent knows something else again. You want one central source of truth, and you want to own it, or at least make sure it's exportable and shared across your agents.

My advice is to start simple. Put it in a Markdown file in your repository, or use the data you already have in Confluence. It will break down as you scale and hit things like in-flight knowledge and scoping, and by then you'll understand the shape of the problem well enough to know what you need.

Douwe's version, from building Pydantic AI, is worth repeating: build for the agents of six months from now, which will be far smarter and far cheaper than today's. Don't add complexity you don't need yet, because the models will probably make it unnecessary. Keep it simple still applies.

As the cost of intelligence keeps falling, what makes a company distinct is its knowledge and the way it gets work done. That's worth curating on purpose, and it's what we're building at Dosu.