Stop Re-explaining Your Company to AI Agents

Pydantic and Dosu teamed up for this session on giving internal AI agents shared company knowledge instead of rebuilding context for each one. Host Laís Carvalho, who works in growth marketing and developer relations at Pydantic, talks with Pydantic AI lead engineer Douwe Maan, Pydantic harness engineer Michael Pfaffenberger, Dosu founder and CEO Devin Stein and Dosu head of ecosystem Taylor Dolezal. The conversation covers cache invalidation, hard-won context, why models trust some sources more than others, how Dosu evaluates whether knowledge helps, labeling eval data with frontier models, and scoping knowledge across teams. For more on where agent knowledge should live, see Agent Memory: Where Does Knowledge Live?. This transcript has been edited for length and clarity.

Transcript

00:00 · Introductions

Laís Carvalho, Growth Marketing and Developer Relations, Pydantic: Hello, hello, everybody. Welcome! We're here today for another Pydantic webinar. Oh, sorry, I'm not supposed to say webinar anymore. We're here today for another #NotAWebinar.

For those of you joining, drop a flag in the chat for where you are. I'd love to know where folks are from. Mexico City, Philadelphia, Austria, very cool. Not everyone is from the US. Charlottesville. Peru! Peru is so beautiful, Lima is amazing. New York City, another European, Italia. Brazil! Oi!

I'm here today with the folks from Dosu and my colleagues from Pydantic, so let's do a quick round of introductions, starting with myself. My name is Laís Carvalho, I'm originally from Brazil, and I joined Pydantic a little more than a year ago. I work in DevRel. From the Pydantic team we also have Michael Pfaffenberger, well known as the creator of Code Puppy and a senior AI harness engineer. And we have my colleague Douwe, who is joining from Mexico City and is the lead developer for Pydantic AI. For a split second I completely forgot what you do. And now I'll hand over to Taylor.

Taylor Dolezal, Head of Ecosystem, Dosu: My name's Taylor, I'm head of ecosystem here at Dosu, and I get to work on a lot of fun problems with all of you, have conversations, and figure things out across MCP, CLIs and agents. There's so much going on all the time with AI. With that, I'd love to popcorn over to Devin to introduce himself.

Devin Stein, Founder and CEO, Dosu: Hi everyone, Devin Stein, CEO and founder of Dosu, but an engineer by trade and at heart.

02:51 · What are internal AI agents?

Laís: Wonderful, welcome. Today we're going to talk about internal AI agents. Douwe, I'll start with you. Can you clarify what we mean when we say internal AI agents?

Douwe Maan, Pydantic AI Lead Engineer, Pydantic: For sure. I think most people are going to be very familiar with them, because most of us use things like Claude Code and Codex all day, and those are internal AI agents: you as the employee are using them to do your job better and get access to context in the company. You can also have them run headlessly, as automations that go and do things by themselves instead of being driven by a user.

It's useful to contrast these with the external AI agents that a lot of people build with Pydantic AI and similar frameworks for their products, like customer support bots. That's not what we're talking about. We're specifically talking about agents that use company tools, MCP servers and knowledge to both augment employees in the tasks they're doing and automate processes inside the company.

04:00 · Knowledge as a building block

Laís: Very good. Taylor, would you walk us through a workflow that shows what these internal AI agents can do, something a bit more practical?

Taylor: Absolutely. I think a better question is, what can't they do? Or at least, what aren't people trying to make possible with these internal agents? What might be helpful is visualizing some of the workflows we help people with at Dosu. A lot of people scan their codebase and try to pull in a bunch of context and knowledge, because there's always so much going on, especially when you're working on a team where everybody's got work, opinions and tickets in flight.

At Dosu, we look at code and the conversations that surround it. Rather than re-crawling, relearning and making sense of all of that information again and again, Dosu can effectively cache that knowledge. We take that knowledge and feed it to your agents, so you get the outcomes you're looking for faster, cheaper, and a little more consistently.

Earlier in my career at Dosu I was head of open source, so I looked at a lot of open source workflows. As MCP got more entrenched and solid as a specification, a lot of people moved to it, and to the CLI, as a way of building these things on knowledge infrastructure. Without naming names, some of the more interesting workflows are companies, some focused on mapping, using Dosu to ingest a lot of knowledge and then using that as a building block for what they serve to their own customers. That's just one possibility we've seen so far, but it's really interesting to see people scale this and think, what if knowledge is a meaningful primitive you can use to build these systems, especially in the world of agents?

Douwe: A question for you, Taylor, which will ultimately help people understand what Dosu does exactly. You mentioned scanning the code. Is this something people connect to their own Claude Code instances, or is it for company-built agents on different technology? How does it fit into people's workflows?

Taylor: You could even ask Dosu that question if you had it connected. Most implementations are through MCP. You connect a GitHub repository, and most people use GitHub and Slack as the main connectors for getting started with Dosu. Then you're pulling in all of that information, and that's when the fun starts. You just ask questions like normal, and Dosu is able to learn, understand and store that knowledge as you go through your daily workflow.

Devin: In terms of really common internal agent workflows, I'd say the number one use case is sifting through internal information, whether that's searching your codebase to debug something or answer a question, or internal documentation and Slack conversations. One of the big use cases for knowledge capture or memory is learning where to find things, which makes agents more efficient when they search for the same things or similar areas of your company over time.

07:57 · Cache invalidation

Michael Pfaffenberger, Harness Engineer, Pydantic: When you built Dosu, I'm sure you experienced the case where a cached bit of knowledge or memory became invalid after a certain amount of time. Humans have a very good internal reference for when a fact becomes invalidated. It hasn't been clear to me that agents have solved that, and some famous computer scientist once said cache invalidation is a hard problem. How do you think about solving that, and has an observability stack helped?

Devin: Yeah, caching is hard. It's one of the things we've been most focused on, and it's part of the reason that, from a company knowledge perspective, we've always anchored on product engineering knowledge, where the codebase is a fairly reliable source of truth. Within Dosu, we monitor your codebase for commits as very clear change events, when we know knowledge related to your product has changed. Then we use an agent to analyze your knowledge cache, the knowledge that has been built up, to see how it's impacted by those code changes. That can be applied to any sort of change event, whether it's a decision made in a Slack thread or a ticket being closed. So you have to do a pretty good job of ingesting all the external events that could mean something changed.

It's not perfect. You're always going to have mistakes, so the most important thing is making sure there's a feedback loop in the system: if an agent finds a piece of stale knowledge, it has the tools and the ability to correct it, so you only make that mistake once. And from an observability perspective, being able to analyze how agents use that knowledge and whether it was helpful is key to maintaining that cache and understanding the impact it's having.

10:17 · Autonomous agents and hard-won context

Michael: Thank you. I do have a follow-up, but I want to let someone else jump in if they have something.

Laís: I think you should go for your follow-up.

Michael: Okay, I'll go for it. We've been zeroing in on coding agents a little too much. Internal agents can simply be invoked by an event feed from some other internal system in an organization, and they may be completely autonomous. Does Dosu cover that use case as well?

Devin: 100%. If you think about the types of context that are helpful for agents to learn, there are a few. One is the obvious one, human intervention: humans in the loop correcting the agent, saying, "Hey, actually remember this," or, "You should do this next time."

But a lot comes up organically within agent sessions. One is what we call hard-won context. That's the search problem I was talking about before: the agent spent a lot of tokens to figure out some small piece of information, in which case it's useful to cache that piece of information, or a more efficient path to get there. Similarly, even autonomous agents make mistakes, then backtrack and re-verify things until they get it right. Agents are really, really stubborn. If they have a goal, they will try to get there. So we figure out where it made mistakes and where it backtracked, and learn what's important so that backtracking doesn't happen again.

Then there's what we like to call unproductive tool calls: where was it using tools inefficiently or making really common mistakes? An internal example: we have an agent that helps with issue triage, and it kept getting our database table schemas wrong. Every single time it interacted with our database, there was this back and forth where it would guess the wrong table names. Learning that our tables use singular names, not plural names, actually made a pretty big difference.

12:38 · Getting signal without traces

Douwe: You mentioned detecting inefficient tool use, or re-finding the same information over and over. If you're instrumenting your agents in an observability platform like Pydantic Logfire, that makes perfect sense. But earlier you described a user with Claude Code on their machine who just connects to Dosu over MCP. How do you get context on whether the data is actually helping the agent accomplish its goal?

Devin: Great question. When an agent is instrumented with something like Logfire, it's very easy to see the trace. Without it, we have a few options. One is you try to recreate that. I saw Claude Code recently released support for OpenTelemetry, so getting the same visibility into trace data within the sessions themselves is becoming more feasible.

Otherwise, the Dosu CLI helps you extract session data so it can be processed. We don't store your sessions on our side, but we do proxy, so we're helping process your sessions locally on your machine to extract that information. The other way is just through MCP, with agents having tools. There you rely on the agent's ability to understand when to use the tool and what information to capture, which varies a lot by model family, as well as by tier within a family.

14:16 · Caching versus memory

Douwe: I definitely want to talk about how you manage this memory or knowledge cache as the model you're using changes over time, because what it needs to know changes. But I've heard you say caching a couple of times, and a lot of what we're discussing is, in tools like Claude Code, just called memory. I'm curious whether that distinction is meaningful. It sounds like you're saying the knowledge already exists somewhere and you're keeping an index of it up to date, whereas memory files in Claude Code are often a source of truth in their own right.

And, not completely related, the title of this webinar is "Stop Building Context for Every Internal AI Agent." You could view every coding agent on every employee's machine as an individual agent, and they are, but then you also have internal workflows for different departments. In Mike's case, as we mentioned earlier, he built Code Puppy, the internal coding agent at Walmart that's used by 100,000 Walmart employees and is built on Pydantic AI. That doesn't mean 100,000 engineers. It means accountants, store managers, people in logistics, everyone is using it. You don't want every agent to be overwhelmed with all the company data, and you have access control to think about too. Some of what you're describing sounds like memory Markdown files on steroids, but I'm curious what problems and solutions you've had to work on to make it work at that scale.

Devin: Great questions. Maybe I'll start with caching, because the knowledge cache is sort of a Dosu-ism. I think it's important when you talk about certain types of context, especially hard-won context, which you see with Claude Code and other memory systems as well, where the information is available within the organization but it takes a lot of tokens to get there. What you're doing is figuring out what makes sense to cache, what's expensive to recompute. For us, the analogy of a cache, where things that are hard to find get cached and therefore should also be maintained, is just a useful way to think about memory.

Michael: So your definition of recompute is basically spinning up a sub-agent and saying, "Go check Confluence for how I do X, Y and Z," and then storing that result in the cache.

Devin: Exactly. Or map the API endpoints for this service, or what services depend on it.

That's why we like the term caching, but ultimately it is memory. The interesting thing about memory, not to go too off-topic, is that it's had an arc, and it gets to what you were saying, Douwe, about company knowledge versus agent knowledge. Two years ago, whenever you talked about agent memory, it was really about user personalization. To some extent Claude is still mostly there, and most of the out-of-the-box harnesses focus on user memory: learning how you like to work, your preferences.

But when you think about knowledge or memory at a company level, team knowledge is the most important thing. You're learning common workflows, things that are commonly looked up, or mistakes made within the codebase, in the context of everyone on the team. For that, a big focus at Dosu, and for knowledge tools generally, is scoping, which is a really hard but important problem. Is this piece of information specific to a user? Is it specific to a branch, something in flight that hasn't been committed to the shared knowledge yet? Does it live at the repo scope, the team level, or organization-wide? Figuring out where a piece of information should live and where it's applicable is one of the most important and challenging problems with knowledge, memory and organizations.

18:35 · Where knowledge lives and what models trust

Douwe: I'd take it half a step further. When you produce this cache, for example of the endpoints of some internal service, or something else that was expensive to look up, should that be stored in the memory cache, or should it go to the source of truth itself, as another document there? I worry a little about ending up duplicating a lot of things and having the memory layer become the knowledge layer in its own right.

Devin: Where that piece of information should live is really interesting. One thing we've noticed is that, depending on where information comes from, models have different levels of trust for it. It's actually one of the reasons we've started to move away from MCP as the way to read knowledge. Current frontier models are, surprisingly, a little skeptical of tool calls. They really like trusting code. If they see the same piece of information from an MCP tool call named something like read_knowledge versus a Markdown file in the repo, they're more likely to trust the Markdown file, at least in current agents.

The workaround we've seen is native hooks, where you inject knowledge into the session. At least within Claude Code and Codex types of harnesses, that information is also more trusted than tool calls. So there are little ways of getting the agent to trust a piece of information.

Taking a step back, something we're working on on the product side that I'm very excited about, without getting too far off topic, is going beyond that: where else should we store or update information within your codebase or product to make agents more likely to trust it? Maybe this information actually does belong in a Markdown file, or in your AGENTS.md.

Taylor: In a similar vein, we had some fun stories earlier in the history of Dosu. So far we've been talking about memory and where we pull it from, but on the other side, when somebody asks a question: one of the most common questions people asked Dosu early on was, "What does this project do?" That kicked off a bunch of research agents and all this work to answer something we could have gotten from the project description on GitHub. So there were many learnings there too, not just in the storage, retrieval and maintenance of memory, but in what it takes to understand the real question someone is asking, which is fascinating.

21:24 · Evaluating whether knowledge helps

Laís: Thank you, that was an awesome kickoff. I'd like to move on to how we evaluate whether all that context and caching is actually improving the agents. Devin, again: can you talk to us about evals? How do you do it, and what do we need to keep an eye on to make sure the caching is actually improving agent runs?

Devin: I'm happy to talk about how we and our customers think about evaluating Dosu, and evals for internal agents in general, which is really interesting. I'm sure Mike and Douwe can add what they've seen from customers.

Evaluation and measurement in the memory space is a really hard and interesting problem. Going back to what I said before, memory has evolved over the past few years from user personalization to organizational, shared team knowledge. The way we use agents has also changed, from chatbots, which are very Q&A-based, to agents actually doing work. If you look at memory benchmarks historically, evaluation is really about recall: did the agent find the right piece of information from memory? That's interesting from a user personalization or chat perspective. But when you look at agents doing work, whether it's a coding agent or an internal agent you've built to do some task end to end, finding the right piece of information is important, but what's more important is that it actually used that information and it had an impact.

So in the product, we evaluate on a few levels. One is giving users a sense of knowledge reuse: how frequently they're accessing knowledge from the layer Dosu is curating. That gives you a sense of efficiency. Then we analyze sessions, using the agent traces themselves to see: given this memory or piece of knowledge that was injected, was it used? Was it impactful, in that the agent actually trusted it? Was it ultimately used in the final response or the task at hand? Doing some post-processing of the session gives you a lot of signal.

Another thing we've wanted to do but haven't yet, and I've seen other companies do this cleverly, is simply asking agents for feedback. With MCP, they're fairly unbiased, so at the end of the session you ask, "Was this tool call helpful? Did you use this piece of information?" That's getting feedback from agent consumers. And then there's always the vibe check from the end user experience: "I told my agent to do this thing, I corrected it once, and I expect it to do better next time."

24:55 · Building eval datasets

Laís: Perfect, thank you. Does anybody have a follow-up?

Douwe: I'm really curious what the datasets you use for evals look like. I know you're using Pydantic Logfire. Are you using evals in Pydantic Logfire and the Pydantic Evals library? I'd imagine you need to simulate a lot of scenarios to test end to end, including how the knowledge cache even gets built. Like you said, it's not just recall, it's whether the crucial bit of data was found and used to get the right result. You basically need to simulate an entire company to do that.

Devin: You're not wrong. There are different levels of evals. For our full end-to-end benchmark, we are essentially simulating a company, where the temporal piece matters. It has to run sequentially so you build up knowledge over time, and those runs are both expensive and slow, so we use them sparingly.

From that full benchmark suite, we step back to things that are more structured. One is memory or knowledge extraction from the sessions themselves. That eval suite doesn't need to replay history: are we identifying what we deem important pieces of information, and how does that change between the models we use or with prompting changes? More classic evals.

Stepping back even further, we have evals on the search side. We want Dosu's read-knowledge tools to be very context-sufficient, so there's a curation and filtering layer before information is returned to the model. We have a set of evals there: given these chunks of information we found, are only the relevant ones returned to the model, and what percentage of the relevant chunks make it?

Where Logfire is extremely helpful for a lot of these tasks is being able to use agents to pull that information from production and then generate the eval dataset from it. That's been a huge unlock. Before, we would have had to manually curate a lot of that data or synthetically generate it. Because we're fortunate to have decently high-volume traffic, we can sometimes shadow launch things, collect a bunch of data, and use that for evals on the lower-level tasks that make up the end-to-end workflow.

27:48 · System One models for extraction

Michael: A follow-up: you mentioned a big part of your evaluation is judging how well you can extract memories for the cache. Have you thought about using a System One model, or even Cloudflare's brand new Clairic model that launched yesterday, which seems like a System One model that specializes in extraction? It seems like it'd be right up your alley. By the way, we're launching support for that in Pydantic AI.

Devin: Thanks for that. Yes, we actually switched over to a System One model once it came out. We ran benchmarks, and it had a huge impact on latency, and recall performance was the same as the previous small, fast model.

Michael: That's what I would expect. With a System One model, you can cast a super wide net to maximize recall, and then the System One model maximizes precision for you.

Devin: Exactly, and it's much faster. I think it reduced our P50 by something like 50%.

Michael: That's awesome.

Devin: It was super significant. I'm very excited about all the System One models that are coming out.

Michael: Are you still using Jev, or have you swapped over to one of the open source ones, the distillations?

Devin: From our benchmarks, Jev is currently still the best, from both a cost and latency perspective. So right now we're still Team Jev, but the competition is wild, and I welcome it. I'm very excited about it.

29:32 · Q&A: handling conflicting information

Douwe: There's a question in the chat related to what we were just discussing: how is conflicting information handled in Dosu? Can it be flagged to a human?

Devin: Great question. Part of the system is making sure there isn't conflicting or redundant information in the knowledge layer we're curating. That happens any time we update the knowledge base, whether on events like commits or in the classic dreaming process that runs on a schedule.

Whether it should be flagged to humans is a very interesting product question, and sort of a philosophical one. From our experience building Dosu, and I think generally in the space, humans are overwhelmed with review right now. They have enough work to review, and reviewing small pieces of conflicting information is something they just don't have time for or want to do. Maybe early on, to build some trust, and we have human approval workflows built into the product. But largely, we feel very strongly that if the knowledge is for agent performance, it should be maintained by agents end to end. That self-correcting loop is the most important thing, so even if it's not perfect, it will recover, and you don't have to be involved.

Douwe: I was also thinking that since Dosu is a knowledge cache more than anything else, there shouldn't be any information in Dosu that isn't in one of its upstream sources. So the real problem is upstream sources that contradict each other. It's not necessarily Dosu's responsibility to decide which is correct, although you might want to flag to the upstream source owners that things have gotten out of sync. I wonder how you think about that.

Devin: We actually do. We'll suggest updates to upstream sources, the human knowledge side, whether that's Confluence or a Markdown file on GitHub.

Taylor: Like Devin said before, we're really trying to focus on the outcome and what's tangible. If we sourced and gave you the right answer but didn't do anything to help fix it upstream, there's an opportunity there, and that's where we want to be as far as product is concerned.

32:11 · When a cached memory hurts

Michael: Did you ever have a situation where Dosu automatically extracting and caching something ended up hurting a downstream KPI or metric? How did you handle that? And more broadly, what's your procedure for changing your cache algorithm, or the behavior of the cache system? Do you have to backfill the cache once you change it?

Devin: Operationally, the approach is essentially refreshing it lazily. But understanding whether a piece of information is actually useful has been a huge focus of ours for the past couple of months. We've been building out these benchmarks, and, classic with evals, sometimes human intuition is wrong about what's actually useful to models and how they handle things.

That's driven some product decisions. It's how we got to hard-won context: trying to tune the threshold for what's complex enough that it actually makes sense to cache. The other really interesting thing has been learning procedures. I know we've been talking about the cache side, but procedures, knowledge about how to do something, are often the most impactful from a token efficiency perspective in our benchmarks.

We're taking that a step further on the product side, because something we've seen is that a lot of the memories or procedures that get learned are workarounds for your system. Sometimes things aren't intuitive, so the agent learns a workaround. We're excited about a future where it suggests fixes or changes, so it's not just learning a workaround but actually improving the system.

34:25 · Q&A: do frontier models still need a knowledge layer?

Michael: That makes a lot of sense. We have an interesting question coming in from one of our attendees. Laís, do you want to take this?

Laís: Of course. Evan is asking: earlier, with weaker models, it felt like knowledge systems that consolidated information from different sources were valuable for agents. However, modern models like GPT-6 Astra seem to do extremely well with direct access to primary sources, such as a Slack MCP or Confluence MCP, and it seems like the industry is moving in that direction. Do you feel Dosu's knowledge base still contributes to accuracy?

Devin: Amazing question, and this is where benchmarking has been super interesting. The short answer is that you're right from a trend perspective. We've seen on benchmarks that as larger and larger models have come out, the way they use and think about knowledge, and their efficiency, has changed. Where we've seen consistent change is the hard-won context category, the more semantic type of memory: the gap there shifts as models get better. Procedural knowledge, though, still impacts performance regardless of model size. That's been a very consistent trend, and we've been leaning into it a lot. However large the model is, it still needs to learn how to do things at your organization.

It is interesting to see the impact of knowledge change from the beginning of this year, with Opus 4.5-type models, to today. The way models treat knowledge has changed quite a bit, because they've gotten pretty efficient within the codebase and at using tools.

Douwe: As that's been happening, has the type of information Dosu stores shifted from actual cached knowledge and summaries toward how to find X, problems to be aware of, gotchas? I wonder if at some point you're just storing the hard-won knowledge and pointing at where knowledge can be found, rather than storing any knowledge itself.

Devin: Yeah, more of a map, essentially. Intentionally, our knowledge artifacts have become thinner and thinner, pointing out to references, because agents are really good at parallel search. And procedural knowledge continues to be consistently the most impactful. So the short answer is yes, semantic knowledge is becoming thinner, more like a map. There are exceptions, though. Human corrections and preferences, things that come up in a random Slack thread, are still very hard to search for at the right time, so those pieces of information are still useful. And the procedural bit continues to be very, very effective.

Douwe: One other thing I was thinking earlier: how does the model know, or how do you instruct it, when to go find data directly in the sources through the connected MCP servers versus when to go to Dosu? This partly answers that, because Dosu becomes the first place to ask where you're going to find this information, which then directs you to the MCPs you should search. Otherwise, I imagine a coding agent is trained to just prefer doing a bunch of grep searches or MCP searches. And if you expect Dosu to always have the answers, you end up building a massive search cache. The map framing makes sense to me, but I don't know if that's over-indexing on something that isn't what you think the product is best used for.

Devin: No, I think that's a good framing, and it's the reason I mentioned before that where agents read knowledge from depends on how much they trust it. That's why the hooks modality of injecting knowledge works best: the agent can do what it's trained to do and start searching its existing sources. In the background, we take the task the model has, send it to our servers, distill the relevant knowledge, and inject it back in. The agent does what it's trained to do but gets additional information along the way, and thanks to System One types of models, that's become very, very fast. You can actually impact the model's trajectory with that approach, whereas if you tell a model to call a tool, it will call it, but whether it trusts the result varies.

39:57 · Q&A: switching model families

Laís: Thank you. Do you have a follow-up, Douwe?

Douwe: I was going to bring up a question from the chat that aligns with this. Purna Chendra says: "I've been working on enterprise knowledge agents for two years and following Logfire updates since early 2025. A common challenge is how to ensure model response reliability and reproducibility when a provider's model reaches end of life."

We actually touched on this yesterday when prepping for this session, and it's very related to the GPT-6 Astra question about frontier models being much better at making the most of the data sources they have. The knowledge cache that even Opus 4.6, which is only eight months old or so, would need is very different from what today's frontier models need. Within a company, not everyone is using Claude Code with the latest model. You have internal agents that get deployed, they're stable, and people don't want to touch them too much. How do you make sure your knowledge cache serves everyone? Or do you have different caches for every model family?

That also ties into the general question I was hinting at. Evals are the answer to how you move to a newer model over time, but a lot of the time that requires reconsidering your entire context story. We saw that with Claude Code a couple of months ago when Fable came out and they removed 80% of their system prompt because it wasn't helpful anymore. Where do you even start when you want to move to a new model family? How do you do that internally, and what would you recommend your users do?

Devin: There's so much to that. I'll start with something interesting and related that we found through benchmarking, and other papers have reached the same conclusion: which memories models find useful, and how helpful a memory is, is often specific to a model family. It's very similar to humans. Something that's hard for you, so you want to learn it and save it for the future, might be easy for someone else. You see the same thing with models: things that are hard for Claude might be easy for GPT, and vice versa. So even among frontier models from the same moment, there can be discrepancies in memories.

The way we handle that is having a consistent model do memory extraction, which gives us a stable pattern, or opinion, of how memories are captured. That's part of the reason we started moving to session analysis, or trace analysis, rather than relying on tool calls alone.

For rolling out new models, the answer is evals, and it gets really complex. On the discrete evals, we've used Logfire, and Logfire's just great. We have our golden eval dataset, but when we upgrade a model, we do something else too. We're generally not using frontier models for our production tasks, because these individual tasks aren't that complex. So we pull a bunch of sample data from Logfire, use a frontier model to label that dataset, and then benchmark both our current model and the new model we're considering relative to the frontier model, as a proxy for improvement: how much more aligned is this model with frontier model intelligence? That pattern has made it much easier and faster for us to roll out model changes. Otherwise, defining what actually matters to you is important, and also really hard.

We've spoken to customers who use Dosu and have internal evals specifically around coding agents, where they want to evaluate not just how performant a new model is but how good it is at being an engineer at their company. Does it follow their style preferences? Usually it's internal tools that models won't reach for on their own. So they make sure new models are good at adhering to that, using Dosu's knowledge for it as well.

44:38 · LLM labeling and humans in the loop

Laís: I have a follow-up, then. You just mentioned you use models to label your datasets. I think we just made a lot of data scientists very mad. How do you decide where your human in the loop gets inserted? There's a very large trade-off on trust here, and I want to hear your thoughts.

Devin: Oh, man.

Michael: Devin, do you mind if I sneak in an answer? It's not going to be a full answer, because I still want to hear from Devin, and the way they do things at Dosu may be totally different. As someone who was a data scientist for a very long time, one thing I observed very quickly at Walmart, when I was building our first enterprise agents, is that we gravitated toward trusting the human for everything. But we quickly discovered that humans have a flaw, and that flaw is laziness. If you go back in time, people in machine learning and the early days of data science observed this, so they'd do things like have three labelers, and any time there was disagreement or variance in the human labels, they'd throw it out. People were solving this even before agents. I just wanted to say that LLM judges, if prompted correctly, can be almost more accurate than humans. So I'm curious what you mean by making data scientists angry.

Laís: What I meant is that right now there's a lot of labeling that gets done manually by humans. That was the core of the question: what's a good framework for deciding when to insert a human into the evaluation loop, and when everything should just be handled by LLMs?

Devin: Maybe my take is controversial, though I don't think so. Think about the rapid pace of model development this year. I don't work at a foundation lab, but my intuition is that a lot of it has come through synthetic data generation. That's actually true for Jev, a very performant System One model trained on 100% synthetic data generated by LLMs, with a teacher model. From what we've seen, models have gotten really, really good at eval and dataset generation, so we've been leaning into it.

We had an eval set, I'm trying to remember the exact task, of about a hundred painfully labeled examples. Classic: we set up a little workflow script and everyone helped label it. Opus, I think it was Opus 5.5, got within about 98% of what we wanted. At that point, if it can label 10,000 things with 98% accuracy, and we can only do 100 and it takes us a day, it's worth the trade-off. And I think models will keep getting better in this domain, because it's how they're being trained and fine-tuned as well.

Michael: I think you decide to bring in a human when there needs to be some ownership of responsibility for a critical decision. It's not that the human will make a better decision. I think that's almost provably wrong at this point; models make better decisions almost all the time. But an organization needs some kind of human ownership of risk, and that's the condition where you say, okay, we have to bring in a human to own the risk of this decision.

49:06 · Q&A: frontier models for golden sets

Laís: Question answered, thank you. We have one other question: can we use frontier models to create the eval golden set and then use agents to evaluate against it, or would that be a circular reference?

Michael: As janky as it sounds, and anyone feel free to disagree with me, I get the feeling this is what everyone is doing.

Laís: Yeah.

Douwe: And on where humans come into the loop: to be clear, no human is going to match 98% accuracy either. Humans are very flawed and make mistakes. In any organization, every manager has to trust the judgment and skills of their reports, and it's exactly the same with AI now. You'll have one agent review another agent's work, and at the end of the day some human takes responsibility, but they have to trust the people below them. I wouldn't be too worried about the circular reference. There's definitely a sense in which Claude models prefer output from Claude models, so you may want to throw some different models into the mix.

Michael: That was Karpathy's thing. He said use a council of LLMs to get everybody's opinion, and you'll do better.

Douwe: Just throw more LLMs at the problem, of course.

Laís: That's the solution: throw more LLMs at the problem.

50:36 · Q&A: connectors, libraries and access control

Laís: Do we have any more questions? Do you have any more questions for each other?

Douwe: There was a question for Dosu from Saimane: does Dosu offer connectors to more data sources than the ones we've mentioned? At an enterprise level, companies have a wide range of sources, like Confluence, Jira and SharePoint. How does Dosu scale to a wider range of platforms?

Devin: That's an easy, quick one. We have native integrations for documentation sources, so Confluence, Notion, Coda and GitHub; communication channels, Slack and Teams; ticketing systems; and code repositories. Outside of that, there's MCP, so you can connect anything via MCP. The difference is that the native integrations are for sources we explicitly capture knowledge from directly or help update.

Douwe: So what you'd typically see is connecting all the company sources to Dosu, and also connecting them to the user's agent through MCP. How does that work when people in engineering, marketing, operations and sales need different slices of that?

Devin: We have a concept we call libraries, which is basically a way of partitioning knowledge. At small companies it's usually one library, maybe two, but at larger companies it's usually partitioned by product or team. It's a way of separating out the MCP connectors for the marketing team from the engineering team. We also pass through RBAC for all those data sources, to make sure people only have access to what they should.

Editorial clarification (October 21, 2026): Upstream permission enforcement requires an eligible Enterprise organization to enable Data Shield. It covers GitHub, Slack and Confluence and does not extend to other connected sources.

Douwe: Which gets very difficult if you're connecting to something like Google Docs, where every document can have its own permissions. I had to deal with problems like this in a previous job, and I'm very glad I don't have to anymore.

Michael: It's a problem. The benefit, though, is that you can break down the silo walls of an organization and create an incredibly rich knowledge exchange between pillars of your organization that could never exist before, because humans alone don't have the mental capacity to do all that retrieval and digest all that information. The downside, of course, is that you'll have an agent reading data its user doesn't have access to, because you never needed security controls for that case before.

Taylor: So long as you get your security right, and maybe we can ask Douwe some of our role-based access control questions when things get really difficult. Thinking back to working in enterprises with thousands of people and engineers, it was always a magical, special moment when you'd bump into somebody working on something similar to you. Now, in the age of agents, with libraries and these other outcomes, you can ask, "Is anyone else working on this?" and get citations and examples. I'm not advocating that you never talk to people again, please still talk to people, but being able to see all of this information is really helpful. You get these magical outcomes, you can work with people and hopefully meet them in person at some point. Seeing knowledge scale this way, and having the ability to access it, is incredible.

Michael: You can at least have your agent talk to their agent.

Taylor: My people will talk to your people, my agents will talk to your agents.

54:27 · Start simple

Laís: Yes, and when in doubt, throw more LLMs at the problem. I think we're nearly at time, though we still have more than six minutes. If anyone wants to share final conclusions, please take the floor.

Douwe: I do have a question for Devin. I'm reading through the abstract for this webinar again—

Michael: It's not a webinar.

Douwe: It's not a webinar. It says webinar in our document! Anyway, it asks: how do you provide that knowledge, keep it up to date, and measure whether it improves their work? Part of the answer, apparently, is just use Dosu. But I'm curious whether you have high-level observations for people building agents who don't want to rely on a vendor for this very complicated problem space. And on the title: "Stop Building Context for Every Internal AI Agent."

Devin: From what we see, and from talking to customers, every company ultimately needs something like Dosu, some sort of shared context layer. Otherwise you end up with many different connectors, and someone builds a separate memory system.

If we fast forward maybe two years, where I think we're going is that the cost of intelligence is going down dramatically, and execution is becoming very, very cheap too. So what companies are is the knowledge and processes that make them up. I see a future where every company has to curate this knowledge set of how things work and how things are done, specific to that company. It's important that you own it, or at least work with a vendor who makes sure that information is exportable and shared across your different agents. You don't want a situation where Claude knows all these things about your business, your internal agents know other things, and your code review agent knows something else. You want a central source of truth for that information.

My advice is always: start simple. As an engineer, keep it simple. Build something, whether it's a Markdown file in your repository or using existing data in Confluence. Start somewhere. It will break down, especially as you scale and run into things like in-flight knowledge and scoping challenges, and then you should start looking for vendors. But start somewhere, get a feel for the benefit and for the shape of the problem and its pain points, and then look for vendors, and maybe reach out to us.

Laís: Perfect. I just dropped a comment that the KISS design pattern continues to apply in the age of agents and AI.

Douwe: This also connects to what you mentioned, Devin, about the cost per unit of intelligence continuing to go down, and it ties into the bitter lesson: more scale, more intelligence solves all problems. It's related to what we said about GPT-6 Astra not needing a lot of the scaffolding and hand-holding anymore. What I tell our users a lot when they're building with Pydantic and looking at how expensive certain things are now: you should be building for the agents of six months from now, which you can imagine are 10, 20, 100 times as smart at a tenth or a hundredth of the price. Don't add extra complexity if you don't need it today, because you'll probably never end up needing it. Three or six months from now, the models will have improved enough to make it unnecessary. So KISS is definitely a design pattern that still applies, and premature optimization is the root of all... what is it again?

Michael: The root of all...

Douwe: Evils.

Taylor: The root of all evals, yeah.

Michael: Get rid of all evals.

Douwe: Premature optimization is the root of all evil. I'm sure marketing could do something with that.

But yeah, keep it simple. At the end of the day, how did humans figure out how to build organizations with thousands of people? They started writing handbooks and onboarding guides. We're doing the same for agents, and agents can end up maintaining those pretty well a little down the line.

Michael: Eventually your onboarding guide gets to be 300 pages, and nobody reads it.

Douwe: But agents will!

Michael: Right.

59:26 · Wrap-up

Laís: Wonderful. We are now at time. Thank you very much, Devin and Taylor, for joining us today. It was wonderful to learn more about Dosu. Thank you for all your answers and all your questions; it was a pleasure to host you. And to my lovely colleagues Michael and Douwe, thank you for joining as well. I'm looking forward to the next one.

Michael: Same here. Thanks for having us, Laís.

Laís: And thank you to everybody who joined. A recording will be available, and if you have any more questions, just drop us an email. Enjoy the rest of your day. Bye-bye!