From Prototype to Production: Agents That Remember

CrewAI and Dosu teamed up for this session on what it takes to move agents from prototype to production when they need to remember what they learn. Host Taylor Dolezal, Dosu's head of ecosystem, talks with CrewAI founder and CEO João Moura, CrewAI head of forward deployed engineering Tom Haddock and Dosu founder and CEO Devin Stein. The conversation covers when to take humans out of the loop, resolving contradictions between agent and human knowledge, CrewAI's Agent Control Plane, evals and ground truth, the share of tokens agents spend regathering context, agent identity, choosing between frontier and small models, and what changes when agents become the main readers of your docs. For more on where agent knowledge should live, see Agent Memory: Where Does Knowledge Live?. This transcript has been edited for length and clarity.

Transcript

00:00 · Introductions

Taylor Dolezal, Head of Ecosystem, Dosu: From prototype to production: agents that remember. We're going to tell some fun stories today. We are Dosu, and we're joined by some of our friends at CrewAI. I'm Taylor, head of ecosystem here at Dosu, and I'll be your host today.

We all want to build truly autonomous, self-learning agents, but it's hard. I'm not sure how many of you relate, but it's pretty hard, huh? Many of us are rebuilding agents because they break or they forget things. We're trying to find the best places to store agent memory, and it can be a huge time sink for the hundreds or thousands of agents being built by engineers and others just like you. Let's talk about it: tokens, time and some contradictory answers.

Dosu and CrewAI are both tackling this problem, and we're going to cover how teams think about building agents that remember, and that learn things as they go. If you have any questions, drop them in the chat and we'll get to them. This should be fun.

First, let's do some intros. I'm going to start with Tom, then go to you, João, and then Devin. Tom is the head of forward deployed engineering at CrewAI. Tom, kick us off. What's it like to be Tom?

Tom Haddock, Head of Forward Deployed Engineering, CrewAI: Thanks for having us on today. It's great to meet everyone. My name is Tom Haddock, and I'm the lead forward deployed engineer here at CrewAI. FDE is a term coined over the last couple of years, mostly by Palantir, but it's no different from the role we've been doing for years. It's the application of a product in a customer environment. Let's say the product provides 50% of the value out of the box. What does the other 50% entail? For some of our customers, we like to say it's 90%, and the FDE gets you over the top on that last 10%.

On the presale side, it's leading platform demos, sharing customer testimonies and scoping out use cases to get a technical win. But our FDEs typically stay on through post-sale to help with implementation, adoption, enablement and troubleshooting, and to actually sit as a software engineer on these projects. Some of them are comprehensive, complex agents that involve a variety of parties, so a lot of it is gaining business context. How does this process actually work today, and how do we want the agent to do the same? I coordinate our team of about seven to ten FDEs at any given time on some of our largest customers, and I do a lot of the individual FDE work as well.

Taylor: Gotcha. I've heard of FTE, full-time employee, but FDE, like you said, is fairly new. I'm excited to dive into some of the stories you have to share on that front. Let's go to João, who is the founder and CEO at CrewAI. Howdy, João.

João Moura, Founder and CEO, CrewAI: I'm doing good. Excited. CrewAI has been a wild journey. If you're watching this market, or playing in it, you've probably got a few extra gray hairs, the same way I do, and probably Devin as well. It has been busy, but it's good busy. We went from agents not being a thing, to agents being a thing, to agents are dead, to agents are back, all in the span of a few months. But it's very exciting.

At CrewAI, we're known for two things. One is open source. We have an open source community and project that many engineers use to build agents. I'm always a little amazed when I go into an incognito ChatGPT, ask for one of the major agent platforms, and CrewAI pops up. Then there's the enterprise side, which is where we go commercially. That's how we partner with customers and help them build, deploy and monitor agents and everything in between. Very excited to be doing this, and thank you for having me, Taylor and Devin.

Taylor: We're excited. It should be a good conversation. And to round things out, Devin is the founder and CEO at Dosu. Could you tell us a little more about yourself, Devin, and some of the things we're thinking about at Dosu?

Devin Stein, Founder and CEO, Dosu: Of course. Thanks, João and Tom, for joining. Hi, everyone. I'm Devin Stein, CEO and founder of Dosu. Dosu is knowledge infrastructure for agents: how do you capture and share context as agents work, to make them more efficient and productive over time? Increasingly, as agents have started doing work inside enterprises, as we'll get into, figuring out ways to unblock them, make them more productive and make sure they're doing their job correctly over time has become a critical role for both engineering teams and organizations writ large. So I'm excited to talk a lot more about it.

04:58 · What the most autonomous teams do differently

Taylor: Awesome. And just a reminder, I will keep pestering you: if you have questions, drop them in the chat. We want to answer them as they come up. Today we have about four sections planned, though we'll probably take a lot of twists and tangents and meander quite a bit. Let's kick things off with agents in enterprises. I'd love to put the spotlight on you first, João, but Tom, Devin, if you have opinions or thoughts, we want to hear them too.

João, we're seeing the most AI-forward companies hand over huge aspects of software development to agents. Among your customers and your community, who's getting the most out of agents with minimal human intervention? Is there anything we should take note of to get to a little more autonomy?

João: That's an interesting question, because that is, quote unquote, the end goal. Everyone wants to start delegating to agents to the point that they don't need someone to babysit them. For the kinds of companies we work with, a lot of the use cases are very critical in the enterprise sense, so they want to make sure someone is overseeing it.

But the ones being more aggressive are the ones that do extremely good work around context, and a lot of work evaluating use cases. They really put in the time to evaluate and move fast, and they get more miles under them. For people coming in with their first use case, they might be better off starting with something simple, something that involves a human in the loop, just to get a comfortable understanding of the threshold for paying off for the organization and the team actually building it. As you put more miles on and learn how to build these things so they become more reliable, so you have memory and they're, quote unquote, self-evolving, you get into a very interesting position where people go for the massive use cases that are really transformational.

The market as a whole always thought this would be magic: you blink an eye and it's working. But every use case I look under the hood of, and Tom has been doing many of those, is never simple. It's always a mix of internal tools, external tools, memory and integrations, and a bunch of things in between. So my main advice is to put miles under you. Really spend a lot of time thinking about memory, improvement and context. I think that's key.

07:56 · When to take the human out of the loop

Devin: One question, João or Tom. How do you advise folks building on CrewAI, or just generally, on the transition from when a human should be in the loop to when you can take the human out? We see this a lot at Dosu around knowledge. There are a lot of places where people initially want to be in the loop. If Dosu is updating your documentation in Confluence, some people ask, "What if there are issues with that?" So we built in mechanisms for people to review things before they're published. But sometimes that leads to a large backlog of things to review, which defeats the purpose of agents to some extent. What do you see, and how do you advise folks on that front?

João: I can tell you, in broad strokes, some of what we do internally. We actually have agents that help triage what needs human review and what doesn't. For example, we use a lot of AI for coding and for reviewing code. But we're not comfortable saying no human is going to look at code anymore. We still want engineers looking at code on the review side. There are certain kinds of PRs, though, that we're comfortable with AI reviewing at this stage. So we have an agent for that specific case that judges whether a human should be involved or not.

We eventually evolved that into a more formal process where you can have a policy: if it's using a certain tool, a certain feature or a certain integration, you can enforce that it goes through a human in the loop. So I usually advise people not to think about it as black and white, where either you have a human in the loop or you don't. There are many shades of gray in there, and you can adjust. But Tom, you're working with these customers day to day, so I don't know if you have a take on that.

Tom: I think it always starts with humans in the loop, because you want to be familiar with it. There's a lot people want agents to do out of the box that contains a lot of nuance, whether it's internal knowledge, preferences or workflow-specific knowledge, and agents are not going to be good at that. When I start new projects and new POCs, I say things like: let's get the scaffolding in place. Let's find out how users actually want to interact with this system. Let's always produce a read-only output. Let's not write to a production database. Let's get humans comfortable with the outputs of these agents and review feedback there.

A good example is an internal triage agent we're relaunching on our internal support module. The number one goal is to properly triage all the support requests. Those can be bound to different departments, like open source, platform or infrastructure. Maybe it's an FDE-related use case, or maybe it's ambiguous. The knowledge and the brain that go into properly triaging aren't going to work on day one. But over time, you incorporate more feedback and build trust with that system, and then you can say, all right, now we can actually kick these off and submit those PRs on the open source side, because we have a better idea of how predictable it is.

I think that's really what it takes, and as I spend a lot of time with business counterparts, it's the most important thing. You really want to build trust with these agents. Like any other piece of software, you don't want to launch a new system in an environment where people don't trust it or don't have any familiarity with it. They want to learn as they go. I always try to build agents and use cases with the least permissions and the least data access, and then when people are comfortable, they can allow them to do a little more at a time.

For example, I don't let agents email on my behalf. I'm not sure if you're different; it's just my own mantra. I like them to provide drafts for me. That's not something I see automating for me, because I want to be responsible for it. But there may be people for whom, over time, it gets much better, and then they say, okay, respond to marketing emails, respond to spam, because they've built trust with it over time.

Taylor: Gotcha. I think the jury's still out on emails. I'd give you the tech answer that's so popular: well, it depends. Which is probably the worst answer to hear, and to make tangible. But depending on who it is, that's an interesting thing to look at and figure out.

I've got two more fairly quick questions on agents in enterprises, and then we'd love to get into how agents do this work. That's where I have a lot of questions we won't have time for.

12:49 · Resolving contradictions in agent memory

Taylor: Devin, I want to put the spotlight on you and ask about agent memory. We've talked a lot in our own webinars and sessions about what it takes to store and retrieve memory, but maintenance of memory is the thing that continues to keep our focus at Dosu. When you're working across a large enterprise with all this different information, someone says something, and depending on their role, it might invalidate other information. Can you talk about how organizations resolve contradictions, maybe with a specific example?

Devin: Absolutely. Related to what Tom and João were saying about building trust, one of the simplest ways to trust agents, or to know they're correct, is for the knowledge behind them to be correct. If they have access to the right context, they're going to give the right information. Otherwise it's garbage in, garbage out.

When we think about how to deal with contradictions within agent memory and organizational context, a lot of it, at least from the product side, is syncing what agents know with what humans know. A common failure mode is maintaining agent memory that's distinct from human memory. That can lead to confusion and contradiction, where the agent learned something and saved it as a memory, but Confluence or Notion says something completely different. So when we think about agent knowledge versus human knowledge, we try to keep the two in sync.

But that also leads to the tension I was alluding to before. People are very comfortable with agents managing agent knowledge and memory. People are sometimes less comfortable with agents managing their existing knowledge. So there's always a product tension: you want information to be consistent, but there's sometimes fear of agents messing with existing systems. You have to figure out ways of building that workflow, and building trust in which types of updates you can let agents do end to end. Basically, you want the friction to update knowledge to be very, very low, and once you have a human in the loop, that friction increases significantly.

Taylor: It's helpful to have agents handling a lot of these things too, Devin. When you take the time to really pay attention to the speed at which agents can update knowledge and documents, and even share things out with teams, it's dizzying. I thought I had enough difficulty staying on top of all the AI news coming out, but if you try to listen to what the agents are throwing back at one another, it can get kind of wild.

15:59 · CrewAI's Agent Control Plane

Taylor: João, I was taking a look at your blog, and I saw that you've shipped an agent control plane. Does this help us? Does it solve all the problems? Can you talk a little about that before we jump into agents doing work?

João: This doesn't solve all the problems. It solves some problems, though, I can tell you that. It's funny: I feel like control plane is the hot word right now. Everyone and their mother has a control plane, or is trying to launch one. Our HR system launched one the other day, and I was like, what? Our HR system? Maybe McDonald's will be next. But it is what it is. It's a hot thing, and everyone's going to try to make sure they're in the wave.

We spend a lot of time thinking about this because of the kind of customers we work with. Once they get over that initial hurdle, they've got their first one or two use cases live, and they get a taste of it: all right, this is good, I'm hooked, I want to automate our company away as much as I can. Then the problem is that they need to federate the building. There's no way one single team builds these agents and scales at the rate they want, especially if they want to be transformational. Some of these companies have 200,000 employees, so they have to find a way to federate and have independent teams build. Then the problem becomes how you share learnings and building blocks, but the most important one is how you add control layers, so that as an IT or technology organization, you can rest assured certain policies won't be broken.

I think the three main things they care about are: one, are these things working, or is something running in a loop and breaking? Two, how much money am I spending, what's driving most of the consumption, and to which providers? How much intelligence am I buying from Anthropic, OpenAI, Grok or whatever? And three, very clear rules. If it's touching our HR platform, it has to remove PII. If it's touching email, it can only run in business hours, for example.

So we put that together into ACP, the Agent Control Plane. We worked with a subset of our customers to create what we think are the thirteen core policies they all asked us for. They can still create custom ones, but we made those thirteen point and click, and they just work. We're very excited about it, because it's not about the control itself. It's that it now allows them to federate the building and say, all right, let's bring 150 people in to start building agents, because we know we're going to have control over this. That's how this is playing out for us, and it's great to have it out there.

Taylor: That's awesome. I resonate, and I'm sure regulated industries do too, João. We have set things, specifications and rules, but it seems like agents believe rules are meant to be broken and sandboxes are meant to be escaped, like an escape room. It will be an interesting next couple of months and years as we harden that and standardize on some things. Hopefully control planes can get us there.

19:35 · How do you know an agent is working?

Devin: One follow-up on that, João. Related to what we were talking about before on trust, how do you know if an agent is working, or even if a policy is working? Is that part of the control plane, surfacing places where a policy was enforced so people can sanity check it? Does working mean there are no errors, or could it be failing in more subtle ways? How do you think about that?

João: That's a very interesting question. There are a few different things. There's working in the plain sense: is this throwing an exception? Then there are different degrees of working. Maybe the execution runs end to end, but a few tools it's using aren't accomplishing what they need to, or there's a permission problem, for example when it's trying to call your CRM.

So we basically define three states. Healthy means no errors and the thing executes end to end. We're not necessarily judging the outcomes, but the process works. It can be in a warning state, where it runs end to end but there are errors along the way: your context window is blowing up, or an integration isn't answering what you expect. And then there's point blank erroring out: this is not running. It might not be every run. Maybe 25% of executions just aren't running because there's some issue.

When you get into policies, things get more complex, because a lot of these policies need to be enforced at runtime. PII is streaming from one system into another, coming in or out of an LLM, and as that goes into traces and observability, you want to remove it. So it's a combination. We have OG NLP models, regular natural language models rather than LLMs, filtering out some information. Some pieces are regexed, filtering on certain pattern matches. And some use LLMs that have been specifically fine-tuned to find certain information and block certain things.

Some policies are deterministic, though. If it calls a certain tool, X has to happen. And the final piece we do is, say it calls the HR tool: this needs to happen, and notify someone. That can be the automation owner, the person who created it, or the people who have access to it. So that's how we think about it: understanding whether this is working or not, without going as far as judging the quality of the output at this stage; how much we're spending; and policies that run almost like a sidecar with your automation, monitoring everything that happens within it.

22:44 · How little agency production workflows use

Taylor: Interesting. Tom, I'd love to ask you a couple of questions. As head of forward deployed engineering, you're working day to day with people who are changing their workflows with agents, especially this year. I'd love to hear some of your takeaways and, optimistically, some surprises you or your team have run into as this year has gone on.

Tom: Definitely. I'll build on what João said earlier. There are really three lines of control we try to enforce. There are the individual parameters, using the CrewAI framework to limit the number of tool calls, limit execution time, things of that nature. Pretty standard. Then you have deterministic or probabilistic guardrails, like LLM-as-a-judge to evaluate the output, or a more deterministic check, like making sure a message is no more than 500 characters. Those happen in the workflow itself. And the last piece is the retroactive application of policies you can enforce after something is deployed. Someone creates a project and submits something, and a policy says, make sure all internet requests are routed through a specific gateway. That's how you achieve that enterprise level of control.

One thing that surprised me as I learned this and became native to CrewAI is how little agency is actually used in a lot of these workflows. João, I don't know a world before CrewAI Flows existed. Flows is the deterministic backbone all these agents bootstrap onto while they do tasks, so they have persistent access to data over the duration of their execution. And it surprised me, because when we're building agents for an entire organization or department, like procurement or marketing, the agent itself is maybe a little bit of reasoning here and there when it has a full set of inputs. The rest of the automation is more or less purely deterministic.

You'd think one agent will just go in, call an LLM and do everything: access your data, aggregate it, run a synthesis, generate a report, send the email. The reality is it's nodes in a workflow. What we've seen enterprises latch onto is starting with one workflow, say a very simple vendor selection process or some step in the broader procurement model, and then adding on capabilities. Now a smart agent can be the interface to that system and route intent to all these different outcomes. At the end of the day, it's a lot of RPA, a lot of software and traditional APIs we've been using for decades, with agency at small steps. You only give agents what they need, and they run their tasks, run their synthesis and produce an output. That surprised me, because I'm so dependent on Claude Code and other agents for development. But when you build agents for business processes in the enterprise, that's the model that sticks the most.

26:05 · Benchmarks, evals and ground truth

Taylor: Gotcha. Thanks for sharing, Tom. If you'd asked me a year ago, five years, ten years ago, whether I'd ever care this much about benchmarks and evals, I would have said no. Now I can't help but check out Artificial Analysis and a lot of other sites in my day-to-day work.

I'd love to start with Devin and then talk to João about this. Benchmarks and evals: there's a lot that's public and published at a general level. But like Tom was saying, as you implement these agents and start working with them within your enterprise, a benchmark might be too general. It might not fit the work you're trying to evaluate. What are your thoughts on benchmarks and evals, and what do you see as important as we navigate this space?

Devin: Big question. Tom was describing deterministic workflows for business processes, which I think is really interesting, and it's very different from the environment Dosu, or typically our customers, operate in. Our customers are heavy Claude Code, Codex and Cursor users. In that domain, the agent largely works. I tell Claude to put up a PR, and it's going to do it. The question is, did it do it the way I wanted, or the way my organization believes it should be done? And did it do it efficiently?

So when you think about benchmarking agents that are more frontier, that are able to do anything, the question is how you measure whether they're doing it well. That's a lot of what we've been building out at Dosu, both in benchmarks for internal purposes and, increasingly, as part of the product. An agent completes a task, but where did it spend a lot of tokens? Did it have a hard time finding information that should have been easy? Did it spend a lot of tokens sifting through Confluence or digging through the codebase to find a small piece of information? Did it backtrack, trying things it thought were the way to do things and only finding the correct way at your organization after a couple of routes? Did humans have to intervene, steering the agent or telling it, "No, actually do this, because X, Y and Z decision was made"? And lastly, the classic: unproductive tool calls. Is it using tools in ways that aren't very helpful because it doesn't understand them, or are there bugs in the implementation or the permissions it has?

So for benchmarking and evaluating frontier agents, where there isn't a deterministic process with a clear success or failure, it comes down to measuring these smaller details about where agents run into friction, and then figuring out what you're optimizing for. On the software side, we see a lot of teams wanting to optimize for token efficiency, especially at larger companies, where it's a lot of money if your agents are spending it on things that aren't productive. The other side is what we've been calling agent autonomy, or agency: how much are agents able to do without humans in the loop? If agents can do tasks end to end without a human, you can parallelize and do exponentially more work than if you need a human to check in at certain steps. Otherwise you're blocked by that engineer's bandwidth. So in general, figure out what you're optimizing for, and then measure as much as you can.

Taylor: Gotcha. I'm hearing a lot of those elements of trust too, and trying to figure out how to apply that and where it goes within these workflows.

Tom: I'd add one thing. The most exhaustive thing we do when building out these use cases, less the SDLC agent use cases and more agentic workflows for the enterprise, is that you still have to have ground truth. What is the acceptable test? Often that has to be collected from a history of prior manual transactions, or you have to do an active assessment. One use case I'm working on is classifying social media posts and whether they should or shouldn't be included in campaigns. The nuance that goes with that is very hard to determine, and an agent's not going to do it on its own. Being able to compare against a ground truth of 10,000 examples, where you can clearly say this is why it is or isn't included, is very important. That's a manual step, but doing it up front gives you a better grasp of what the agent's doing and why.

31:18 · Ad hoc and embedded use cases

João: It's interesting, because I always think about two different kinds of use cases. There are use cases that are more ad hoc, and I mean that in a good way. You just want to throw something at an agent and have it figure it out. I think a lot about OpenClaw and Hermes agents. They don't need to be generic, though most of them are. You can have agents a little more tailor-made for whatever you want to accomplish, and you just send them a thing, and they'll use their tools, skills and memory to get it done. Those use cases, we all love and we all use. We generate PowerPoints, spreadsheets, all that analysis.

Then there's what you folks are referring to, which I'd call embedded use cases. They're less ad hoc. Sometimes they have a trigger, sometimes they're recurring. In those, you want the benefits of agency for decision making, extracting, researching, putting together reports and structured data. But you might want different levels of agency at different steps. If I just want to extract some data, maybe I don't need an agent; maybe I just want a single LLM call.

The labs in general are pushing the narrative that everything is going to be vertically integrated: there'll be one single god model that rules them all, spawns the sub-agents and gets it done. And we can already feel the AGI. You go into Claude Code, turn on Fable, execute ultracode, and it's going to get a lot done for you. But there's still the question: is this the most efficient model? Is this the most efficient way of actually accomplishing this? I don't think incentives are super aligned between the model labs and enterprises trying to do this. Enterprises want to do it in the most efficient way possible, with the least amount of risk and getting as much value as possible. And I think that creates different shapes of use cases.

Taylor: I really like that. I heard an interesting story this weekend. Someone was working in the Codex app on their Mac and asked Codex to do something, and it's not important to the story what it was. They got an alert in Codex asking, "Are you okay with giving Codex computer use access to Codex?" And they were like, oh, I think I just solved AGI. I thought that was a good way to use your agents to feed your agents, and still eval trust and everything on this front.

34:22 · 94% of tokens go to re-reading context

Taylor: Changing gears a little bit. One of the things I love about Dosu is just how curious the team is. The things Devin shares in Slack are wonderful to read: really thoughtful articles and discoveries that teams are sharing. One of them is from Atlassian's developer experience team. I love reading these reports, and I'll post the link in the chat shortly. In this report, they mention a stat that 94% of token consumption was going toward regathering codebase context, along with some other really interesting bits of information. It makes me curious where the other 6% is going. Devin, is there anything that really caught your eye as you read this report?

Source: DX, State of AI Impact in Engineering: Q3 Report. Editor's note: DX identifies the 94% figure as cache-read tokens' share of total token usage. It does not measure the share of spending or newly retrieved codebase context.

Devin: Obviously I was very excited to see that stat, given what we focus on. A lot of the purpose of agent memory and knowledge is capturing repeated learnings and processes and surfacing them again, so agents don't need to relearn them. So it was very validating to see that 94% number. I was actually shocked it was 94%. From our internal benchmarks and what we see on the coding side, it's often in the 70 to 90% range, so 94 is on the upper bound. But at the same time it's intuitive. If you think about what you do as a human at work, a lot of it is gathering context and understanding in order to make decisions and do things. Even in the pre-coding-agent world, people used to say most of the time spent coding is just thinking about the problem, and a very small percent is implementation. We're seeing the same pattern with coding agents.

From a business perspective, if you're spending 94 or 95% of your tokens on regathering context, that's obviously a very large opportunity for efficiency. And going back to what João was saying, the foundation model companies' incentive is always going to be quality. They want the outputs to be good and the agents to be smart. They're not worried about efficiency. I think the 94% shows how early we are in the efficiency story. Right now, agents are just unbelievably inefficient at navigating information within an organization, and I think we're going to keep seeing a lot of improvement there.

37:04 · Every token should be worth more than it costs

João: That's a good point. I'd say the labs are definitely incentivized around quality, but honestly, they're incentivized around token consumption. That's the main thing.

But I agree with you. We talked about this as we were planning this panel. This is how I think about it, and it sounds lame and obvious, but it honestly helps me parse use cases, and I think more people should think about it this way. Ignore all the technical pieces: your harness, your vector database, your embedding model. Strip all of that away for a second. At the end of the day, you're paying these things by the token, and you're using them because you want their ability to reason, to give different answers depending on what you throw at them. So you're paying by the token, which means every token you get from them should be worth more to your company than what you're paying for it. You should be able to do something with it that's more valuable to you than what it cost.

How do you make sure you get the best tokens out of it? You need to do extremely good prompting. That's it, at the end of the day. The better the prompting, the more you access the region of that space where the tokens you want for that specific thing live. So much of your harness, your vector DB and your embedding model goes into optimizing every single request to be the most optimal request possible. That makes it so much simpler for me to wrap my head around.

If you work with that as a guiding principle, you start thinking, oh, I need memory, I need something like Dosu, because whenever I make an API request to one of these models, I want all the right ingredients in that prompt. Not too much, so I'm not paying 94% for something I don't want. Not too little, so I'm not getting tokens that are less valuable to me than what I'm paying for them. I need every single request to work. So often people aren't thinking about that. They grab an agent, throw in some tools, turn on memory, and there you go. They're not doing any of that optimization, and there's so much more to gain, not only in efficiency but in quality as well. That's my mental model for some of this.

39:50 · Agent identity is the biggest hurdle

Taylor: Gotcha. In terms of the agent friction we've seen, I'd love to hear from you, Tom, working with customers on agents and adoption, and then I've got some questions for you, Devin, as we round things out. To the audience: I know we've thrown a lot at you, so compact your context a little bit and think on some things. We'd love to hear your questions before we round things out. We could talk for hours, but there aren't enough of those in the day.

Tom: On the friction we've seen with agent development, I'll speak anecdotally. The thing I've spent a lot of time on the last couple of months, and definitely will for the next couple of months and next year, is individual agent identity management. As we move away from the model of accessing data on a user's behalf, agents now have their own identities, and that's true across a variety of applications in the enterprise stack. Getting consistent about how you onboard agents, and what scopes they have via OAuth into the rest of your enterprise stack, is the biggest hurdle we've faced.

If I were starting fresh, I'd build in a lower-level environment, a sandbox or dev environment, and use user-level permissions to access Databricks or Snowflake data on my own behalf. But when you promote this into production, the agent should be doing that task on its own behalf. Then it becomes traceable. You can verify what it has access to and what permissions it has, and you propagate the user session token and the user access token all the way from when that agent was first woken up down to the tool level. So the agent operates on its own behalf, but with guidance from that user.

This model is the most complex thing I spend time on, and it's highly difficult to implement from scratch. But if I were starting an organization today, whether you're using CrewAI Enterprise and its shared connections or building agents on your own, I'd spend a lot of time integrating with the identity provider you already have, and build on the assumption that agents will operate on their own behalf.

42:29 · Frontier models versus small models

Taylor: Gotcha. Continuing that friction conversation, one thing that gets really interesting is when it's time to change models, whether you ramp up to something like Fable or start optimizing and ramp down to something like Haiku or Luna. Devin, we've had many fun conversations about the things you can do with smaller models. They're actually pretty smart. When it comes to model sizes and the friction around them, do I use a harness, how do I stay efficient with memory, context and tokens, do you have anything to share with people who've got it working and are looking at upgrades or ways to optimize?

Devin: There's a lot to that topic, so I'll try to keep it short. Right now I see two different use cases. If you're building an internal agent for specific use cases, you generally don't need frontier intelligence. If it's an RPA-type workflow, some sort of workflow with historical examples, it's amazing how smart smaller models like Haiku and Luna have gotten. They're really capable and way more affordable for running things at high volume. So if you're building agents, generally you don't need frontier.

If you're on the other side, using agents for general-purpose work, specifically software engineering, because that's where agents are most mature right now and the space is evolving so fast, my opinion is that you should always lean toward the best model. If you're actively using agents for software development and want to keep pushing on the frontier, just use the frontier. Don't play the cost or model optimization game just yet. Think about whether you'll get a higher ROI from using the frontier. That's my high-level suggestion.

And something true for both of those, related to evals, that I think people aren't doing enough of yet is collecting sessions. Collect sessions, whether for internal agents you build on platforms like CrewAI that make it easy, or for your internal Claude Code and other agent sessions. You learn so much from being able to observe and analyze those sessions. So definitely start collecting sessions.

45:03 · Sessions as the feedback loop

Tom: I'd add on to that. I have my crew team for the SDLC, and I've been checking out Hermes and some of the other popular frameworks and harnesses. I stood up an entirely new project on Hermes and got a team of eight to ten software engineering agents working: AI engineer, backend, frontend, API, all of that. It was doing a pretty good job early. To Devin's point, I'm using large models for the exhaustive tasks, like research, planning, intake and decomposing projects into small story points and tickets. Then I'm using small models like Luna to execute those small-scope PRs and submit them back into the local project.

What I've found is that I can go through an entire project where maybe it took the agents thirty minutes to develop, and another thirty minutes to an hour of back and forth to get to an MVP. I'm having Claude watch that entire session, as Devin just said. That session file becomes immediate feedback, and in my opinion that is the human in the loop, going back to how we started this conversation. Now I can chat with the Claude agent overseeing the process and ask what enhancements or changes I can make to the core project team so there's less human intervention going forward. Why was that one ticket blocked? Well, it didn't have that foundational knowledge or that acceptance criteria. So I go in and tweak that, maybe at the skill or prompt level.

That iteration is how you get better. I'm on project ten or fifteen at this point, and I have much more efficiency. My models are still running efficiently: I use complex models for the probabilistic tasks and smaller models for very fine-grained, scoped tasks. But all that session feedback becomes highly relevant.

46:49 · Agents as users of docs and open source

João: I can also tell you a few things we're noticing out there that are even changing the way we think about development. Starting in June, we officially had more agents looking at our docs than people. Right now the split is 70%: 70% of the access to our docs is agents. That just shows you the volume these things are operating at. It's remarkable. We're also seeing a huge bump in how many people use our open source with coding agents, and we're learning there's different behavior. Humans usually reach for certain features that agents don't, and the other way around. Funny enough, in a conversation we had internally, coding agents seem to be 2.4 times more likely to try to use our paid product. So we asked, should we consider them customers? What does it mean to serve them?

It's even changing the way we develop the framework. Right now, if we add new features to the open source that change the AGENTS.md file people use to build a crew, we have a GitHub Action that checks it across around 223 different sessions, with Opus, Haiku and Sonnet, and with Codex, Luna and Sol, to make sure agents know how to create crews, build them and operate them, and that it becomes second nature, because that's the way people are building these things now. And honestly, a lot of these frameworks, CrewAI included, are now three years old. That's not that old, but three years ago engineering was very different.

Taylor: The speed at which things change on that front, too. At Dosu, we constantly talk about the Opus 4.5 moment. And now, getting into Opus 5.5, things feel very different. It's truly a wild space.

Tom: To João's point, my research agent now has instructions to find open source projects and ignore the paid ones. Whether I'm developing for clients or my own builds, I want this thing to be scalable. There are cool projects out there, like REA (Reverse Engineer Anything), and the commoditization of all these different projects and builds. That's something I incorporated immediately in one of those feedback sessions: we need to do a better job of finding free resources.

49:39 · Wrap-up

Taylor: True. There are a lot of good examples to look at on that front, open source and otherwise, and we can learn from our history. Memory is important as we go from prototype to production. I think that's about it. I'm sure we could continue for hours; there are so many topics to jump into, but there just aren't enough hours in the day. Thank you so much, João, Tom and Devin, for joining today. I hope you had fun listening to all of this. Catch us online if you want to ask us questions, because as you can see, we have no problem having these discussions. Thanks, everybody. Enjoy the rest of your day, and we'll see you around.