Ranking AI Systems: AI Memory Benchmarks

Ranking AI Systems is a new Dosu series in which Taylor Dolezal, Devin Stein and Michael Mangus rank the tools, benchmarks and systems behind AI agents. The first episode ranks AI memory benchmarks by how useful each one is to a team building memory for coding agents, picking up the measurement question our agent memory series closed on. The seven benchmarks fall into three groups: the conversational LoCoMo and BEAM, the general agent DolphinBench and STATE-Bench, and the coding-focused VibeMemBench, SWE-ContextBench and EvoMemBench. Along the way they cover why long context has saturated the older conversational benchmarks, the trade-offs of synthetic data, the difference between giving an agent a fish and teaching it to fish, why benchmark tasks are much smaller than real engineering work, and why off-the-shelf memory systems often do worse the more naturalistic a benchmark gets. They close on what the next generation of benchmarks should measure once task completion stops being interesting. This transcript has been edited for length and clarity.

Transcript

00:02 · Introduction

Taylor Dolezal, Head of Open Source, Dosu: Welcome, everyone. Today we're going to be benchmarking and ranking some benchmarks. We're talking about memory benchmarks with Devin and Michael, hearing what they think about them and what the future might hold, and working through three groups: conversational benchmarks, general agent benchmarks, and coding agent specific ones.

00:30 · LoCoMo

Taylor: Let's start with the conversational benchmarks. One that has come up quite a bit in our work and across the industry is LoCoMo, from Snap. LoCoMo uses generated, human-edited conversations that span many sessions, then asks questions about people, events and details that were mentioned previously. Answers are scored using F1, which balances how many words in the response match the reference answer and how much of the reference answer the response covers. You can think of it as testing whether an assistant can remember and connect details from a long-running conversation. Initial thoughts on LoCoMo?

Michael Mangus, Head of Product, Dosu: LoCoMo is probably the most reported memory benchmark. It's sort of the old man of the category, and for me that's also its main undoing. There isn't enough context in these conversations to actually be challenging for current models. A LoCoMo conversation is on the order of tens of thousands of tokens, which fits easily within the context window, so I'd say it's pretty much saturated as a benchmark. The best solution for LoCoMo these days is to just put the entire conversation in context for the model, so there isn't a lot of improvement to be had. Aside from the limitations of being a conversational benchmark, I don't think it's a challenging enough problem to get meaningful variation.

So for me, LoCoMo is D tier. It's really high tier when it comes to popularity, and in that sense it's a useful comparison point, because it's reported by so many different systems. You can find a LoCoMo score for a lot of things. I will say our own LoCoMo score at Dosu is basically zero, because our system doesn't consider any of the content of these conversations interesting enough to record. It just doesn't remember anything from them, and honestly, I'm not too ashamed of that. I know some companies go back and forth about who has the higher LoCoMo score. I'm willing to say we have a bad LoCoMo score, and it doesn't bother me.

Taylor: Michael gives it a D — hopefully not D for Dosu. Devin, where would you rank it?

Devin Stein, Founder and CEO, Dosu: I like what Michael said: we have a low LoCoMo score, and we're proud. My take is pretty similar. LoCoMo has a lot of historical relevance from when people started getting interested in memory for agents, or really for chatbots. That's the era it was created in, the chatbot era of LLMs, and you can tell from how the benchmark is set up. I think of it as the AI boyfriend, AI girlfriend benchmark: if you have a very long conversation going, can the agent remember it?

Context windows have gotten so long that this isn't very relevant for most use cases, and the way we use agents has completely changed. It's not typically one long-running chat anymore. You have many sessions across different chat interfaces and surface areas. So it's a historically relevant benchmark, and it was useful when it came out, but it's a D for me as well. It doesn't have the meaning it used to.

Michael: It also just has bugs. There's a whole category in LoCoMo that's missing gold standard labels, so nobody reports it. It's not without its operational flaws either.

Taylor: I like the consensus so far. You're benchmarking pretty consistently here.

Michael: Inter-rater reliability so far: very good.

05:01 · BEAM

Taylor: Next up in conversational benchmarks is BEAM, which contains 100 generated conversations reaching up to 10 million tokens, with 2,000 questions about facts, preferences that change, contradictions and other memory abilities. An AI judge checks answers against a list of required details and awards full, partial or zero credit. You can think of it as testing whether an assistant can keep track of a very long conversation as information accumulates and changes. Let's flip the script and start with you, Devin.

Devin: My take on BEAM is that it's LoCoMo++. They saw LoCoMo becoming saturated, where the best memory approach was to shove it all in context, so the natural next step was to build something far beyond the context windows of current large-context models: over a million tokens. But the evaluation criteria and what you're optimizing for are fundamentally the same. It's still a conversational benchmark. It's still AI boyfriend, AI girlfriend, but plus plus — really long-term AI boyfriend, AI girlfriend.

In our view, that's a use case from the past, the chatbot era before agents. It's also very single-player: you're having a conversation with a single assistant for a very long time. It doesn't think about cross-session, cross-person, or different people having different memories. So I still see it as a retrieval benchmark disguised as a memory benchmark, like a lot of the conversational ones.

One thing it does well, and this will be a recurring theme for more modern benchmarks, is synthetic data generation. There's a big focus on how they generated these conversations synthetically, without human annotation or a human in the loop. That's become a big unlock as models have gotten better: you can define what a good conversational memory benchmark looks like and generate all of that data synthetically. That's become the way people create memory benchmarks, and to my knowledge, this was one of the first in the memory space to really take that approach. So it's better than LoCoMo, but I'm going to be a little harsh and give it a D as well. I still don't think it's that relevant to how agents are actually used today.

Michael: I agree, and the synthetic data point is interesting because it's both a strength and a weakness. It lets you generate datasets that are large and challenging for the models. Like Devin said, it overcomes LoCoMo's weakness, because the context gets big enough to strain the models and differentiate performance. At the same time, I always worry about naturalism with synthetic data. The way LLMs communicate still feels pretty alien. I can still readily identify text written by an LLM, so I sometimes wonder how well it transfers to actual human inputs. But it's definitely a better benchmark than LoCoMo, in my opinion.

The other thing that bumps it up to a C for me is that it does have some software engineering conversations. Devin is right that it's ultimately a conversational benchmark, but there are conversations in there about building Python projects and things like that, so it gets a little closer to the domain we're interested in. It's worth noting this is our coding agent memory benchmark tier list, so there has to be some coding represented for a benchmark to climb out of D tier for me, and I think BEAM meets that bar.

The other thing I'd say is that a lot of these conversational benchmarks suffer from something I've been talking about a lot lately: the distinction between teaching an agent to fish and giving an agent a fish. A lot of them, including BEAM, still measure how well you give the agent the fish. In other words, do you retrieve the specific fact you were supposed to retrieve, rather than learning a procedure, or learning how to learn? That's something we'll get to later with some of the more sophisticated benchmarks, and it's much more interesting to test. So it's not going to make its way out of C tier, but I'm giving BEAM a C. If you have to choose between LoCoMo and BEAM with modern models, definitely choose BEAM.

10:13 · DolphinBench

Taylor: That makes sense, given we're talking about coding agent memory and these earlier benchmarks are more focused on conversation. Let's move into the general agent benchmarks, starting with DolphinBench from Mem0. DolphinBench gives agents about 500,000 tokens of history for each of three fictional users, then asks them to complete 600 tasks through simulated applications. You can think of it as testing whether an assistant can use what it remembers about you to carry out a request correctly. Devin?

Devin: DolphinBench is a step in the right direction. The origin is basically looking at something like BEAM and saying, this is heavily conversational, but that's not how agents are used today. The reason you use agents today is not just to retrieve information but often to take actions. So DolphinBench takes a similar approach to synthetic data generation, but in a more modern way. They simulate history in more of a knowledge work scenario, so it actually has an organizational focus. We've drifted away from the AI boyfriend, AI girlfriend use case toward something more like working with an agent at a company. There are three personas — a CEO, an engineer and a product manager — and it simulates a history of messages with those users. Then it tests whether, given a memory, the agent takes the right actions.

So it's better than BEAM, the modern take on BEAM, but it's still a retrieval benchmark at the end of the day. The difference is that it also tests the actionability of the memories that are retrieved. It's not just "did this retrieve the right thing," but "did the agent then take the right action?" That matters, because, as we've talked about before, agents don't always listen to the memories they get. Sometimes they say, "I'm going to ignore that and do what I think is right." So it gets a little into actionability, not just retrieval.

They also do an interesting thing on the dataset generation side, where they use frontier model intelligence when generating the data to "prove solvability," as they put it. That's maybe alluding to the problems in LoCoMo and historic benchmarks where things just weren't solvable. They're able to say it's possible to get 100% on this benchmark.

The only thing I'll say is that it's a little self-congratulating from the Mem0 team. Good on them for putting out this research; I think it's needed. But they come out on top, and I think there's a bias toward how their product operates and how they want it to be used.

I also like the Pareto frontier messaging. I don't think we're necessarily evaluating the right things, but it gets into not just retrieval but the cost and latency implications of memory. At the end of the day, organizations care about that: not just whether the agent retrieved the right thing, but what impact memory has on task performance from a cost and latency perspective.

All that to say, it's better than BEAM and heading in the right direction, but from a coding agent perspective I'm still giving it a C, because it doesn't get into real end-to-end workflows. It's much more conversational, followed by some specific action you want the agent to take.

Michael: I agree with all of that. It's also worth noting DolphinBench is brand new — it came out this week — so we're still learning what it really measures. Something I liked is that they measure compliance with prohibitions, which not many benchmarks represent well: not just what you said you wanted, but what you said you didn't want, and whether the agent obeys when you tell it not to do something. That's a useful thing to measure.

I do agree with Devin that there's still too much personal preference in the data. The very first thing you see when you click through the dataset and look at the personas is a memory that this person likes to get their coffee from Blue Bottle. I just don't think that's an especially useful domain. I understand why it is if you're Mem0, but not for coding agents specifically. I'm torn between B and C for DolphinBench, but I have to agree with Devin. The main problem is that personal assistant work, as useful as it may be for some systems, isn't very useful for what we're building at Dosu.

16:09 · STATE-Bench

Taylor: Moving to our next benchmark: STATE-Bench, from Microsoft. STATE-Bench contains 450 simulated customer support, travel and shopping tasks, with a learning track that lets agents draw on experience from earlier tasks. It checks whether the agent reaches the correct final state and follows the required procedure, and reports task success. A good way to think about it is whether experience helps an agent reliably follow a workflow, such as checking refund eligibility before issuing a refund. Michael, thoughts on STATE-Bench?

Michael: Of the benchmarks we're talking about today, this is the one I personally know the least about. I do know that, in terms of agent domains, it still suffers from poor fit for the kinds of problems we have. The task domains in the dataset are travel, customer support and a shopping assistant, which to me falls in the same category as DolphinBench. There are certainly a lot of people building agents for whom that's a useful set of trajectories to evaluate on. It's just not the kind of trajectory I think transfers all that well to software engineering.

That said, I like some of the structural ways they've approached building it. Taylor, as you mentioned, there's a dedicated agent learning track in the evaluation metrics that's distinct from how well the model performs the task. I like that there's that tailored memory and online learning component. Domain fit still isn't great, but at least it's about successfully navigating a trajectory to accomplish tasks, even if those aren't engineering tasks.

For me it's borderline. I was going to be generous and give it a B, because of how they've structured it and the perspective they took in constructing it. But am I going to use it to evaluate Dosu directly? Not really. So maybe it's really a C for that reason. If I'm giving DolphinBench a C, I kind of have to give STATE-Bench a C.

Taylor: What do you think, Devin?

Devin: Overall I agree with Michael. What I like about STATE-Bench is that, even from the title of the paper — benchmarking AI agents on enterprise workflows — it's actually thinking about how agents are used at enterprises. The domain overlap with Dosu or coding agents isn't very high, but it's one of the first we've looked at with an emphasis on procedural knowledge. It's trying to capture the repeated workflows agents do, and with coding agents that's a really big area of knowledge capture: how do you debug something, or what's the right way to create an eval or benchmark something internally? These are very bespoke flows that many engineers do, and STATE-Bench gets at the idea that agents at enterprises need to understand repeated workflows and how they transfer across sessions.

That said, the learning track felt a bit like an afterthought, like "let's also add this on to see how memory might impact it." When I dug into the data to see what you're supposed to learn, there aren't that many opportunities for memory to really move the needle. Especially because the average cost per task is three to four cents, sometimes twenty cents. If you look at your own coding agent usage and costs — say, using a tool like our open source project to analyze your token spend — I guarantee you're spending more than twenty cents on any substantial work. So it doesn't quite have the meat you'd need for a proper agent memory and learning benchmark, in my opinion. All that to say, I'd give it a C as well. It's directionally correct, but it doesn't quite focus on the types of tasks you'd see in production.

21:16 · Why benchmark tasks are smaller than real work

Michael: I wanted to pick up on that cost point. There are a lot of complicated reasons for it, but I've noticed across a lot of benchmarks that the tasks just aren't on the same order of magnitude of effort as a lot of real-world engineering tasks. Obviously it's easier to get funding as an academic lab for a cheaper task set; you don't need as big a grant. The other component is that there seems to be good evidence that the more expensive a task is, the more variable its cost is. If I repeat an expensive task many times, I get a wider standard deviation around the central tendency than I do with a cheaper task. When you're constructing a benchmark, there's a temptation to pick lower-variance tasks because it's easier to build statistical models around them. You don't need as big a sample to draw conclusions. So there's an incentive to benchmark on low-variance tasks, but they're not representative of what actual work is like. That cuts across a lot of these benchmarks.

Devin: That's a really good point. Another thing I'll add: especially for coding agents, and probably most enterprise workflows, a lot of the complexity comes when you don't actually know what you want, or the whole solution, yet. With synthetic data generation, you're often starting from a PR or some end state you want to reach and then recreating it. I don't think existing benchmarks do a good job of recreating the winding path that real-world tasks typically take.

Michael: That's a great point. If you're doing academic research, you're really interested in control; you want to isolate variables. In the real world we don't have that luxury most of the time. There's a lot less control over the solvability of the task, its clarity, the outcomes and so on.

23:27 · VibeMemBench

Taylor: That gets us through our first two sections. Before the last one, it looks like we mostly have consensus, with not too much variance in the stack ranking, but we'll see how the tier list ends up. On to coding agent specific benchmarks. The first is VibeMemBench, from Alibaba, which pairs 111 coding tasks across 90 repositories with many thousands of records of earlier agent work in those repositories. It tests whether lessons from previous coding agent work help future agents solve another task in the repository correctly and with less effort. This brings us closer to the code, and to what's going to be helpful for teams like us at Dosu. Thoughts on VibeMemBench?

Michael: This is another one that's hot off the presses. It was published in September, so it's very new, and you won't see many people reporting results on it yet. But it's the closest published work I've seen to how we've been approaching evaluation internally at Dosu, and to the setup where we've seen the most success with our memory system. It has a lot of historical trajectories for context on a relatively broad sample of repos. They focused on pulling from SWE-rebench, which tries to avoid contamination. We'll talk about SWE-bench tasks a bit later, but there's a lot of contamination in some of the more popular, older SWE-bench-style tasks. So I'd say it's relatively clean in terms of not having too much trained directly into the models.

The naturalism of the data is really high. The combination of coding targets — tasks to be done — with a much bigger collection of historical agent trajectories on those repositories is good, because it gives you cross-task transfer: how do you teach the agent to fish over a big sample of data? Maybe I'm biased because it's the closest to what I've been working on, but I've been working on it that way because I think it's the best way to do it. So right now, VibeMemBench is my only real S tier on the board today: really high naturalism, and a real possibility of measuring the interesting parts of memory — how things work with a long history rather than a short one, and with low control rather than high control. It's still new and maybe unproven in a lot of ways, but this is the direction I'd like to see coding agent benchmarks go. And as you said, Taylor, it's the first one we've talked about today that explicitly targets coding and engineering work as the main output.

Taylor: It's helpful to have that control, with a corpus of code as context, while getting closer to real-world circumstances where you have a feature or something new to repair or work with. Devin, thoughts on VibeMemBench?

Devin: I'll echo a lot of what Michael said. I was very excited by VibeMemBench. The name is kind of funny, but it's the most naturalistic in terms of how people actually use coding agents. There are still some synthetic processes for recreating PRs from historic information, so you don't get the winding path we were talking about.

What I really like, beyond the benchmark construction, is that they were really thorough in analyzing the best-case scenario for memory on the dataset they created. They analyzed all the traces with an oracle: here are the memories, repeated pieces of information or mistakes we know should positively impact the trajectory — what's the best case? They show there is a boost, but it's not that significant. Then they take a bunch of off-the-shelf memory providers and test how they do. Spoiler alert: not very well. Some are actively harmful; Mem0, for example, does very poorly on coding agent benchmarks.

They also go a step further and ask why existing memory solutions aren't working, and start building a taxonomy of what went wrong. There are interesting findings there: is it a coverage miss, meaning the memory wasn't captured correctly, or a search failure? Interestingly, there aren't many search failures, because retrieval is what memory systems have optimized for. The other most common failure mode, which ties back to our previous webinars, is what they call form degradation: the information wasn't preserved correctly, or it's buried in too much other information. So how you store memory is really important for how agents will use and interpret it later.

Overall, VibeMemBench is the only one we've talked about so far that's applicable to this domain and starts to bring the kind of rigor we've been trying to bring at Dosu to understanding how memory, knowledge reuse and knowledge transfer affect coding agents. I'm going to give it an A. It's almost an S, but I'll be a little controversial. For me, it's breaking ground on what a good coding agent benchmark looks like, but I think there's better work to be done on the benchmark itself. They did a really good job with a clear, obvious way to build a coding agent memory benchmark — the PR replay approach, looking for task overlap, then analyzing the failures. But given those failures, I have a lot of ideas, and there's a lot of work we could do to reshape how these benchmarks are generated so memory can have a bigger impact. So I'd give it an A.

30:50 · SWE-ContextBench

Taylor: Next up is SWE-ContextBench, which groups related GitHub issues and pull requests so agents can use records of earlier coding attempts while tackling another problem. There's a light version with about 300 experience tasks and close to 100 evaluation tasks. A good way to think about it is whether an agent can recognize it's encountered a similar problem before and reuse that experience. Michael, how are you thinking about ranking SWE-ContextBench?

Michael: I like the idea a lot. As you mentioned, Taylor, it explicitly targets reuse between tasks, which is one of the reasons I like VibeMemBench, so I'd put SWE-ContextBench in the same bucket. What I dock it for is, first, that if I'm not mistaken, a lot of these tasks are ultimately SWE-bench tasks, or derived from them, and I think those are largely memorized by current-generation models. There are papers that look into this, and I think the evidence is pretty good: if you ask a model to solve a SWE-bench task without giving it a checkout of the repo or any search tools, it still knows which file to edit.

The reason that's a problem for a tool like ours is that the main way memory makes your agent more efficient is by reducing turns — eliminating tool use the agent would otherwise have to do. If the model already knows which file to edit without any tool use, your trajectories are already compressed, and memory loses the ability to fill that gap, because parametric memory, what's trained into the model, is already filling it. That's a risk with SWE-ContextBench, although they draw from multiple data sources, so maybe it's not as bad as I'm making it out to be.

The other thing is that it's still pretty simple. They include some example trajectories in the dataset, which I like and wish more benchmarks would do, but they tend to come from smaller and older models. It doesn't tell me much about the current state of frontier LLMs. That's a theme: a lot of academic work struggles both to have the budget and to publish fast enough to tell us about current-generation, high-parameter, high-effort model configurations that people use for real software work. We've sampled our own baseline trajectories from the same task set for some of our evaluation, and it's helpful, but it also suffers from tasks that are a bit too small. It's hard to find good opportunities for significant savings from memory at the scale of these tasks.

So it could be better. I want to give it an A, but maybe I'm being too generous, so I'll give it a B. It's conceptually right in a lot of ways; I just wish it were a bit newer, a bit more challenging, and a bit less trained into the models.

Taylor: Devin, do you feel similarly?

Devin: Very similarly; not too much to add. Conceptually, I like SWE-ContextBench. They got at the right idea — how effectively can you transfer knowledge across similar tasks — which is really important for memory and coding agents. But the eval is somewhat simplistic compared to the deep analysis in VibeMemBench. They're really just looking at accuracy, time efficiency and cost efficiency.

On accuracy, something we've talked about a lot internally is that most agents today, especially with frontier models, are pretty good at doing tasks. So accuracy, meaning how many tasks were successfully completed, is becoming less useful as a metric, except maybe on ultra-hard benchmarks. For naturalistic work at a company, coding agents can pretty much do everything at this point. It's about how well they do it, how they do it, and the path they take to get there. But a lot of these benchmarks still use accuracy as a proxy for whether the agent got better and learned something.

Something else that calls back to our previous webinars: Sonnet 4.5 did really well here, but it was also the model extracting the memories. We've seen over and over that agents do better with memories they create themselves, when the recipient model is the same as the donor model. That came up in this paper as well. Overall, it's directionally correct, but there are issues with the SWE-bench tasks they sample from, and it's a bit simple on the eval side. I'll also give it a B.

36:38 · EvoMemBench

Taylor: Last but not least, we have EvoMemBench — don't try to say that five times fast. It combines question answering, tool use, web search and simulated household tasks to test memory both during a task and across separate tasks. A good way to frame it is that it tests two kinds of learning from experience: keeping track of what's happening now, and carrying useful knowledge or procedures into future work. Michael, your thoughts on EvoMemBench?

Michael: It definitely has a good shape. I like the kinds of data it's trying to evaluate. Taylor, you mentioned long-term patterns earlier, and Devin spoke about the value of that in STATE-Bench. EvoMemBench has the same concept built in with its cross-episode learning. To succeed on that part of the benchmark, the agent has to abstract out the patterns shared across episodes, which I like.

It's also good that they distinguish knowledge from execution — another way to put it is semantic versus procedural memory. That makes sense as a move, and it's increasingly represented as a concept in memory systems: the difference between knowing about something and knowing how to do something.

It's still not really direct coding work on a lot of these trajectories, but it's closer. Some of it is having an agent do successful web navigation, which my coding agent sometimes needs to do, at least in a limited sense. Then there are embodied agent tasks, which probably don't transfer well to our domain — the same category as STATE-Bench or DolphinBench. I'm sure it's a good benchmark for somebody, just not so much for me on those tasks.

Overall, the paper behind it has the right perspective, and it's good at pushing on whether memory systems actually help agents. Devin touched on this earlier: a full-context baseline — just shoving all the relevant context into the context window — is still competitive with, or even outperforms, a lot of memory systems. That suggests a lot of the abstraction or retrieval precision those systems are trying to build isn't actually steering agents toward more productive trajectories. I thought that was a useful finding. Earlier I said DolphinBench and STATE-Bench are borderline B or C for me, depending on how much we penalize them for not being about coding agent work. EvoMemBench is close enough that I'll put it in B tier. That feels right for what we're doing at Dosu.

Taylor: Devin, what are your thoughts on EvoMemBench?

Devin: I agree with a lot of what Michael said. It's similar to STATE-Bench in looking at actual workflows agents do and the impact of memory on them, but unlike STATE-Bench, memory is a core part of it rather than what felt like an afterthought. I like how they categorize different types of memory for this benchmark: cross-episode knowledge, cross-episode execution knowledge, and in-episode memory. I like the framework they apply to different memory types and where that information comes from. Again, the procedural focus is really important for real agents at companies, but it's not necessarily in the coding agent domain. The success criteria are answer accuracy, task completion, and efficiency in tokens and number of steps. So it's pretty good. Overall, I'd also give it a B.

41:14 · Final rankings

Taylor: Any second thoughts before we wrap? Now that we've gone through everything, do you feel good about your tier rankings, or any last-minute changes?

Michael: I feel a little bad having BEAM in the same tier as DolphinBench and STATE-Bench, because of that conversational penalty. What I kind of want is an F tier to put LoCoMo in, so I can move BEAM into D tier. Sorry, LoCoMo. Like Devin said, it was a great benchmark at the time. That's my only hesitation with the final ranking.

Devin: I agree. We could basically knock LoCoMo off and feel a bit better.

42:02 · What memory benchmarks should measure next

Taylor: There were so many other benchmarks we could have talked about. To end today, I'd love to hear how you're thinking about the future. Are there benchmarks you'd like to see, or things that are close but not really a benchmark yet? Let's start with you, Michael, and then Devin.

Michael: I think the biggest open question here is why memory systems seem to perform worse the more naturalistic the tasks get. Broadly speaking, there are two possible reasons. One is that the memory systems really aren't helpful: they're targeting the wrong things about how coding agents work, so they can't actually change the agent's behavior. The other is that we're measuring the wrong outcomes. That's the more interesting answer, but harder to grapple with, because every benchmark builds in assumptions about what success looks like.

Devin made a good point earlier about task completion. In my day-to-day work, it's pretty rare for the agent to just not do something. It'll output a change. It may not do it efficiently, it may not be exactly what I wanted, it may be kind of low quality, but it will do the task. So if task completion is no longer the most interesting question, what is? Framing it in terms of cost can make sense, but sometimes you get higher-quality results when you pay more; you do sometimes get what you pay for with higher effort levels from coding agents.

That's what the next generation of coding agent benchmarks has to answer. What do we want to know about agent performance? How can we decompose it into more granular variables we can actually manipulate and control? Right now it's such a complex system that it's hard to isolate a variable and change just one thing about the agent's performance. And how can we build naturalistic datasets that mirror what it's like to work in a private repository — which has no chance of being in training data — while making sure the datasets we do produce don't just get trained into the models, because models get trained to complete the benchmark successfully, and then we end up on a treadmill of contamination?

That's why part of what we're doing at Dosu is building tools to build benchmarks, and there's been interesting recent work from other organizations on this too. How can we build a benchmark on a specific customer's data, or internally on our own data, that we know has no risk of contamination and really high naturalism, but that we also know many memory systems struggle to help with?

Devin: Well said. The only thing I'd add, building on VibeMemBench, is that the biggest open question really is measurement, like Michael said. What it means to have a high-performing agent, or set of agents, at an organization is a really interesting question that I don't think we've gone deep enough into yet. A lot of these benchmarks simulate agents doing tasks, but a lot of work at companies still happens with humans in the foreground: "No, actually do this," or "I don't like that, do that." We're missing that human interaction element, and that's often where memory can have a really big impact. You don't have as many interruptions, you don't have to correct the agent again in the same way, and it doesn't confuse a topic it confused before. Figuring out how to simulate that in benchmarks is really hard, and it's something we're really excited about. Measurement, and understanding what good memory and good agent performance look like at organizations, is going to be a big area going forward.

46:40 · Closing

Taylor: Awesome. I hope you all enjoyed our tier list today. We'd love to hear how our rankings stack up against yours, and join us for more as we get into other topics around agent memory. Devin, Michael, thank you so much for your time. Looking forward to digging into more benchmarks. Thanks, everybody. Talk to you soon.

Devin: Thanks, Taylor.

Michael: See you all.