AI Memory News: Muse Wonder, Haiku 5.5 and Decision Models

In episode 7 of AI News with Dosu, Taylor Dolezal, Devin Stein and Michael Mangus discuss model releases, agent memory and what happens when agents' tools fail. The conversation covers Meta's Muse Wonder, Haiku 5.5, PlanBench-XL, Cognition's memory system, Google's developer docs, decision models and AI-written tests. For the team's ranking of agent memory benchmarks, see Ranking AI Systems: AI Memory Benchmarks. This transcript follows the published video and has been edited for length and clarity.

Transcript

00:00 · Cold open and AI news rundown

Devin Stein, Founder and CEO, Dosu: We've jokingly said before that docs are dead. Docs traffic is predominantly agents right now. If agents are the ones consuming your docs, just make it easy for agents to consume them.

Taylor Dolezal, Head of Ecosystem, Dosu: So what exactly has happened this past week with AI, with models, and with everything else going on as it pertains to knowledge infrastructure and managing your agents? I feel like we've got quite a bit to go into this week.

00:32 · Meta's Muse Wonder

Taylor: Kicking things off: yesterday, at the time of this recording, Meta's Muse Wonder came out. Have either of you gotten a chance to use it or formed any opinions just yet?

Devin: Meta's Muse Wonder specifically, from a benchmark perspective, is really impressive. Now Meta is in the game in terms of frontier models. What I really liked about their release, and I personally haven't played around with it much, is that they were pretty opinionated about what they want it to be good at. Specifically, they want it to be good at their Muse product tasks, so personal agent and assistant types of tasks.

A few things were interesting. They actually had a big emphasis on memory as part of the post-training, so it's really good at managing user-specific memories, as you would want for a personal agent. We also ran it through some of our Dosu Decant benchmarks for token usage and token efficiency. At least for coding tasks, which is where that benchmark is oriented, it's a performant model, but it's also expensive at coding tasks relative to Opus 5.5 and Sol 6.1. So I don't think I'm going to be adopting it as my daily driver. But overall, it's a very impressive launch from Meta.

Michael Mangus, Head of Product, Dosu: I'm always just glad to see more competition. There was a period of time a few months ago where I was worried this was going to turn into a one-horse race, so I'm glad to see that. I think it was also interesting that, if I understand right, you need a subscription for the full model. It wasn't lost on me that they released Wonder, and then Wonderful was the full model with the subscription. I'm a fan of puns, but I thought that was a bit heavy-handed.

Devin: That was clever. On the Meta subscription side, I actually already have a subscription, because, a little pro tip, Spark is a decent model already. With their subscription, if you're working on an open source project or some side project where you really don't care about Meta training on your personal data, you can use their contributor model, which is a discounted rate for allowing them to use your data. It's basically free coding with a pretty good model. So I use it for side projects where I'm like, eh, use this data. I'm just burning tokens for fun.

Taylor: Hopefully it's a wonderful life, at least for Meta.

03:05 · Haiku 5.5 and daily drivers

Taylor: Moving along, we've had some releases from Anthropic too. I'm personally very excited to see Haiku 5.5 come out. I'm happy about that pricing structure, and it seems like a really capable model.

Michael: I think it's crazy that it went almost a full year without an update to Haiku.

I'm excited for it. I think it's always good for us for some of our smaller, simpler tasks. I'm sure we'll talk about this later, but there's more pressure on the decision model side that I think is going to crowd out some of the small model use cases. I don't think summarizing an email really requires a maximally sized latent space, and we might be getting to the point where the small models just aren't going to change that often. I think that's interesting.

Devin: I had a tweet from six months ago begging Anthropic to release another Haiku model. I've always been a huge fan of small models because of all the background tasks we do at Dosu, where you just need something that's smart enough and fairly cheap. From our early benchmarking, it's very good. It's unclear how much better it is than GPT-6 Luna, which is a really good, small, cheap model.

Michael: Although I've got to say, Sonnet has always felt like it's in a bit of a strange place for me. There aren't a lot of tasks where I want that mid-tier model. Most of the time I either want the maximum intelligence I can get, or I want a Haiku, or, like I was saying a minute ago, Jev or a decision model for some of the really small stuff. So I feel like Sonnet has always been in this awkward in-between for me. It definitely has some use cases in our internal pipelines, but I don't really reach for it in my own day-to-day work very often. Maybe Devin does, I'm not sure.

Devin: I agree. I don't think I've ever used Sonnet or Haiku in Claude Code. Just never. I'm always using Opus. Even when I hit my usage limit with Opus, I'll probably go use Sol 6.1 over switching to Sonnet or Haiku. Benchmarking when these models come out is just for Dosu product-related tasks where we have evals. But for personal use, I'm always on Opus.

05:21 · When tools break

Taylor: Moving along, I took a brief look at the PlanBench XL paper that came out, but I haven't had enough time to fully digest it. I wanted to talk a little more about it. When tools break, the failure modes of agents, is the takeaway I got from this one.

Michael: I thought this was a really interesting paper, and I think it has a lot of implications for our product going forward. It's something I've noticed a lot in our benchmarking: models are bad at recovering when plan A fails. What they do in this paper is manipulate tools to make certain tool calls impossible, and then look at how that degrades task performance. It turns out that even if there are workarounds available, the models really struggle to find those workarounds and use them effectively.

I think one of the results of high environment friction is actually increasing misalignment in the models. For a long time, the problem with LLMs was that they were sycophantic. They were way too compliant with your instructions. But I feel like the recent generations, which have been post-trained really heavily on verified rewards, are swinging the other direction. If they find themselves sliding down that optimization gradient, they get kind of desperate and start doing strange things. In many ways, I think that's actually the root cause of the Hugging Face incident, for example. You give a model a task it can't complete, but it so badly has been post-trained to try to complete it that it starts doing things you wouldn't necessarily want it to do.

We see this with testing too. It'll either fake executing tests or write tests that are extremely thin in what they actually exercise, essentially just testing that it implemented the code it implemented. So I think this is really interesting, because there's a degradation in the quality of the output above and beyond "I couldn't use that tool." I think it actively makes the model worse over the rest of your session, and I think they demonstrate that pretty well with the evidence they collect in this paper.

Devin: I think that's well said. I'd just add, from my personal experience, I see this with Claude Code or Codex on my personal tasks. If it's trying to do something and it can't, that's when it starts to do weird workarounds where I'm like, wait, why did you do that? Whether it's manipulating the database to get a test to pass, or ignoring certain types of tests because they're not passing. I see this sort of workaround friction even in my own day-to-day use.

Taylor: I relate to that. I use 1Password, and I like signing my commits, so I have a biometric sign-off as things get committed and pushed up. If I walk away from my machine, there's still a human in the loop before something gets pushed. But in many cases, I've had models say, "Okay, I can't access 1Password, or this isn't coming back, and you don't have a key on disk, so I'm just going to push over HTTPS," or something else like that. Albeit that's not the worst case, but deleting production databases or doing weird things to pass tests has come up quite a bit.

08:31 · Where does agent memory live?

Taylor: Cognition introduced personal memory for Devin this week, and it looks like they're starting to place that knowledge closer to the code itself, in Git. So: agent memory that maintains itself, and then, where does knowledge live? I'd love to hear your takes on what Cognition is thinking with Devin versus what we're doing at Dosu, and how we place knowledge and where we store it.

Devin: I saw this announcement from Cognition, and honestly, I was surprised they didn't already offer this. This is basically Cognition's response to Claude Code's native memory or Codex's native memory. Two things I thought were interesting about it. One is that the current definition of how the memories are stored is very user-specific, which surprised me, because Cognition is all in on cloud agents, and cloud agents are inherently a team-based product, not necessarily a personal one. So it was surprising that they're starting with user-oriented memories and preferences rather than team-based memories and preferences.

The other thing I thought was interesting was the agent memory repo open source project. They're basically trying to open source the standard for how memories are stored in a repository. This is competing with the Open Knowledge Format, or OKF, from Google. We've been really interested in this knowledge standardization space, because it does feel like there's a really strong need for standardization beyond just Markdown, which is, I guess, the closest we have to a standard today.

What I like about the agent memory repo is that it feels more specific to coding agents, more agent-native memory, whereas OKF feels more wiki-oriented, and a lot of its examples come from Google's or GCP's products related to data analysis. I'm excited about the agent memory repo. We'll see how things play out between that and OKF, or whether another standard emerges.

Michael: I think it's interesting that there seems to be some convergence on the strategy for delivering memory. What Devin, the product, has implemented is really similar to what Claude Code does, insofar as there's a MEMORY.md that gets injected early on in a session. It contains, I think, some recent notes and then an index of other information, with tools for the model to look things up and dig deeper. That's increasingly how we're thinking about iterating on our memory system internally as well. There seems to be a big advantage to what I call a push-pull architecture: it sends memory into the session no matter what, and then has tools for the agent to optionally follow up on some of those details. I find in benchmarking that it does steer more effectively.

Cognition, Anthropic and others have also converged on that model. Even if you go back to the Vercel blog post from much earlier in the year about making your AGENTS.md effectively a documentation index, I think this is essentially that idea carried through to its logical conclusion in a lot of ways. So I think that memory delivery strategy is becoming standard now.

I think what we're increasingly learning, and we've talked about this in previous episodes as well, is that there isn't just one purpose consumers have in mind when they say they want their agent to have memory. Like Devin spoke to a minute ago, there are obviously questions of personal memory versus organizational memory.

Going back to the earlier example of Muse and its active memory, the way they're benchmarking that is, again, long-context conversations with many turns, where they're looking to reach back deep into the context. If that's what we're benchmarking on, that obviously leads to a different memory design than something like avoiding procedural friction, like with the tool friction paper we were discussing a minute ago. Those are just different purposes. So I think what we're going to learn over the next few years is that you have to evolve memory systems to serve certain needs, instead of treating it like one huge problem space that you're going to tackle with a single solution.

13:06 · Google gives agents more ways into docs

Taylor: In a similar vein, Google added some ways for its agents to work with developer docs and command line tools, giving more on-ramps for knowledge, learnings and documentation overall, which I think is pretty exciting. You no longer have to pull up the right SDK or framework pages. You can give your agents more tools to go read things and stay current. I wasn't sure whether this was more competition with something like Context7 for pulling in docs. I think that's usually what I'll reach for as I'm working with my agents. But Devin, to start with you: thoughts on Google making this move, and whether it'll be helpful or there's anything to learn from it?

Devin: I think we can keep this one pretty short. We've jokingly said before that docs are dead, and I think this is an example of that. We're seeing these statistics about the web generally, but docs traffic is predominantly agents right now. If agents are the ones consuming your docs, just make it easy for agents to consume them. Don't worry about the UI experience. You don't need an "ask about your docs" feature anymore. Just make it easily exposed to agents, whether through a CLI, an API or MCP. That's all that matters today.

Michael: I really question the utility of some of this, especially for things like BigQuery. Frontier models know about BigQuery. They don't really need to reference general documentation. What would typically help them are procedural runbooks for how to do very specific operations. But even more than that, what they really want to do, if left to their own devices, is verify things themselves by experimenting, not by reading documentation. I think the days of agents primarily reading documents for context and then trusting them are kind of behind us. That behavior has been largely de-emphasized in the post-training of models over the past nine months, maybe a year.

15:08 · OpenAI's Decisions API

Taylor: Cool. Well, as promised, it's decision time. OpenAI released their Decisions API, which is somewhat similar to TypeSafe's Jev, likely in response to it, and it's a new model and a new concern to think about. Have either of you gotten a chance to use OpenAI's Decisions API and compare? I think I might have seen some work from you, Devin, on that front.

Devin: Yeah, I did some benchmarking. It's decision model mania right now, ever since the Jev launch. It's actually pretty wild. It speaks to the general market and how easy it is to fast-follow, even fine-tuning models. In today's world, agents are really good at building models, so you've seen this rush into the space once Jev launched and hit, from what was leaked, a hundred million in ARR in one week, which is insane.

From our benchmarking, the Decisions API is a better way to use GPT-6 Luna for decisions, but compared to Jev, it's slightly slower and slightly more expensive. So at least for our internal evals, it didn't make sense to switch from Jev to the Decisions API. I think it's an interesting offering, especially for people who want to stay in the OpenAI ecosystem, maybe because they have a bunch of credits. I've also been watching a lot of the decision model work coming out of Cloudflare. They have a model I haven't benchmarked, but it's supposedly on par with Jev and cheaper. I'm really curious to see what TypeSafe, the company behind Jev, does in the coming months to differentiate.

Michael: I'm excited for this new product category. I think zero-shot classification is one of the most widely used applications of LLMs, and a lot of the time it feels like overkill. So I'm really excited to see Jev, OpenAI and many other players enter this space of models optimized for that single, fast decision use case. I think that's going to be really helpful for a lot of what we do.

Taylor: I'm honestly surprised we haven't seen anyone work on the decision model for which decision model to use, in true engineer fashion. It's only a matter of time before we get to model routing.

Michael: Somebody's building it right now, Taylor.

Taylor: I'll be busy this afternoon.

17:35 · Do AI-written tests help?

Taylor: To cap things off: who checks AI-written tests? Raise your hand. There was a fascinating paper we were sharing around our Slack this week asking whether AI-written tests help, and based on the evidence, it appears not. You might be better off not having your AI write those tests.

Michael: I had an UltraCode session write me thousands of tests over the weekend, and I can't imagine they're all good. What's interesting to me is that I believe the unit test is actually more about human understanding than it is about preventing defects in your software. What unit tests serve to do is make sure I'm thinking about the problem, and the constraints I've put on the system, in the right way. Increasingly, as coding agents are the ones writing the code and plumbing up these relationships, I'm not so sure unit tests are going to serve much of a function in the future.

They've always been something a lot of engineers struggle to do well: defining tests in terms of the contract or the behavior of the system, rather than testing the implementation they chose. So it's not surprising to me that models have some of those same problems. I think what we have to focus on is end-to-end testing and evaluation, making sure the product functionality comes out the way you expect. I'm not that worried about the mechanics of every little step along the pipeline anymore. I'm reaching a point where I'm willing to delegate that to a coding agent, and I'm not sure I really need tests, or that it needs tests, to do that task effectively. At least not unit tests.

Devin: I would agree. I'm seeing the same thing. Agents often write tests as a way to verify their own behavior, but the tests they write aren't necessarily useful to keep around. But it is interesting: it's the first example of the tools agents need to write software differing from what humans actually need.

Michael: Where I am excited is that agents make it easier than ever to write complicated end-to-end tests. They're so good at giving me example data and building the database seeds and fixtures that were so annoying to put together, and at giving me build tooling to understand what's happening while a complex system is being constructed. That's where I'm actually really excited about testing evolving. It's going to be a lot easier for me as a human to interact with and verify little steps along the way that I wouldn't have bothered writing tooling for, or getting data for, in the past.

20:14 · Closing

Taylor: Wonderful. That's all the news we have this week. I'm sure we won't have any shortage of it as we continue the series and look at everything going on in the space. Thanks, everybody, for joining us. Let us know what you think, and let us know your favorite news and the notable things you're finding. We'd love to chat about it. Thank you, Michael. Thank you, Devin. Catch y'all soon.

Michael: Thanks, y'all.

Devin: Thanks, everyone.