LLM Wikis and How to Give Your Agents Memory

Karpathy's LLM wiki idea has become one of the fastest-moving conversations in agent engineering. LangChain's Harrison Chase sits down with Brace Sproul (LangChain), Devin Stein (Dosu), and Jeff Huber (Chroma) to define what an LLM wiki actually is, argue about whether it's the right abstraction at all, and get specific about the two hard parts nobody has solved: maintenance and measuring ROI.

Transcript

Harrison Chase, Co-Founder and CEO, LangChain: Welcome, everyone. We're really excited to be doing this. I'll wait a second for Devin and Jeff to join us — and there we go. Alright. Thank you all for tuning in, and thank you Jeff, Devin, and Brace for being here.

This was a last-minute webinar. Normally we try to plan these about a month ahead of time, but it's such a hot topic that we decided to put it together last minute, after talking with Devin last week and then texting with Jeff. Memory and knowledge have always been hot, from the early days of RAG through to memory. And wikis — with some of the work happening in the coding space, and with Karpathy — have emerged as an interesting way of representing this. Everyone on this call has been thinking about knowledge management in some form for a while, so I'm excited to do this. I think we've got some opinions here as well. Devin told me he has a hot take that wikis are the wrong abstraction, and we can get more into that later.

Demos: OpenWiki, Dosu, and Chroma

Harrison: To start, I figured it would be good to have everyone talk about — slash demo — what they've been working on, for no more than two minutes each. This is a hard stop; I'm going to actually time it. Let's go Brace, Devin, Jeff, in that order. If you could each introduce yourself and then do a two-minute-or-less walkthrough of what you're building and how you're thinking about wiki slash memory slash knowledge. Brace, you're up.

Brace Sproul, Head of Applied AI, LangChain: Sounds good. Hey, everybody. I'm Brace. I lead platform AI here at LangChain, and I've been thinking a lot about wikis and memory recently. Let me figure out how to share my screen, and then I can walk through a demo of OpenWiki, which is our new memory CLI agent that we just released.

OpenWiki is a new memory CLI, and we actually released a new 0.1 version this morning with a general-purpose memory module. The idea behind it is that, in our experience, generating and maintaining memories is difficult, time-consuming, and tedious. So we wanted to abstract all of that into a CLI that's easy to set up and then just runs in the background without you needing to think about it.

It's a CLI you can install through npm. Getting it set up is easy, and you can use it to document either your own personal work and general-purpose memory, or codebases. For this I'm going to show off the personal memory module, since that's what we just released this morning.

You run openwiki personal init, then you configure your model provider and your model, and then we ask you to set a prompt. We call this a wiki brief. You can think of it as a prompt you'd give to your memory agent so it knows what to focus on, what not to focus on, what you want it to remember, and how you want your memories structured. We provide a default, as you can see here, so I'll accept this.

The nice thing about OpenWiki is that it automatically updates itself. One of the most tedious parts of memory and documentation is that you need to keep it constantly up to date. So we just have it run a cron job every day — we suggest something like 2 a.m. — and we can also make it wake up your Mac in the middle of the night to do that. Then you connect your providers, like Notion, Gmail, and Slack, and it ingests data from all of them. Once they're all connected and it runs, it will —

Harrison: Time.

Brace: — automatically write docs.

Harrison: Alright, we're harsher than the Academy here. I was trying to get some music to play you off, but I couldn't figure out how to do that in time. Devin, do you want to go?

Devin Stein, Founder and CEO, Dosu: Yeah, thank you, Harrison. Thanks, everyone, for joining. I'm Devin Stein, CEO and founder of Dosu. Dosu is knowledge infrastructure for agents and humans — we help capture, curate, and share knowledge to make agents more token-efficient and reliable. I'll do my power demo. Can you see my screen?

Harrison: Yes.

Devin: So with Dosu, we plug right into your coding agent workflow, mainly through MCP. As agents work on a task, they use one of our APIs or tools for reading or writing knowledge.

If we take an example task — let's say we're creating a new user-facing database table — what Claude is going to do in this case is work, and learn about your project and your organization as it builds the feature. With Dosu, it's able to capture all of those learnings as persistent knowledge for your organization. For the sake of time: you'll see a call like this, where it writes down whatever it learned. In this case, how schema tables and database migrations work here.

In the background, that kicks off Dosu's librarian agent to generate what we call a knowledge index, or a topic page. It's human-readable, which is very important, but optimized for agents — so it's concise, highly referential, and information-dense. Whenever an agent works on that topic again, we auto-inject the page into the Claude Code session, and we automatically keep the knowledge up to date as the code changes.

And then lastly — the hardest part about wikis, which I'm sure we'll talk a lot about — is understanding the impact. So we also give you analytics to understand how your agents are actually using the wiki and what impact it's having on your organization.

Harrison: Awesome. Right on time, super prompt. Thank you, Devin. Alright, Jeff.

Jeff Huber, Co-Founder and CEO, Chroma: Okay, great. I don't have a live demo to share, but I'll attempt to describe why we're interested in this topic.

Brief history: if you go to the aigrant.com website, you'll see Chroma described as programmable memory for AI. That's from 2023, I believe, when we sort of coined that. So I've been thinking about this problem for many years now. You may know Chroma as a so-called vector database. I think the problem we actually solve is that we figured out how to store massive amounts of unstructured data and make it extremely queryable by AI and by humans, and that's the key job to be done. We also released a model in the spring called Context 1, which is state of the art at the task of agentic search.

One way we think about this wiki thing is that it's a new kind of index. It's a way to process large amounts of unstructured data and create some level of a map on top of it. Very crudely: if you have no indexing at all, that's just a big blob of data — you can send in probes and guess and check. If you have embeddings, sparse embeddings, you have the topology of the map, which is still kind of guessing and checking. And then the wiki adds the road network, all the pipes and the connections between these places, and allows agents to better traverse things. It avoids context rot and lots of other problems. So we're really interested in this idea of wiki as a new form of indexing.

We've also been experimenting on our own team with building our own organizational memory by ingesting lots of unstructured data. I built one of these things. I'm very interested to hear these hot takes about wikis being right or wrong — we've certainly found that they are really hard to build and very hard to maintain. And it's not obvious. Should pages be long? Should pages be short? How do you maintain freshness? How do you maintain consistency? It introduces a lot of new and complex problems that are maybe partially addressable by the prompts and changing the prompts, but I think many of them are actually systems engineering problems, where you need to think about concurrency control and whatnot. They are not easy problems to solve. So we'll see what we come up with at Chroma, but that's some context on why we're interested in the topic.

Harrison: Awesome. Great intros from everyone. Real quick before we dive in: there is a Q&A section, so please put any questions you want answered there. We'll probably spend the last fifteen minutes or so on audience questions. And if you like a question, upvote it — I'll sort through them by upvotes, so use that to make sure we're answering what's top of mind.

What is an LLM wiki?

Harrison: Maybe to get started, it would be helpful to try to define what a wiki even is. I'll take an initial stab, and then you can each tell me where I'm wrong or what I'm missing. When I think of a wiki, naively, I think of a set of files in some directory structure that is usually generated by an agent and usually for an agent. I'll stop there with that brief description. Is there stuff I'm missing? How do you all think about wikis?

Jeff: I mean, the definition of a wiki is overfit to the technology and overfit to the use case in some ways. How does Wikipedia actually store data? Is it in a file system? No. Is it for agents? Also no. Is it a wiki? Yes.

So I'd try a more abstract definition of a wiki, which is roughly: some level of hyperlinked pages. It's in some ways an internet. You have a bunch of pages, they're linked together, and they're intended to be informational. They're intended to be clear. The layout of the page is intended for information consumption — there aren't a lot of distracting advertisements everywhere. I'll stop there. That's maybe some thoughts on what wikis are.

Brace: Yeah, I kind of agree with Jeff. It's a collection of files — in today's day and age, usually markdown files — that document something, whether that's actual docs, or notes, or memories. At its core it's a large collection of files that are linked together somehow, through page linking or an index, and that contain text and documentation that agents or humans can consume.

Devin: Yeah, I would agree. Generally I agree with Jeff's definition — pages that are linked together. But specifically for the LLM wiki, I think Andrej Karpathy's tweet did a good job of framing it. I really think of it as compiled knowledge: you're taking sources and building an index. I very much agree with Jeff's framing, where each page is acting — I'm biased here — like a cache. It's compiling knowledge for future reference so you don't have to re-explore that information.

Wikis as caches of compiled knowledge

Harrison: Yeah, I thought that was a really interesting analogy when we were having lunch, Devin. Could you expand on the cache framing?

Devin: Yeah. This gets meta into the purpose of a wiki, or the wiki abstraction.

The thing I don't love about wikis today, in terms of how the implementations look, is that if you tell an agent to generate a wiki, it generates something that looks like a wiki for a human — because it's trained on human wiki data, and because that's all that has existed. What that usually looks like is that it has some level of ordering. You often see a getting-started page, an architecture page; there's a common structure that LLMs or agents go to when you tell them to generate a wiki. That's inherently, in my mind, human-oriented, because there's an expectation that someone's reading it front to back.

Separately, there's the question of why you have a wiki at all. If you think about it from first principles for agents, there are two types of knowledge that live in a wiki. One is what I call necessary knowledge, which is information that doesn't exist anywhere else — the agent has no way to access it unless it's documented in the wiki. The other type is more compiled knowledge that you're generating from sources, and that's functional. It's there because it's going to make the agent more efficient, so it doesn't have to re-explore that information to get to the same conclusion. Sort of like caching knowledge, which I think is an interesting analogy.

And if you take that analogy further: like a cache, you don't want your entire database to be a cache, because that'll be very expensive — you have to keep it up to date with the source of truth. So you only want to cache what's relevant and is going to be accessed frequently, or is expensive to recompute. Generally I think the wiki concept makes sense, but the default implementation comes at it from a human perspective.

Like a cache, you don't want your entire database to be a cache, because that'll be very expensive. You only want to cache what's relevant and is going to be accessed frequently, or is expensive to recompute.

For agents, for humans, or both?

Harrison: It sounds like you very much think of these LLM wikis as being for agents. Jeff and Brace, do you agree with that? Yes, wikis can be anything — but when people talk about LLM wikis, or when you think about what you want to build in the wiki-adjacent space, do you mostly think of building it for agents or for humans? And is that the same thing?

Brace: I think historically this term has been used to describe documentation that both humans and agents consume. If you think about DeepWiki from Cognition, which I think was the first wiki to get popular, that was a website humans could go to and look through, but they also had an MCP server for agents to consume. That's not a bad way to think about it.

But going forward, if we think about where agentic development is going, most things are moving into an agent. We humans are doing less and less of the nitty-gritty, which means we're reading fewer docs and instead asking our agents to communicate with docs and surface that information to us. So I think going forward, these wikis should be primarily designed for pure agent interfacing, and then you use your agent to surface that information up to you.

Jeff: If you're talking to a venture capitalist, obviously the answer is all for agents, exclusively. In practice, I don't think that's the case.

Take a thought experiment. Let's say it was optimal for your agent to store this entire wiki not in natural language or English at all, but only in latent vectors that are completely illegible to humans. In a way, that would be cool that it exists. However, it breaks our ability as humans to have any level of control over the wiki, and any level of legibility and shared workspace. So it's actually really important for these wikis to be in English and not in latent vectors. What that motivates, or tells you, is that the wiki is for both. It's for the agents, but it's also for humans as well.

Humans in the loop

Harrison: Maybe dive deeper into that. As you're all building products, how do you think about the human in the loop, the human steerability? What is the right way for humans to interact with this wiki creation, curation, whatever you want to call it, process? What does that human-in-the-loop process look like for you?

Devin: I have two thoughts. First, I was going to say something very similar to Jeff. You could take it to the extreme where agents have their own wiki language that we don't understand, purely in latent space. But I think there's a difference between being human-interpretable, which is actually very, very important, and the audience being a human — is it the same person who's going to be reading that page? So I do think we want to optimize for agents as the audience, but keep it human-interpretable.

Harrison: And the second — how humans are using those loops. Human in the loop for the docs.

Devin: Okay. So as part of the product, we keep people in the loop because a lot of it is tied to the PR review process. We do knowledge reviews, similar to a code review, so humans are observing how knowledge is changing — but they don't necessarily have to approve every change. What we've heard from our customers is that they're already overwhelmed with code review; they don't want knowledge review to be a burden as well.

The other side of it is that I like to say knowledge is eventually consistent. Meaning that if it's incorrect, as long as you build in self-correcting mechanisms, it will ultimately be fixed in the future. So if you write an entry that's incorrect, and you provide mechanisms for the agent that reads it and finds out it's incorrect, it can just correct it. You lose some token efficiency in the meantime, but eventually it will be corrected if it's important.

Brace: Yeah, I kind of agree with Devin's last point about agents correcting themselves. The way we set up a wiki, if it's in code mode, is that the wiki lives in markdown files in your codebase. So there's a human in the loop because it puts up PRs, and you need to review those before they get merged. But for the general-purpose memory that lives on your computer, it just writes whatever it wants without a review step, because that would be very tedious.

So what we did in OpenWiki is give the agent a way to add questions. It can add open questions it doesn't know the answer to, and then you can go in and answer them. Or, in later runs, if it thinks it found the answer, it can go and mark that as answered and reference the part of the wiki that answered the question. As a human you can also go in there and add questions, or answer them yourself.

And the last thing, which we haven't done in the agent yet but which I want to do today or tomorrow, is a good changelog. Because the wiki is modifying itself without you approving all the changes, you may not want to approve every individual change, but you probably want some visibility into what changed. If it's all markdown files you can look at a diff, but that's going to get very time-consuming. So instead I'm thinking a very concise changelog: "Hey, I edited this file to document this new product you added," or whatever it might be. And then you can retroactively go in and say, "Actually, this part of the changelog doesn't sound right. Can you dig deeper into that part of our new product?" or "Can you correct it to do X, Y, Z?" That way you don't need to read the individual changes, but you have a high-level overview of what was changed, and then you can go chat with your agent to have it correct things.

Jeff: I was going to say, we're kind of in the prompt engineering era of this stuff — we're not even in harness engineering or loop engineering yet. Eventually it will be loop engineering, not to be buzzword-compliant, but just because that's what it is. Right now, I think the human touch in the prompts themselves is still quite important. The exact signals to climb on to improve these things are also challenging to figure out automatically, so human judgment is quite helpful.

The nice part about a wiki relative to raw source search is that you can also correct it. If something's wrong — if it has a date wrong — you can say, "you have the date wrong." Maybe it hallucinated it, and people are figuring out how to fix hallucination, but maybe the source material was actually wrong, and you're fixing the source material. Updating a Slack message from three months ago doesn't really make any sense. So this controllability and steerability aspect of a wiki is really attractive.

But the bitter lesson story — again, not to throw out buzzwords — is that eventually we should to some degree learn to trust the agents to know when to come to us and say, "hey, human, I need help with this. This is ambiguous, or this is vague, or I don't think I have the expertise here to come to a determination. Can you please help?" I think that idea of agents interviewing humans is super interesting. We're just really early in all of this, so maybe it's wrong to call it majorly unexplored — but by definition, majorly unexplored.

The hardest parts: updates, scale, and maintenance

Harrison: What is the hardest part of building wikis? Or is it all —

Brace: It's the —

Harrison: — just so easy. There's nothing hard. That can't be right.

Brace: I think prompting for updates is the most difficult part right now. It can write a first pass very well. But as that expands — if you get 300 updates into your wiki, and your wiki is a billion tokens long — getting it to identify where in the wiki to update when it gets new information, and then concisely update or insert or remove information, is probably the hardest part. That's what we've seen in OpenWiki, and also in other agents we build, like in Fleet, where it can update itself. Oftentimes it wants to just insert new lines into the prompt instead of modifying what it already wrote. Having that balance between modifying, removing, and inserting new information is a tricky thing to get right with agents right now, especially as they deal with more and more context.

Devin: Yeah, I'd say there are maybe three things that are tricky.

I actually do think generation is hard — not on small repositories or projects, but at scale. Some of our customers have tens of millions of lines of code and large organizations. When you have something at that scale, figuring out what is important is actually a hard problem, because if you just let an agent figure it out by itself, it's going to focus on the wrong things. So identifying what is actually important is the first challenge, at least at scale. On smaller projects, agents do quite fine.

The second is maintenance. I think the maintenance problem is really, really hard. Like Brace is saying, prompting is one part of it, but then there's also doing it at scale and efficiently. If your wiki becomes massive and you're spending a ton of tokens to maintain it, are you actually getting ROI? Does it make sense to be maintaining all of that knowledge, even if you're doing it effectively — which is a challenge in and of itself?

And the last, related piece is observability, and understanding whether you're actually getting ROI. The attractive part of wikis is that everyone intuitively gets the value. But actually measuring impact right now is not easy.

Jeff: Agree with what Brace and Devin said. My meta framing is that the hardest part right now is the harness engineering, where the goal is accuracy but also cost management. You probably don't want to throw Fable at all of your ingestion and all of your updates — you're going to cry. So can you get away with something cheaper? It depends on your prompt and on everything else, and how all this stuff mixes together. It's a multivariate optimization problem, which I think is pretty challenging to solve in a good way.

Brace: I want to plus-one what Devin said at the end about measuring ROI on wikis. Measuring ROI on many agent use cases is really difficult, and on wikis especially, because your agent can read from your wiki, but did what it read actually improve its quality? This is something we're still thinking about and don't have great answers for yet, but I think it's a very important thing to do — especially if you want to sell this to people. You need to tell them, hey, it's actually going to make your life, or your agent, better. And it's very tricky to do well.

Measuring ROI: token efficiency and consistency

Harrison: I was going to ask exactly that. How do you think about measuring the impact of wikis?

Jeff: Are your agents better as a result of it existing?

Harrison: How do you measure that?

Jeff: You're the evals guy, you taught me a bit, Harrison.

Brace: The ideas we tossed around for OpenWiki on codebases were to run a coding agent benchmark over a task without the docs and then rerun it with the docs. But if it's something like SWE-bench, those benchmarks are very saturated — agents will perform well with or without better documentation. So what we're going to look at is efficiency: is it able to get to the same right answer, but quicker, with fewer tool calls and fewer tokens?

We tried off-the-shelf benchmarks, and those benchmarks are already in these agents' — or these models' — training data, so it doesn't really make much of a difference. They just kind of know what to do. So I think this requires designing in-house evals. But the way we were thinking about it is: is it able to get to the right answer with fewer tool calls and fewer tokens consumed? Because in theory it shouldn't have to search as much if the wiki contains all the relevant information. It should be able to find that concise answer and then apply it to the solution.

Is it able to get to the right answer with fewer tool calls and fewer tokens consumed? In theory it shouldn't have to search as much if the wiki contains all the relevant information.

Devin: Yep, I agree with that. We've done some internal benchmarking, and for customers that have evals, we work with them on that. But token efficiency is really the goal. From internal evals, what we'd call a cache hit — where you're working on a topic that has a wiki entry — runs can cost about half as much, because the agent is doing less context gathering. That's how you're going to see the token efficiency.

The other interesting learning, which also makes sense intuitively, is that the outputs are more consistent. If you run the same agent on the same task ten times, you do get variations in output, even if it gets to the right answer. But if you have the same starting point, which is the wiki, the trajectories downstream are also more reliable and consistent.

From internal evals, what we'd call a cache hit — where you're working on a topic that has a wiki entry — runs can cost about half as much, because the agent is doing less context gathering.

What wikis unlock: long-horizon tasks and proactive agents

Harrison: Maybe two more.

Jeff: I want to share the thought that there are two ways to think about it. One is plugging this wiki into your existing workflows, in which case hopefully you'll see cheaper and faster, because you're doing less exploration, and also better — fewer human interventions. That's the frame for existing agent workflows.

And then the second question, which is not vibe posting but is actually genuinely open to me, is: what new kinds of things does having this prebuilt thing enable? All new things are evaluated on vibes to start. So that basket of work that people aren't already doing today, but that having the wiki allows them to do, will probably first be evaluated on vibes, and then later we'll have stricter evals.

Harrison: Any good examples of that latter category — things that are net new that you can do with wikis?

Jeff: My theory is that wikis are in a way agent memory, and very good memory would hopefully give agents the ability to operate on longer-horizon tasks. Obviously it depends on your context, but for an organization, any task you can think of as long-horizon, you could basically deploy an agent into where you couldn't have before.

Brace: I haven't thought about this too much, but one thing worth thinking about is proactive agent actions. With most memory systems that are, like, ChatGPT or Claude, they remember what you tell them, but they don't remember things proactively, because they don't connect to external sources and can't ambiently update themselves. But if your agent can connect to data sources like Slack or your email or Twitter, whatever it might be, then it can proactively remember things — say, from your Twitter feed, something you haven't seen yet about research — and then it can suggest or take actions on your behalf without you even needing to prompt it.

That's very meta, and I want to caveat that we haven't truly tried it out internally yet. But that's what I'm thinking with connecting agents to other sources, so it can go out on its own and find things it thinks you might be interested in or want to remember. And then if it's an engineering agent, maybe it can prototype new features or tasks for you, or draft new content, or present ideas to you proactively instead of reactively.

Devin: The only other idea I'll add, which I think is really interesting, is agent collaboration. If you have agents across teams and products at a company, having a wiki as a central repository that they're all contributing to and reading from and writing to opens up — it's probably long-horizon tasks ultimately, but I think that collaboration layer is really, really interesting.

Are wikis the wrong abstraction?

Harrison: I want to go into questions soon, so a reminder to go look at the Q&A section and upvote the ones you want answered, or add your own. But before we do: Devin, you gave me the hot take that you don't think wikis are the right abstraction, and yet here you are on a webinar about wikis. What's the nuance?

Devin: I sort of alluded to this before. The way we talk about wikis today is very human-oriented. But if we reapproach the problem from an agent-first perspective, you re-examine both the content and the structure from a token efficiency standpoint, and I do think the shape is different. So wikis are generally directionally right, but I really don't think where we're going to be if we fast-forward a year is the same thing. I think we're going to think of it more as a cache, or an index, as Jeff referred to before.

Harrison: Can you give me a concrete example of what that means in terms of the shape of a wiki? What shouldn't it be? What should it be?

Devin: I think the language is going to be a lot more terse — we see this a lot. You don't need prose in the same way for agents as you do for humans. It should be highly referential, even more so than existing wiki articles, because agents are really good at that: okay, here's the information, and if it needs to do additional exploration, here's where it can go. That's more agent-friendly than human-friendly — a human doesn't want to keep digging, but agents are very good at it.

And then the other piece is eviction. For wikis for people, you kind of want to keep things around forever, because it could take you a couple of hours or days to get to the same conclusion. But for agents, I think we'll probably see something where if a wiki page isn't being accessed, you just evict it — and being more strict about what information even makes sense to write into the wiki in the first place, because maybe it's really easy and cheap to recompute.

Harrison: Jeff, I think you described the wiki as a set of hyperlinked pages at the start, so I'm assuming you agree with Devin's comments around linking. Anything else to add in terms of the best way to hyperlink these pages together?

Jeff: There's a bunch of variables around how you give the agent the ability to do the progressive disclosure thing. How long should pages be? How short should pages be? Should you be able to dump just the headings and use that as progressive disclosure? With linking — how aggressively should you do active linking, versus passive linking, where you'd use a search term to go search the corpus and find the other page? How do you clean up backlinks as they grow stale or get removed? There are a lot of these maintenance-agent problems here. And I don't think I have the answers to any of them, and the answers probably vary depending on your use case. That's all I have to say.

Q&A: wikis beyond engineering

Harrison: Cool. Alright, going into some of the Q&A. First: for organizations that are not full of programmers and engineers, but rather knowledge workers, do you have a meaningful metric for measuring value there? And I'd maybe add a more basic question on top: do you see wikis being used for knowledge work, or is it mostly for coding at this point in time?

Brace: I think yes to knowledge work. If you can create a general-purpose wiki from all your company sources, then — assuming the non-programmers at your org are already using agents for some tasks, or at least ChatGPT for asking questions — if you can hook this wiki up to that chat interface they're using, hopefully you can make them a bit more productive, or answer their questions quicker than them needing to go to humans.

For measuring ROI on this, I think it's just as tricky as measuring it for anything else. Are they able to resolve more questions with their agent when it queries the wiki versus going to humans? I don't know. It's still kind of an unknown to me, besides: can they get answers quicker and faster than without the wiki?

Jeff: I'll take the extreme opinion here, which is that I wouldn't even necessarily use the term wiki. Fundamentally, the wiki is the state of the organization. It's the state layer of the organization, or could be. I think that tool itself will be both the communication, coordination, and execution layer for every company on Earth. So it certainly impacts knowledge workers, but everybody.

Fundamentally, the wiki is the state of the organization. I think that tool itself will be both the communication, coordination, and execution layer for every company on Earth.

Devin: Yeah, and I'll say that we have examples of customers we work with who — I mean, Dosu is very focused on the product engineering knowledge side of the organization, but we have folks who are building this on the GTM side or the customer success side, where having a wiki entry for every customer just makes a lot of sense, because it's a very common query. That said, going back to the challenges of wikis: knowledge-work wikis across an organization are very complex from an RBAC perspective. Who is able to see what?

Q&A: wikis vs knowledge graphs

Harrison: Next: how is a wiki different from creating a knowledge graph, or a context graph? And maybe another variant of this — why a wiki instead of a knowledge graph?

Brace: I mean, is a wiki a type of knowledge graph? If you think about Karpathy's wiki concept, it's a bunch of files that link to each other. Or Google's — what is it — the Open Knowledge Format. One key part of that is linking pages back and forth to each other, and that's kind of graph-like. So it's a different way of storing the nodes of your graph. If they link back to each other in this web of links, it kind of is graph-like.

Devin: Yeah, I mean, I guess —

Jeff: Oh, go ahead.

Devin: Go for it.

Jeff: I was going to say that the RDF triple store, Neo4j-style rigid graph database thingy is just too opinionated. You want the agent to have more flexibility. The wiki gives you linking, and that's what you want. You want linking. You don't actually want noun-verb-noun.

Devin: Yep, I was going to say the same thing. Graph databases are more like a Cypher query language — much more structured. I also think wikis lend themselves well to the semantic space, which is how you usually surface information: you're in a similar semantic space, versus a graph-like query.

Harrison: I was going to ask, Jeff, since we've got you on — wikis and vector databases. Do they go well together, or are they separate? How do you think about them?

Jeff: I don't know what a vector database is. No one's ever given me a good definition of a vector database. As far as I can tell, it's just a term that Andreessen Horowitz made up. Well, it turns out every database is a vector database, because every database has vector search now.

Again, the way we think about the problem is: how do you store massive amounts of unstructured data, and how do you make it very searchable? That's the problem we want to solve, and you should assume we will solve all of the problems related to that as well. So that's why we're interested in this.

Q&A: knowing when a wiki is wrong or stale

Harrison: Next question: can you elaborate on ways to know when a wiki is either wrong or out of date? There's a follow-up too — do we need a validation proxy? Say the agent will interview the human. What would the triggers for that be?

Brace: I think this is tricky. Knowing when it's wrong is an evals problem, or hopefully it's never wrong, because you can correct that before it gets to that point.

For when it's out of date, the first thing that jumps to mind is the OpenWiki concept of the open questions file. It can have open questions, and maybe it can add notes. If it adds some context which it thinks could become out of date, it can update this open questions — or "loop back" — file: hey, revisit this on this date, because it may become stale.

That's if we're assuming humans don't look at this data and can't or won't manually modify it — because if humans do that, then a human can audit it. But if it's just for an agent, you try to prompt into it the idea of, when something might become out of date, write that down, and then have it revisit that "context might be out of date" file at some point in the future to review its own context.

Devin: I'll just say what I said before: I think knowledge is eventually consistent, as long as you build the loops necessary to self-correct.

Harrison: What do the loops look like?

Devin: The simple loop is: the agent reads it, it finds out it's incorrect, and it has a tool to correct it. That'd be the most basic loop.

Knowledge is eventually consistent, as long as you build the loops necessary to self-correct.

Q&A: how agents should access the wiki

Harrison: What's the best way to expose an agent to the wiki? File system operations and bash, or search tools?

Brace: I think it's probably a mix of both. What we've found with agentic search — where it just uses files and tools, not specific to wikis but for trying to have agents find specific context in large directories like codebases — is that they do really well with files and tools. That's not always the optimal path, though. You might want to have a mix of agentic search plus full-text search for finding keywords, or maybe a bit of semantic search. I don't think there's a one-size-fits-all solution here. You probably do want a mix of retrieval modes. But whether you let the agent pick that retrieval mode, or you just throw all of them at it whenever the agent asks a question — that's kind of an unknown to me.

Jeff: I can chime in on this. I think there's been a lot of misinformation about search, primarily promulgated by people who would have you token-max. They're happy to have you rediscover 85 files on every single query.

The bitter lesson applied to search is that you should give the agent all the tools and then trust the agent to use those tools effectively to discover the right information as fast as possible. So I'd echo what Brace said: give it grep capability — finding patterns in data with regex is super helpful. But also give it sparse vectors, BM25. Also give it dense vector semantic search, and let the agent decide the right blend of tools and the right ordering of those tools to get to the right information as fast as possible, and hopefully find it at all. That's one of the key problems: not just finding it fast and cheap, but also finding it, period.

The last thing I'll say is that I don't think file systems are or will be the Lindy, data-at-rest representation of this data. File systems are very good at some things — specifically extremely fast file access, which is necessary for things like linting, or very code-oriented operations. But they're also very bad at a lot of things. They're bad, or slow, at version control. They're bad at access control. They're bad at discovery. We're just reinventing everything all over again. There's a reason databases were invented; we're not flat-file maximalists from the fifties. So I think we have to discover that again, but that will be rediscovered.

Harrison: But there's a difference between storage of the files and how they're exposed to the LLM, right?

Jeff: Of course. But there's also, I think, an ideal way to store the data such that it can be exposed to the LLM.

Harrison: Which I guess depends on the ideal way to expose it to the LLM — which gets back to how agents should access it. Do you want to give them read/write? Do you want to give them search? What is the right way? If you need to give it semantic search, you're probably not going to store it in a file system, as an obvious example in one direction.

Jeff: Yeah.

Closing

Harrison: Cool. I think we're just about at time, so thank you, everyone, for joining. There are a bunch of questions in here as well, and I'll try to share some of these with Devin and Jeff. You should follow them all on Twitter — they think a lot about this space and have some good spicy takes.

The last shout-out I'll give is that if you like conversations like this, we're bringing Interrupt, our user conference, to London and New York in the fall, and we'll be doing a bunch of fireside chats and talks about all things agents. So thank you again, Devin, Jeff, and Brace for joining, and thank you everyone for tuning in. See you next time.

Jeff: Thank you.

Devin: Thank you.