Everything Gets Rebuilt: The New AI Agent Stack | Harrison Chase, LangChain

0:00I think two things basically happened. Like the models got better, but then also we started to discover these like primitives of a harness that would really let the models do their best work. And we saw an explosion of people building agents. Do you think that the models end up eating the framework layer or do you think the framework and infralayer eats the models? I think the harness is the most important thing. The cloud models are great, but the harness is really what made that work. Hi, I'm Matt Turk. Welcome to the Matt Podcast. Today, my guest is Harrison Chase, co-founder and CEO of LangChain.

0:30Harrison has been one of the key figures in the rise of AI infrastructure and agents from LangChain's early days as an open source framework to the broader evolution of LangGraph, Deep Agents, LangSmith and AgentBuilder. This episode is a deep dive into the frontier of the AI stack. As AI moves from simple prompts to agents that can plan, use tools, write code, and manage memory, the big question is what new infrastructure is required. We talk about agent runtimes, harnesses, observability, and where the future of AI infra is heading. Please enjoy this great conversation with Harrison Chase. Hey, Harrison, good to see you.

1:07Thank you for having me. Excited to be here. So for anybody watching this on YouTube or Spotify video and who's a regular watcher of the Mad Podcast, you'll notice that we are in a different venue today. We're not in the usual studio. We are in an epic venue at the Chase Center in San Francisco. We're recording this as part of the Daytona Compute Conference today. So I thought a good place to start would be to frame the evolution of agents over the last few years. This seems that there was a huge moment. I think sometime around the holidays, December and January, when everyone kind of realized at the same time how far agents had come in just a few months.

1:52So help us maybe compare and contrast the first generation of agents compared to what we have today. Yeah. So I think a lot of the ideas behind the agents today were actually present in some of the early day stuff. The difference was the models just didn't work back then. So LinkedIn came out maybe half a month or a full month before ChatGPT. And one of the main things we added at the start was this idea of running an LLM in a loop and calling tools. And there's this great paper called React, which basically said to do exactly that. and you know it worked for the data set that they ran it on which was like wikipedia question answers but it didn't work in the real world and then in march i think uh auto gpt came out and that was the same thing it ran in a loop called tools gave it a bunch of stuff it really was like a precursor to open claw in a lot of ways and then the way that i would describe the trajectory of agents since then is basically there was this core really simple idea just run the llm in a loop have it call tools, you know, give it a prompt, give it some instructions, give it a bunch of different tools.

2:53But that didn't work really well. So people ended up building scaffolding around the models to make them do things in a more predictable and reliable way. And that's why we at Langchain, we built LangGraph, which is another framework really aimed at that kind of like graph-like workflows and giving more structure. And when you really want like super high reliability, you want to use something like that. But I think sometime in maybe like November, December with some of the newest Claude models, the models just got really good. And you kind of discovered that they could actually just run in a loop.

3:24And a lot of this, this wasn't just the models. It was also the harness around the models. So what I mean by that is if you look at things that came out about a year ago, Claude code, Manus, deep research, they all had the same thing of running the model in a loop, having it call tools, it could write some code, it could read and write files. And so I think two things basically happened. The models got better, but then also we started to discover these primitives of a harness that would really let the models do their best work. And I think over break, people basically realized that. And we saw an explosion of people building agents for different things using these same core primitives.

3:57What kind of agents are we talking about? Are we talking about coding agents? I think you said somewhere that every agent should be a coding agent. So we see a divergence between like two different types of agents out there. One of them is like conversational agents. So these would be like the customer support, customer experience, chatbots. These require really low latency. Voice is oftentimes the medium that they interact with. And that's one style of agents that are mostly like conversational. They don't do a ton of tool calling. They'll maybe do like one or two because they can't do too many or will take too long.

4:26But then we see this other style of agents, which Sequoia came up with this name, Long Horizon Agents. And I really like that. They can operate over long horizons. They can do some planning. They can maintain coherence. And yes, a lot of them end up looking like coding agents. And I think there's probably like, there's a few reasons for that. But one, code is really useful. You can use code to do a bunch of different things. You can use it to parse text files. You can use it to do things programmatically. Like you want to loop over 100 different files rather than doing 100 different tool calls.

4:54You can write a script that does that. So code is like really generally useful. But then also the models are trained on code. And so all the big model labs have been RLA and code and bash and editing files into those models. And so that is the stuff that works the best. So I think we see the split of agents, long horizon versus chat. And then, yeah, for the long horizon, it's basically turned out that coding agents or things that look like coding agents are the stuff that works well. And do you think conversational agents become coding agents as well as they go deeper into the stack? This is a really good question.

5:27I mean, we talk a bunch about this internally because we're debating whether we should build like a different type of agent harness for these types of agents. I think there will kind of be a convergence when there are agents that can reliably like kick off and manage other long horizon agents. So one of the things that we're seeing in coding is that people want this experience of being able to kick off a bunch of other, like do a bunch of work, kick off a bunch of agents, but keep on chatting with like the main agent. And that's very similar to like a conversational agent in some sense, right?

5:58Like you've got that like constant kind of like back and forth, latency, TPD. But then, you know, these voice agents, I think, will obviously want to do more and more like long running things in the future. And I think the way that you do that is you basically have two agents, one that runs in the background and is kicked off by this other kind of like conversational agent. So it could all kind of like converge into this single harness that just supports basically long-running async background agents as a tool. So you mentioned a minute ago that part of what triggered the acceleration agents is the models getting better, which makes me wonder who wins eventually.

6:38Do you think that the models end up eating the framework layer, or do you think the framework and infralayer eats the models and ultimately the models are commoditized underneath? I think the harness is the most important thing. I don't know what will happen, but I think if you like, I think Manus is a great example. Like Manus was an end user product, but their harness was so good. Like that was the secret sauce of what made it work. And it worked with any of the models under the hood. And when you look at Cloud Code, like, yes, like the Cloud models are great, but the harness is really what made that work.

7:09And Cloud Code isn't just a harness, though. It's also that UI. So I actually think, so one, I think there is a pretty tight coupling or there's not that much difference between a harness and a UI on top of it right now, at least. And it's still very early. Um, but you look at like codex, it's a coding app, but they also have their own harness, cloud code, Manus, um, a lot of the deep research stuff out there. It's this interesting combination of harness and UI. And so I think that's, I think the harness is really, really important. And then, yeah, I think one of the interesting things is that a lot of the people building the harnesses also build the model.

7:42And so this is, this is one thing that like interests me and confused me because I think a very logical argument to make is like, great. Okay. We make the harness, we make the model, let's RL the model to be really good at that particular harness. you look at some of the tools that Cloud Code uses, it doesn't actually use the tools that are RLed into the model. So like Anthropic models have some like file editing tools. They have a completely different set of tools in the actual harness. So I don't know really what's going on there. I've asked them a few times that I haven't gotten a story response.

8:09But so I don't know what happens, but I do know the harness is really, really important. Like I think this is the thing that matters. And then do you come at it from an end user application? Do you come at it from a model? I don't know. Great. And to make this broadly accessible and interesting for a large group of people, what's a harness in plain English? It's how the model kind of interacts with its environment is what I would say. So it's the set of tools that it has. And some of these tools can be really specific. And I actually wouldn't count those as part of the harness. But some of these tools can interact with a more general environment.

8:42So if we think about coding agents, I would say the file editing tools it has are part of the harness. I would say the ability to run code is part of the harness. If you take a harness and give it a particular tool for interacting with Slack, I would argue that's you kind of like customizing and building on top of the harness. And that's how we think most agents should be built. Like we think most agents should be built by taking a harness and giving it some instructions and giving it some kind of like tools. And those tools could be specific tools like a Slack tool, or they could be kind of like configurations of tools that are built into the harness.

9:16So what I mean by that is most harnesses today have sub agents built in. They have skills built in. And so you could configure them with particular skills. But the fact that those like skill abstractions and sub agent abstractions exist, like I would argue that's part of the harness. Other things that the harness does is like take advantage of like prompt caching. It does context compression. So when you're getting up to a certain length, it will compress it back. And so these are things that are like pretty general purpose, like all of these apply across all different types of applications. And so these are things that are general purpose as an application developer, you shouldn't really have to worry about, but you can basically configure them with different prompts, different tools, different skills, different sub-agents, and make them yours and make them your own agent that you then expose to your end users.

10:00Great, thank you. All of this is fascinating, and I'd love now to turn various pieces of what you just described and double-click to go into some depth. So let's start with system prompt, which I think is part of the key architecture. So detailed system prompt, what does that do? Yeah, that drives the agent. It kind of like tells it what to do. The way that I think about it sometimes is if you have a standard operating procedure for how a human should do things, like that should influence a lot of what the system prompt is. And so this is loaded up as soon as you start the agent. It's basically loaded up and it tells the agent what to do and it drives it.

10:37And where does it live? It depends how you create the agent. So in, so, okay. So yeah, if we look at like coding agents like Cloud Code or something like that, there's a system prompt that's built into the harness, and that tells it how to interact with the generic tools. But then a lot of that prompt is basically augmented by things that you as a user of Cloud Code provide. So you provide a clod.md file, and that's inserted into the overall system prompt. You provide skills and sub-agents, and those are inserted. And so I think in practice, what we see is that this system prompt generally is an amalgamation of a few different things.

11:09Some of them are like built into the harness and some of them are built in by whoever's customizing the harness or choosing what to expose to the harness. You mentioned tools. I think there's a concept of a planning tool as well. What does that part do? Yeah, so there's a few different types of tools. Some tools are basically tools that are built into the harness. So we and a bunch of other harnesses out there have a planning tool that basically creates a plan. And it could actually like write it to file and then let you kind of like edit it over time. It could do nothing. It could just let the agent call the tool.

11:43And the reason that's valuable is that then that puts that into the context window of the agent. So it's kind of like giving it a mental scratch pad for it to kind of like think about. So there's different levels of what that planning tool can do. Other tools like... And it's literally after you do this, you do that. and this is how you operate? So most planning tools are a list of tasks to do. Each task has kind of like a description, a status. Those are like the important things. And then you can track the status can be like done, like working on it now or like to do in the future, basically.

12:18It can, of course, be whatever you want, but that's the most common type of thing that we see. and then most harnesses don't actually like enforce that you do that plan. It just kind of puts it in there and it lets it track it but there's nothing that like splits it up and says okay you created this plan now let's take the first thing and go do that and then after we're done let's go to the second thing. That used to be the case earlier when these LLMs weren't as good that used to be the case. You'd have an explicit planning step and you'd come up with a plan and then you'd go and you'd go to another one and you'd go to another agent and that would do the first thing and then you come back.

12:50But there's all sorts of ed cases. Like what if the plan adjusts halfway through? Okay, now I have to add a step where I check, should I adjust the plan? And it just has become too kind of like convoluted. And so now what most things do is they just have that plan in the text file and the main agent can like use that to help guide its actions. But there's nothing that says I'm explicitly doing this step or I'm explicitly doing another step. Great. What about sub-agents? Sub-agents are great because they let you basically isolate context. So this main agent's like running in a loop and it's accumulating context over time as it calls tools and interacts with things.

13:23And that's great because it has all this context, but that's also bad because it has all this context and that blows up the context window. And so subagents are great because what you do is the main agent basically gives it a task, gives it a string, and the subagent spins up with a completely fresh kind of like context window. So it starts from scratch and then it does a bunch of work and then it responds and the main agent just sees the response. So you get this nice isolation between different tasks. The downside is that you have isolation between different tasks. So why is that a downside?

13:50Because then you need to communicate between the two agents. And so if the communication between the agents is bad, then it won't work. So a very real thing that we see happening sometimes is the main agent will spin up a subagent. The subagent will do a bunch of work and the key stuff will be halfway through its trajectory. And then its final message will be done. And the main agent is like, what do you mean done? I can't see anything else. And so that's an example where the subagent doesn't have good enough instructions. It hasn't been communicated well enough to the subagent that it needs to communicate its final answer back in its final message.

14:22And so communication is the hardest part of life, by the way. It's the hardest part of startups, hardest part of relationships, hardest part of working with agents is getting them to communicate. And so subagents are great, but they do add that extra layer of communication. And how does the system know to create a subagent? It's all in the prompt. It's all in the prompt. Yeah. That's the beauty of these types of agent harnesses. You know, like earlier when we were doing things with LandGraph, people would be like, okay how do I add like a step to make sure that the agent does this before x or how do I how do I you know enforce that the the for better or worse and this is why this is why line graph still is a place I'll get to that later but like for better or worse the way that you get these things to do anything is you just tell them to do it and and that's great because it's like flexible but that is also not like 100 % reliable and so we actually see still pretty good adoption and pickup of Langrath in heavily regulated industries where you want like a ton of control and precision and reliability because as good as these kind of like coding agents are they are pretty unpredictable in terms of what they do and and there's no guarantees on anything it's why they're so enticing because you just tell them to do things and they do things but there's no guarantee and so that's a downside as well another part is the file system as you mentioned why do agents need a file system?

15:36My mental model for this is it all comes back to like context engineering, like what the agent sees, what the LLM sees in particular. And the way that I think about a file system is it basically lets the LLM manage its own context window. So it can decide what to read from files. So rather than you could imagine in an alternate world where you put everything that is in a file, you just dump that into the context window. That would blow it up, right? And so if you let it read files, great, that lets it choose what to pull in. When you let it write to files that's basically saving it so that if you do compress the context over time, you can return to it and you can read it in the future.

16:13We use file systems to offload large tool call results. And when I say we use, we have an agent harness called DeepAgents. When I talk about our planning and our file system stuff, this is all stuff that we do in DeepAgents. Most other harnesses do similar things, but the one I'm talking about in particular is DeepAgents. So what we do is if you call a tool and it comes back with like 60 ,000 tokens, we don't show that all to the LLM because that's a ton of tokens. Rather, we actually put that in a file and then say, hey, here are the first like thousand tokens. If you want to read the rest, go read this file.

16:42We use it for summarization as well. So when you get to a certain context window length and it's about to overflow, what we'll do is we'll run a summarization step, but we'll actually dump all the original messages into the file system. So if it wants to go look things back up, it can. And so we use it in a variety of ways. I would say the overarching theme is it actually like lets the LLM manage its own context. And I think the general theme of like these more and more autonomous agents is that they let the LLM do more and more and managing its own context is kind of like, that's an increased version of letting it call tools or something like that.

17:18And the file system is literally a file system. It's not a database where it can be different things. Great question. Great question. It can be anything. The important part is that it's exposed to the LLM as a file system because LLMs are great with working with file systems. And so one of the cool things that we have in Deep Agents that is pretty differentiated is this file system. It could be the real file system on disk or in your Daytona sandbox or anything like that. It could also be a database that just has like a thin layer on top of it that exposes it as a file system. Not everything needs to be a file system.

17:51If you have a SQL table, like let it write SQL. That's pretty easy for it to do as well. But when you're working with like large amounts of text, even if those are stored as like a row in a SQL database, it's often nice to give it kind of like the interface of a file because that's how all of them know how to interact with it. So yeah, it could be anything under the hood. Database, S3, real file system. So detail system prompt, planning tools, subagents, file system. Is that the list of like core components of the modern agent architecture? Those are the four that when we launched Deep Agents, and so the story behind launching Deep Agents was we saw Manus, we saw Cloud Code, we saw Deep Research, they all had these four things.

18:33And we were like, okay, that's pretty common, let's put it into a Python package and make it easy for people to build their own versions of that. So those were the four things at the time. Those are still probably the core things. Some other things that are frequently used, I mean, bash and executing code is a big one that's not always used because sandboxes like Daytona are still new. And so people are still discovering how to run them and how to manage them. And so it's often easier not to do that, but we're seeing more and more want to do that. And so that's where things like sandbox has come in handy.

19:07Skills are a new primitive that didn't exist when we launched Deep Agents, but are now very, very, very interesting. Do you want to explain what skills are? Yeah, skills are great. So they're basically like a bunch of files. There's usually one kind of like skill.md file, which is a big markdown file that contains instructions on how to do something. And there could be other things in a skill as well. There could be other scripts that it could run, but it's basically these instructions for how to do particular things. And rather than being loaded into the system prompt, they are just like referenced in the system prompt.

19:34So you'll tell the agent, Hey, you have access to like this code writing skill, and you have access to this documentation skill. And then if it decides that it needs to use those skills, it will just go basically read those files on demand. People call that kind of like progressive disclosure. You tell the outlaw only what it needs to know when it needs to know. It's another way of letting it manage its own context window as well. So that's a key part that we support in deep agents and most harnesses support. I mean, other interesting things that we're thinking a bunch about, like async sub agents are really interesting.

20:03I mentioned this earlier, but like, I think this is something that most. Like harnesses don't do that well. I think technically Cloud Code has support in it, but I don't even know when it triggers it or it's hard to like observe them and manage them. But I think this will become more and more important. Great. Can you talk about context compaction? We alluded to it a little bit in the context of sub agents. What is it? What is it needed? And how do you do it? Yeah. So compaction happens when you basically build up a bunch of context and you want to condense it down. You want to compact it into something.

20:35Why would you want to do that? most models can't handle infinite context. And even the ones that can handle kind of like a million tokens or something like that, you often don't want to pass that many tokens to them. So it reaches some state and you want to compact stuff down. And so then the question becomes, how do you compact this whole history of what happened into something much smaller? And so the way that we do that in deep agents is we pass that whole history or we pass the part of the history that you want to compact. Because you actually don't want to compact all the messages. You want keep around like the last and messages, let's say the last like 10 or so messages, because if you compact everything, it actually like throws it off completely.

21:12And so these last like 10 or end messages are pretty important for letting it kind of like keeping its flow. But then you take all the previous messages, and you basically condense it. And then this is where, you know, we do some prompt engineering to basically say, okay, pull out like the main objective and pull out the, you know, important things to remember the files that are important. And so then that That becomes a new summary that's put into the context window. And then we put the whole original messages into the file system as well. And that was a new thing that we did to basically, these summaries aren't perfect.

21:42And so yes, hopefully we think that the summary works for 80%, 90%, 95 % of use cases. What if there's some really important piece of information that you can only get from the raw history? Great. That's when we want to let you do that. And so that's why we kind of dump that into a separate thing on disk. So that's how we currently handle compaction. One interesting thing there, actually, that we haven't yet released as of this recording, but will probably be released by the time it comes out, is we actually give the agent a tool to trigger its own compaction. So right now, in I think pretty much every framework out there, it's triggered when it reaches some kind of threshold.

22:17Like, hey, you're at 80 % of your context window, let's compact. In the spirit of letting the model do more and more, we're going to give it a tool to let it call that on its own. So if you're chatting with it and you're like, okay, agent, go do X. And it just goes and it's at like 60%. That wouldn't normally trigger it. But then you're like, go do something completely unrelated. Go do Y. It should trigger that because there's nothing about that that needs to kind of like get kept in history for it to do Y. And it's just distracting and costs more and stuff like that. So this is still pretty new, but we're giving it a tool to basically call its own compaction.

22:50I think Anthropic has some things in their API that I haven't really seen anyone use, but it's in that vein of letting them all decide when to compact, which I'm totally for because it's very much in the spirit of letting the model do more and more. As you describe all of this, I'm trying to figure out what the concept of memory means because it seems like there's memory in the file system, there's memory in the sub-agents. Is memory in other places as well? What is memory for agents? Memory is super important. I mean, I think a lot of what we've been talking about so far I would describe as short-term memory, which is really within a particular thread or conversation.