Context Engineering Our Way to Long-Horizon Agents: LangChain’s Harrison Chase

0:00People use traces from the start to just tell what's going on under the hood. And it's way more impactful in agents than in single LLM applications. Because in single LLM applications, you get some bad response from the LLM. You know exactly what your prompt is. You know exactly what the context that goes in is because that's determined by code. And then you get something out. In agents, they're running and repeating. And so you don't actually know what the context at step 14 will be because there's 13 steps before that that could pull arbitrary things in. So like what exactly is everything's context engineering.

0:28Context engineering is such a good term. I wish I came up with that term. It actually really describes everything we've done at Langchain without knowing that that term existed. But traces just tell you what's in your context. And that's so important.

0:57Welcome to Training Data. Harrison, you were our very first guest on Training Data, and the AI space has moved so quickly in the 18 months or so since we originally interviewed you, And so I'm delighted to get you on the show today. Topics of the moment, I think there's nobody better than you to talk about some of these topics. We're going to talk first about long horizon agents and agent harnesses. Pat and I had this blog post on this yesterday. I know this is something that you are deeply fluent in. And then we're going to talk about what's the difference between building long horizon agents versus building software and the role that you see LinkedIn playing in that ecosystem.

1:29And then finally, I just want to chat with you about the future. I think you single-handedly, you know, kind of saw the agent opportunity, I think, before anybody. You know, we were back in the GPT three days. And I think you see the future for what's happening with agents. And so I'm just excited to chat with you open-endedly about the future as well. I am really excited as well. Thank you guys for having me back. It's quite an honor. I'll tell my mom again that I'm back on the podcast. Wonderful. Okay. Let's start with long horizon agents. Yes. That was a great term. You guys wrote a great article.

1:59Sonia's good at naming things. We're not going to get into the backstory there. What do you think? What do you agree with? What do you disagree with? I mean, I agree that they're starting to finally work. I think the idea of running an LLM in a loop and just having it go was always the idea of agents from the start. Auto GPT was basically this. Then this is why it took off and captured so many people's imagination because it was just an LLM running in a loop completely deciding what to do. The issue is the models weren't really good enough and the scaffolding and harnesses around them weren't really good enough.

2:31And I think the models got better. We learned more about what makes a good harness over the past few years. And now they start to like really, really work. And you see this in coding first. And I think that's the domain where they're taking off the most. And that's spreading to other domains. But you can give a task to an agent and you still need to communicate to it what you want it to do. And it needs to have the right tools and all of that. But it can actually operate for longer and longer periods of time. And so that's the long horizons kind of like framing of it, I think, is really, really apt and really, really good.

2:59Awesome. Awesome. What are your favorite examples of long horizon agents? And I guess what shapes do you see them taking? So coding is the place where there's the most. I think that's the one that I probably use. Yeah, that's the one that I use the most. Adjacent to that, I think like really good ones are AISREs. So Traversal, I think, is a Sequoia company and they have an AISRE that operates over longer time horizons. research in general and i'd call like ais or is kind of research like they're taking a incident and they're going and digging through logs like research in general is a really really good task because it ends up producing like a first draft of something and the issue with agents is they aren't like reliable to nine nines of reliability but they can do a ton of work and more and more work over longer time horizons so if you can find these framings where they run for a long period of time but produce like a first draft of something those to me are like the killer applications of long horizon agents right now.

3:52So coding is an example of that. Coding, you usually put up a PR. You don't directly push to prod unless you're vibe coding, which is also starting to get better and better. AISREs usually surface it to a human who comes in and then reviews it. Report generation, you don't send it out to all of your followers right away. You look at it, you edit it. It creates a first draft of something. So we see this in finance a bunch. This is a huge research opportunity. customer support we see a lot of things pivoting from kind of like the initial the initial customer support was like first line response like someone messages you just respond really quickly and there's still that and that's going great but now there's examples carn is a great example of this where it's like humans and ai working together when the first line fails you escalate to a human you don't just have the human handle it you have this long horizon agent run in the background produce a report of everything that happened and then hand it off to the to the to the agent there to the human agent there.

4:46Agent starts to get confusing in customer support. So I think the killer use case of all of these is places where you have this first draft type of concept. And then how much of the why now do you think is the models themselves are just so good versus people are doing really smart things on the harness side? And maybe even before we get to that, can you say a word for our listeners on how you frame the harness versus the model in terms of the actual composition of an agent? Yeah. So, and I'll maybe bring in like framework as well. Cause I think early on, I mean, that's how we described Langchain as that's what Langchain is.

5:22It's an agent framework. And now, and now we have deep agents, which I'd call an agent harness and we get asked about what's the difference. So a model is obviously the LLMs, tokens, messages in, messages out. The framework would be abstractions around that. So making it easy to switch between models, adding abstractions for other things like tools and vector stores and memory and things like that, but pretty like unopinionated about what actually goes in there. The value is more in abstractions, which can be good, can be bad. Harnesses are more like batteries included. So when we talk about deep agents, we're talking about, we actually give it a planning tool by default.

5:58So it has a tool that comes built into the harness. That's pretty opinionated that this is the right way to do things. We do compaction. So you have these long horizon agents, they're running for longer periods of time. Context windows are larger, but they're still not infinite. And so at some point you need to compact that. How do you do that? There's a lot of research going on there right now. One of the other sets of tools that we and a lot of people are giving to these agents are tools for interacting with the file system, whether directly or via bash. And it's kind of tough to separate from the models because the models are being trained on a lot of this data as well.

6:33And so there's this kind of evolution between like, I don't know if we could have known that like these file system based harnesses are the best thing. Like if we fast, if we go back two years ago, I don't think we could have known that because models weren't really being trained on that as much as they are now. And so they're kind of evolving together. So I think it's like a combination of things. It's the models absolutely are getting better. reasoning models are helping a lot. But it's also the fact that we're figuring out all these permittives around compaction and planning and these file system tools being really useful.

7:06And so I do think it's a combination of both. I remember in that very first episode we did together, you described laying graph, I think, as almost the cognitive framework of the agent. is that the right way to think about what the harness is? Yeah, I think that's right. Yeah, so we build deep agents on top of Lengraph. It's one particular kind of like Lengraph instance. It's very opinionated. It's more general purpose. And so I think early on, we talked about general purpose architectures and more specific architectures. And what we've seen is that a lot of the specificity for tasks previously that might've been in Lengraph because you need to put more structure on the models.

7:50Now that specificity is moving into the tools and the instructions. So there's still the same level of complexity. It's just in natural language. And so prompting and editing those prompts and maybe automatically updating those is becoming a part. But the harness is remaining a little bit more fixed. What's the hardest thing to get right on the harness side? And do you think individual companies can actually excel at the harness engineering side of things? Who do you admire there? I think a lot of the companies that are doing the best harness engineering are coding companies. Honestly, I think that's the place where it's taken off a bunch.

8:28I mean, you look at Cloud Code, I would argue a big reason for the popularity of Cloud Code is the harness itself.

8:35Harrison Chase:Does that, by the way, imply that harnesses are better built by foundation model companies than by third party startups? I don't know. So the next company I was going to mention is Factory, which is another coding company. And I think you look at the hardest they've done there. AMP is another coding company, has a really good harness. I think there's pros and cons. There definitely is some aspect of the harness being tied to a model. And maybe not just not a specific model, but a family of models. So like all Claude models, like Anthropic fine tunes on some specific tools. open AI fine tunes on different ones.

9:16So like, I think probably, probably, probably when we were doing this last time, we maybe talked about how prompts need to be different for one model versus another. Harnesses also need to be slightly different for one family of things versus the other, but there are similarities. Uh, all of them use the file system in, in, in some sense. Um, so I think this is, I, I, I actually don't know the answer to that. It's a really interesting, uh, thing. Um, we see that a lot of the coding come a lot. Everyone who's building a coding company is basically building their own harness right now. And there's all these leaderboards and you can see, it's actually kind of interesting.

9:48If you go to Terminal Bench 2, which I think is probably one of the more kind of like popular coding benchmarks right now, you can actually see they have like the agent harness and then the model. And so you can see the variation in performance and cloud code is not at the top of that. So there's differences, but I think it it doesn't necessarily mean that the model labs are better at it. It just means that you have to understand how the models work. And people who look at what makes a harness tick around the model can get some performance gains there. What do you think goes into making the harness tick?

10:22What do you think the guys at the top of the leaderboard are doing exceptionally well? I think part of it is definitely understanding what tools the models trained on. So I think OpenAI trains really heavily on Bash. I think Anthropic has some explicit kind of file editing tools. And so I think leaning into that is part of it. Compaction's becoming more and more of a thing. So especially as you start doing longer horizon tasks, you start to fill up the context window. And so what do you do there is a really big question. And there's a bunch of strategies for kind of approaching that. I'd argue that's part of a harness.

10:56I mean, so all of these harnesses also this is where like skills and mcps and sub agents start to come into play as well and you can use those in like different ways and and i don't know how i don't think a ton of like skills or sub agents are trained into the models yet like those are still pretty new yeah um and and so like one of the things that we see in our harness is like when you have a sub agent the the the main model needs to communicate with it like well it needs to it needs to give it all the appropriate information it needs to let the sub agent know that it needs to like give it its final response out.

11:28So like we would see some failure modes where the sub agent, because basically what happens is you kick off the sub agent and then only the final response is passed back to the main agent. And so we'd see some failure modes where the sub agent would do a bunch of work and then it would be basically like, look at my work above. And then, you know, pass that back to the main agent and it can't see. And it's like, what are you talking about? And so like that type of like prompting to get these pieces to work together is a big part of it. So like skills, sub agents, MCP, there, there are like prompts in all of these harnesses that make them work well or don't make work well.

11:58And there are hundreds of lines long if you look at some of the ones that are out there.

12:03Harrison Chase:Can I ask you a question on how this has evolved? And since you've always been really kind of on the bleeding edge of what are people doing around the models to make them work in the real world, right? If we think about in our simplistic view on like what the big inflection points over the last five years have been, it feels like there was a big inflection point around pre-training when ChatTBD came out. Feels like there's a big inflection point around reasoning when O1 came out. Feels like just recently there's been a third big inflection point around these long horizon agents with Cloud Code and Opus 4.5.

12:36Harrison Chase:In your world, the world of all the stuff around the models that makes them work in the real world, would you have a different set of inflection points? What have the major changes been? I remember we talked about cognitive architectures a couple of years ago, and now we're talking about frameworks and agent harnesses. Like what are the major leaps in sort of the design around the model? Yeah. What have they been? So I think there's maybe like three eras, I would say. I'd say like early on, and this is when Langchain was just started, like these were still the raw like text in, text out, like not even chat based models.

13:10And so they didn't have any of the tool calling. They didn't have any content blocks, any reasoning at all. They were really just like really, really basic. And so the things that people were doing mostly like single prompts or like chains and it wasn't even possible to do anything like that complicated. Then a lot of the model labs started training in a lot of like the tool calling into the models and they got really good at kind of, or they tried to make them good at like thinking and planning and they still weren't, they still weren't good yet. They sort of weren't good as they are today. But they were good enough to like decide what to do.

13:45And this is where like the custom cognitive architectures would come in more into play because you'd ask it explicitly like, what do I do here? But it was like a very like point in time. And then you go down this branch and then like, what do I do here? And maybe there's a loop and there started to be some loops, but it's still a little bit more kind of like scaffolding around it. And then there was an inflection point. And I don't know where exactly that was. I would say, I think we noticed it probably in like June, July of this year where we saw Claude Code taking off, Deep Research taking off, Manus taking off.

14:11And these all use the same architecture under the hood of just the LLM running in the loop. but like cleverly like a lot of a lot of harness is context engineering like everything around contraction context engineering sub agents context and skills context engineering so we basically saw them using the same core algorithm but making just like improvements on context engineering and we're like oh that's interesting that's pretty different than before and so that's when we started working on deep agents i think for a lot of people in the coding community i think probably like opus four or five was when they started to like really feel this it might have also just coincided with winter break when everyone went home and started using cloud code.

14:46That's how good it was. But I think like around like November, December, like I think there's has been this like. At least I sense a pretty big like vibe shift and like people just like, yeah, you throw hard problems at these things and you get long horizon agents. And so I don't know whether it was early 2025 or late 2025, but at some point the models got good enough. And that's when we moved from like scaffolds to harnesses.

15:08Harrison Chase:And what's next on this arc? I wish I could tell you. I mean, I do think that this algorithm of just running the LLM in a loop and letting it orchestrate its own, letting it really choose what to pull into context and doing stuff there, that is like, it's so simple and so general purpose. I mean, that was the core idea of agents all along, and we're finally there. If you look at some of the manual scaffolding, maybe some of that goes away. So compaction is still very manual. The harness author decides what to do with it. But Anthropic has some interesting things where they let the model decide when to compact things.

15:44We don't really see a ton of people using that. Maybe that'll be a part that's next. Part of what we're really interested in is memory as well. If you think about memory in the context of this, that's also context engineering, right? It's context engineering over longer time horizons. And it's a slightly different set of context, but it's still giving that to the LLM. And I think the core algorithm is pretty simple. It's run the LLM in a loop and we're finally there and it kind of works. And so I think there'll be a bunch of context engineering tricks around it. And maybe some of that is giving the context engineering actually to the LLM, like the anthropic thing.

16:24Maybe some of that is just pulling in new types of context. The models will probably get better. They'll probably get better and better at these types of longer horizon tasks. That'll be great as well. One of the big questions on my mind, so a lot of these harnesses that we see are very coding specific. And that's where we first started to really see these long horizon agents take off. And even for non-coding tasks, I think you can make an argument that writing code is really useful and can be general purpose.

16:54Harrison Chase:I was going to ask you, are coding agents, is that a subcategory or are coding agents just agents? meaning the job of an agent is to figure out how to get a computer to do useful stuff and code is a pretty good way to get a computer to do useful stuff i don't know uh this is one of the big so i i i very very strongly believe that like right now if you're building a long horizon agent you need to give it access to a file system yeah like there's so many things you can do with a file system in terms of context management like when we talk about compaction one strategy is to summarize but put all the messages in the file system so that if it needs to look it up it can Another strategy is when you have big tool call results, don't pass it all to the model, put it in the file system and let it look it up.

17:36Now you can do all of that without a real file system actually, without letting it write code. So we have a concept of a virtual file system where it's just backed by Postgres or something like that and it's more scalable. But there are obviously things you can do with code that you can't do with a virtual file system. You can't run code in a virtual file system. So writing scripts is really useful for that.

17:54Harrison Chase:Yeah. And I think a coding agent can be general purpose, but I don't know if that means that today's coding agents are, if that makes sense, because I think a lot of the coding agents today are pretty optimized for coding tasks. Yeah. And so I think it's possible that a general purpose agent is a coding agent, but I don't know if like the reverse is true, if if if that kind of like makes sense. Yeah. Yeah. We're thinking about that a lot as well. Are all agents coding agents? Yeah. That's one of the biggest things that we're thinking about right now. Yeah. Maybe can we transition into talking about what goes into building a long horizon agent versus building software?

18:29Can you maybe describe the software development stack for 1.0 code development and what's different now? And I thought you had a really good X article on this. Maybe just summarize the punchline. I've been trying to think about this a bunch because we like to say that built and I think a lot of people would agree that like building agents is different than building software. But like what exactly is different? Because I think it's it's easy and lazy to say that it's different. But what actually is different? These might sound obvious, but hopefully that's good and they're not controversial. But like when you're building software, all of the logic is in the code in the software and you can see it there.

19:03When you're building an agent, the logic for how your applications works is not all in the code. A large part of it comes from the model. And so what this means is that you can't just look at the code and tell exactly what the agent would do in a specific scenario. You actually have to run it. And so what does that mean? And I think that's the biggest difference, by the way. We're introducing these non-deterministic systems into it, and it's a black box, and it lives outside. And I think all that's true. That's the biggest difference. So what exactly does that mean? I think one thing that that means is that in order to tell what the application is actually doing, you can't look at the code.

19:39You have to look at actually what it does in real life. And so I think one of the things that we do that is most popular is Langsmith. One of the core parts of that is tracing. Why are traces so popular? Because they tell you exactly what goes on inside your agent at every step. And it's different than software traces where in software, you kind of have your system over here and it emits a bunch of stuff and you look at it when And maybe there's some errors, but you don't need everything. And you usually only turn that on when you put it in production, because if it's local, you just put a breakpoint or something like that.

20:15In agents, people use traces from the start to just tell what's going on under the hood. And it's way more impactful in agents than in single LLM applications. Because in single LLM applications, you get some bad response from the LLM. You know exactly what your prompt is. You know exactly what the context that goes in is, because that's determined by code. And then you get something out. In agents, they're running and repeating. And so you don't actually know what the context at step 14 will be because there's 13 steps before that that could pull arbitrary things in. So what exactly is... Everything's context engineering.

20:45Context engineering is such a good term. I wish I came up with that term. It actually really describes everything we've done at Langchain without knowing that that term existed. But traces just tell you what's in your context. And that's so important. And so what does that mean? That means that the source of truth for software is in code. And for agents, it's a combination now of code and traces are where you can see the source of truth. It's technically in all those millions, billions of parameters, but you can't really do anything with that. So now that means that traces become a place where you start to think about testing.

21:20Because now you can test some parts still of the harness and you can do some unit testing offline. But like in order to get the what the test cases are, you probably want to use the traces to construct that. You probably want to be testing online. That's probably more important in agents than it is in software is online testing because behavior doesn't emerge until it's actually being used with real world inputs. We see traces becoming a point of collaboration for teams because if something goes wrong, it's not, oh, let's go look at the code in GitHub. It's let's go look at the trace. We see this in our open source as well.