LangChain’s Harrison Chase on Building the Orchestration Layer for AI Agents

0:00It's so early on that like it's so early on there's so much to be built. Yeah, like, you know, GPT -5 is going to come out and it will probably make some of the things you did not relevant, but you're going to learn so much along the way. And this is a strongly, strongly believed like a transformative technology and so the more that you learn about it, the better.

0:38Hi, and welcome to Training Data. We have with us today Harrison Chase, founder and CEO of Langechain. Harrison is a legend in the agent ecosystem as the product visionary who first connected LLMs with tools and action. And Langechain is the most popular agent building framework in the AI space today. We're excited to ask Harrison about the current state of the future potential and the path ahead. Harrison, thank you so much for joining us and welcome to the show. Of course, thank you for having me. So maybe just to set the stage, agents are the topic that everybody wants to learn more about. And you've been at the epicenter of agent building pretty much since the LLM wave first got going.

1:18And so maybe first just to set the table. What exactly are agents? I think defining agents is actually a little bit tricky and people probably have different definitions of them, which I think is pretty fair because it's still pretty early on in the life cycle of everything LLMs and agent related. The way that I think about agents is that it's when an LLM is kind of like deciding the control flow of an application. So what I mean by that is if you have a more traditional kind of like rag chain or retrieve logmented generation chain, the steps are generally known ahead of time. First, you're going to maybe generate a search query, then you're going to retrieve some documents, then you're going to generate an answer, and you're going to return that to a user.

2:01And it's a very fixed sequence of events. And I think when I think about things that start to get agentec, it's when you put an LLM at the center of it and let it decide what exactly it's going to do. So maybe sometimes it will look up a search query. Other times it might not, it might just respond directly to the user. Maybe it will look up a search query, get the result, look up another search query, look up two more search queries, and then respond. And so you kind of have the LLM deciding the control flow. I think there are some other maybe more buzz wordy things that fit into this. So like tool usage is often associated with agents.

2:37And I think that makes sense because when you have an LLM deciding what to do, the main way that it decides what to do is through tool usage. So I think those kind of go hand in hand. There's some aspect of memory that's commonly associated with agents. And I think that also makes sense because when you have an LLM deciding what to do, it needs to remember what it's done before. And so like tool usage and memory are kind of loosely associated. But to me, when I think of an agent, it's really having an LLM decide the control flow of your application. And here's said a lot of what I just heard from you is around decision making.

3:15And I've always thought about agents as sort of action taking. Do those two things go hand in hand as agent behavior more about one versus the other? How do you think about that? I think they go hand in hand. I think like a lot of what we see agents doing is deciding what actions to take for all on tens and purposes. And I think the big difficulty with action taking is deciding what the right actions to take are. So I do think that solving one kind of leads naturally to the other. And after you decide the action as well, there's generally the system around the LLM that then goes and executes that action and kind of like feeds it back into the agent.

3:58So I think that in, yes, so I do think they go kind of hand in hand. So Harrison, it seems like the main distinction then between an agent and something like a chain is that the LLM itself is Deciding what step to take that next what action to take next as opposed to these things being hard code Is that like a fairway to distinguish an agent's? Yeah, I think that's right and there's different gradients as well So as like an extreme example you could have basically a router that decides between which path to go down And so there's maybe just like a classification step in your chain. And so the LOMs still deciding like what to do, but it's a very simplistic way of deciding what to do.

4:36And you know, at the other extreme, you've got these autonomous agent type things, and that there's this whole spectrum in between. So I'd say that's largely correct, although I'll just note that there's a bunch of nuance in gray area as there is with most things in the LOM space these days. Got it. It's like a spectrum from control to like fully autonomous decision making and logic. Well, those are kind of a lot of respect to my agents. Interesting. What role do you see Langchain playing in the Aging ecosystem? I think right now we're really focused on making it easy for people to create something in the middle of that spectrum.

5:13And for a bunch of reasons, we've seen that that's kind of the best spot to be building agents in at the moment. So we've seen some of these more fully autonomous things get a lot of interest and prototypes of the door and there's a lot of benefits to the fully autonomous things. They're actually quite simple to build, but we see them going off the rails a lot and we see people wanting more constrained things, but a little bit more flexible and powerful than chains. And so a lot of what we're focused on recently is the being this orchestration layer that enables the creation of these agents, particularly these things in the middle between chains and autonomous agents.

5:53And I can dive into a lot more about what exactly we're doing there, but at a high level, that's the being that piece of orchestration framework is kind of where we imagine length chain city. Got it. So there's chains, there's autonomous agents, there's a spectrum in between and secret sweet spot is somewhere in the middle, enabling people at all agents. Yeah, and obviously that's changed over time. So it's fun to like reflect on the evolution of lane chain. So you know, I think when lane chain first started, it was actually a combination of chains. And then we had this one class, this agent executor class, which was basically this autonomous agent thing.

6:30And we started adding in like a few more controls to that class. And but eventually we realized that people wanted way more flexibility and control than we were giving them with that one class. So like recently we've been really heavily invested in length graph, which is an extension of length chain that's really aimed at like customizable agents that sit somewhere in the middle. And so kind of like our focus, you know, has evolved over time as the space has as well. Fascinating. Maybe maybe one more final kind of setting the stage question. And one of our core beliefs is that agents of an ex -big wave in AI and that we're moving as an industry from co -pilots to agents.

7:10I'm curious if you agree with that take and where, why not? Yeah, I generally agree with that take. I think the reason why that's so exciting to me is that a co -pilot still relies on having this human in the loop. And so there's a little bit of almost like an upper bound on the amount of work that you can have done by an external kind of like By another system And so it's a little bit limiting in in that sense. I do think there's some really interesting thinking to be done Around what is the right UX and human agent interaction patterns But I do think they'll be more along in the lines of an agent doing something and maybe checking in with you as opposed to a co -pilot that's constantly kind of like in the loop.

7:57I just think it's more powerful and gives you more leverage if the more that they're doing, which is very paradoxical as well because it comes, the more you let it do things by itself, there's more risk that it's messing up or going off the rails. And so I think striking this right balance is going to be really, really interesting. I remember back and I think it was March -ish of 2023. There were few of these autonomous Miss agents that really captured everyone's imaginations, like BabyAGI, audit GPD, a few of these. And I just remember Twitter was very, very excited about it. And it seems like that first iteration of an Asian architecture hasn't quite met people's expectations.

8:43I think why do you think that is, and where do you think we are in the Asian hype cycle now? Yeah, I think maybe thinking about the agent hype cycle first, I think the auto GPT was definitely the start and then I mean it's one of the most popular GitHub projects ever. So one of the peaks of the hype cycle. I think and I'd say that started in the spring 2023 to summer of 2023, Then I personally feel like there was a bit of kind of like a law slash down trend from the late summer to basically the start of the new year in 2024. And I think starting in 2024, we've started to see a few more realistic things come online.

9:35I had point out some of the work that we've done at Lengcheng with Elastic for example. They have kind of like an elastic assistant and elastic agent in production. And so we're seeing that. We saw kind of like the Clarna customer support bot kind of like come online and get a lot of hype. We've seen Devon. We've seen Sierra. These other companies start to emerge in the agent space. And so I think with that hype cycle in mind, talking about why the auto GPT style architecture didn't really work, It was very general and very unconstrained. And I think that made it really exciting and captivated people's kind of like imaginations, but I think practically for things that people wanted to automate to provide immediate business value, there's actually a lot, it's a much more specific thing that they want these agents to do.

10:27And there's really like a lot more rules that they want the agents to follow or specific ways they want them to do things. And so I think in practice what we're seeing with these agents is they're much more kind of like custom cognitive architectures is kind of like what we call them where there's a certain way of doing things that you generally want an agent to do. And there's some flexibility in there for sure. Otherwise, you know, you would you would just code it. But it's a very like directed way of thinking about things. And that's most of the agents and assistants that we see today. And that's just more engineering work.

10:58and that's just more kind of like trying things out and seeing kind of like what works and what doesn't work and it's harder to do. So it just takes longer to build and I think that's kind of why, you know, that's why that didn't exist a year ago or something like that. Since you mentioned cognitive architectures, I love the way that you think about them. Maybe can you just explain like what is not what is a cognitive architecture and like is there a good mental framework for how we should be thinking about them? Yeah, so the the way that I think about cognitive architecture is basically what's the system architecture of your LLM application.

11:34And so what I mean by that is if you're building an LLM application, there's some steps in there that use LLMs. What are you using these LLMs to do? Are you using them to just generate the final answer? Are you using them to route between two different things? Do you have a pretty complex one with a lot of different branches and maybe some cycles repeating. Or do you have kind of like, you know, a pretty loop. Would you basically run this LLM in a loop? These are all kind of like different variants of cognitive architectures. And kind of architectures is just a fancy way of saying like from the user input to the user output, what's the flow of data of information of L, I'm called, that happens along the way.

12:20And what we've seen more and more, especially as people are trying to get agents actually into production, is that the flow is specific to their application and their domain. So there's maybe some specific checks they want to do right off the bat. There's maybe three specific steps that it could take after that, and then each one maybe has an option to loop back or has two separate substeps. And so we see these more like, if you think about it as a graph that you're drying out, we see more and more basically custom and bespoke graphs as people kind of try to constrain and guide the agent along their application.

12:59The reason I call it cognitive architectures is just, you know, I think a lot of the power of LLMs is around reasoning and thinking about what to do. And so, you know, I would maybe have like a commutive mental model for how to do a task. And I'm basically just encoding that mental model into some kind of like software system, some architecture that way. Do you think that's the direction the world is going? Because I kind of heard two things from me there. One was, it's very bespoke. And second was, it's fairly brute force. Like it's fairly hard coded in a lot of ways. Do you think that's where we're headed?

13:35Or do you think that's a stopgap? And at some point, more elegant architectures or a series of default sort of reference architectures will emerge? That is a really, really good question. And when I spend a lot of time thinking about, I think so, like at an extreme, you could make an argument that if the models get really, really good and reliable at planning, then the best thing you could possibly have is just this for loop that runs in a loop calls the LLM decides what to do, takes the action, and loops again. And like all of these constraints on how I want the model to behave, I just put that in my prompt and the model follows that kind of like explicitly.

14:18I do think the models will get better at planning and reasoning for sure. I don't quite think they'll get to the level where that will be the best way to do things for a variety of reasons. One, I think efficiency, if you know that you always want to do step A after step B, you can just put that in order and two reliability as well. Like these are still not deterministic things. We're talking about, especially in enterprise settings, you probably want a little bit more comfort that if it's always supposed to do step A after step B, it's actually always going to do step A over step B or after step B.

14:57I think it will get easier to create these things. Like I think they'll maybe start to become a little bit less and less complex. But actually this is maybe a hot take or interesting take that it has. You could say like, so the architecture of just running it in a loop, you could think of as like a really simple but general cognitive architecture. And then what we see in production is like custom and complicated kind of like cognitive architectures. I think there's a separate access, which is like complicated, but generic, custom or complicated, but generic cognitive architectures. And so this would be something like a really complicated, like planning step and reflection loop or like tree of thoughts or something like that.

15:45And I actually think that quadrant will probably go away over time because I think a lot of that generic planning and generic reflection will get trained into the models themselves, but there will still be a bunch of not generic training or not generic planning, not generic reflection, not generic control loops that are never going to be in the models basically, yeah, no matter what. And so I think like those two ends of the spectrum I'm pretty bullish on. I guess you can almost think about it as like the LLM does the kind of like general, the very general all, um, agentic reasoning, um, but then you need domain specific reasoning.

16:22Uh, and, and that's the sort of stuff that, that you can't really build into one general model. 100 % like I think, I think a way of thinking about like the custom cognitive architectures is you're basically taking, you're taking the planning responsibility away from the LLM and putting it onto the human. And some of that planning you'll, you'll move more and more towards the model and, and, and more and more towards the prompt. But I think they'll always be like I think a lot of a lot of tasks are actually quite complicated in some of their planning And so I think it will be a while before we get things that are just able to do that super super reliably off the shelf It seems like we've simultaneously made a ton of progress on agents in the last six months or so like I was reading a paper the Princeton SWE paper where their coding agents can now solve 12 .5 % of GitHub issues versus, I think, 3 .8 % when it was just rag.

17:19So it feels like we've made a ton of progress in the last six months, but 12 .5 % is not good enough to replace even an intern, right? And so it feels like we still have a ton of room to go. I'm curious where you think we are both for general agents and also for your customers that are building agents. like, are they kind of getting to, I assume, not five nines reliability, but are they getting to kind of like the thresholds they need to kind of deploy these agents out to actually kind of customer facing deployments? Yeah, so the sweet agent is, I would say a relatively general list agent in that it is expected to work across a bunch of different GitHub repos.

17:59I think if you look at something at like V0 by Versailles, that's probably much more reliable than and 12 .5 % right? And so I think that speaks to like, yeah, there are definitely custom agents that not five, nine, several liability, but that like are being used in production. So like elastic, I think we've talked publicly about how they've done, I think multiple agents at this point, and I think this week is a RSA, and I think they're announcing something new at RSA that's an agent. And yeah, those are, I don't know if the exact numbers on reliability, but they're all able enough to be shipped into production.

18:38General agents are still tough. Yeah, this is where kind of like longer context windows, better planning, better reasoning will help those general agents. You shared with me this great Jeff Bezos quotes. They're just like focus on what makes your beer better. And I think it's referring to the fact that at the time of the 20th century, breweries were trying to make their own electricity, generator electricity. I think similar question, a lot of companies are thinking through today. Do you think that having control over your cognitive architecture really makes your beer taste better? So to speak metaphorically or do you see control of that to the model and just build UI and product?

19:21I think it maybe depends on the type of cognitive architecture that you're building going back to some of the discussions earlier. If you're building like a generic cognitive architecture, I don't think that makes your beer taste better. I think the bottle providers will work on this general planning. I think like well work on these general cognitive architectures that you can try off the bat. On the other hand, if your cognitive architectures are basically you codifying a lot of the way that your support team thinks about something or internal business processes or the best way that that you know to kind of like develop code or develop this particular type of code or this particular type of application.

20:00Yeah, I think that absolutely makes your your beard tastes better, especially if we're going towards a place where these applications are are doing work, then like the logic, the bespoke kind of like business logic or mental models for anthropomorphizing these alums a lot right now, but like the models for these things to to do the best work possible, 100%, like I think that's the key thing that you're selling in some class and I think UX, then UI and distribution and everything absolutely still plays a part, but like yeah, I draw this distinction between general versus custom. Harrison, before we get into some of the details on how people are building these things, can we pop up a level real quick?

20:43So our founder, Don Valentine was famous for asking the questions, so what? And so my question to you is, so what? Let's imagine that autonomous agents are working flawlessly. What does that mean for the world? How is life different if and when that occurs? I think at a high level it means that as humans we're focusing on just a different set of things. So I think there's a lot of, like, wrote, repeated kind of work that goes on in a lot of industries at the moment. And so I think the idea of agents is a lot of that will be kind of like automated away, leaving us to think maybe higher level about like what these agents should be doing, and maybe leveraging their outputs to do more creative or building upon those outputs to do more kind of like higher leverage things, basically.

21:34And so I think you know you could imagine bootstrapping a entire company where you're outsourcing a lot of the functions that you would normally have to hire for. And so you could play the role of a CEO in an agent for marketing, an agent for sales, something like that. And I'll allow you to basically outsource a lot of this work to agents leaving you to do a lot of the interesting strategic thinking, product thinking. I mean, maybe this depends a little bit on what your interests are, but I think at a high level, it will free us up to do what we want to do and what we're good at and automate a lot of the things that we might not necessarily want to do.

22:19And are you seeing any interesting examples of this today, sort of live and in production? I mean, I think the biggest, there's two kind of categories or areas of agents that are starting to get more traction, one's customer support, one's coding. So I think customer support is a pretty good example of this. Like I think, you know, often times people need customer support. We need customer support at Lengcheng. And so if we could hire agents to do that, that would be really powerful. Coding is interesting because I think there's some aspects of coding that, I mean, yeah, this is maybe a more philosophical debate, but I think there's some aspects of coding that are really creative and do require like really, I mean, lots of product thinking, lots of positioning and things like that.

23:09There's also aspects of coding that limit some of the, or not limit, but get in the way of a lot of the creativity that people might have. So if my mom has an idea for a website, she doesn't know how to code that up, right? But if there was an agent that could do that, she could focus on the idea for the website and basically the scoping of the website, but automate that. And so I'd say customer support, absolutely, that's having an impact today. Coding, there is a lot of interest there. I don't think it's as mature as customer support, but in terms of areas where there is a lot of people doing interesting things, that would be a second one to call out.

23:45Your code on coding is interesting because I think this is one of the things that has as very optimistic about AI. It's this idea sort of closing the gap from idea to execution or closing the gap from, you know, dream to reality where you can come up with a very creative, compelling idea, but you may not have the tools at your disposal to be able to be able to put it into reality and AI seems like it's all suited for that. I think Dylan had figured out talks about this a lot too. Yeah, I think it goes back to this idea of like automating away the things that get in the way of making, like the phrasing of idea to reality.

24:19It automates away kind of like the things that you don't necessarily know how to do or want to think about, but are needed to create whatever you want to create. I think it also, one of the things that I spent a lot of time thinking about is like, what does it mean to be a builder in the age of kind of like gender to AI and in the age of agents? So what it means to be a, you know, a builder of software today means, you know, you, you either have to be an engineer or hire engineers or something like that, right? But I think what it means to be a builder in the age of agents and gender say, I just allows people to build a way larger set of things than they could build today.

25:05Because they have at their fingertips all this other knowledge and all this other kind of like all these other builders they can hire and use for very, very cheap. I mean, I think like this, you know, some of the language around like commoditization of kind of like Intelligence or something like that is these LLMs are providing intelligence for free I think does does speak to enabling a lot of these new builders to emerge You mentioned reflection and and chance odds and other techniques like maybe can you just say word on like what we've learned so far about what some of these I guess cognitive of architectures are capable of doing for a genetic performance and maybe just I'm curious what you think are the most promising cognitive architectures?

25:55Yeah, I think there's maybe it's worth talking a little bit about why kind of like the auto GPT things didn't didn't work because I think a lot of the cognitive architectures are kind of like emerged to counteract some of that. I guess way back when there was basically the problem that LLMs couldn't even reason well enough about a first step to do and what they should do as the first step. I think prompting techniques like Chain of Thought turned out to be really helpful there. They basically gave the LLM more space to think about and think step by step about what they should do for a specific kind of single step.

26:39Then that actually started to get trained into the models more and more and that kind of did that by default. As that kind of like is basically everyone wanted the models to do that anyways. And so yeah, you should trade that into the models. I think then there was a great paper by Shenu called React, which basically was the first cognitive architecture for agents or something like that. And the thing that it did there was one, it asked the LLM to predict what to do. That's the action. But then it added in this reasoning component. And so it's kind of similar to chain of thought in that it basically added in this reasoning component He put it in a loop.

27:20He asked us to do this reasoning thing before each step and you kind of run it there And so that was kind of like and actually that's that that like explicit reasoning step has actually Become less and less necessary as the models have that trained into them Like just like they have kind of like the chain of thought trained into them that explicit reasoning step has become and less necessary. So if you see people doing kind of like react style agents today, they're oftentimes just using function calling without kind of like the explicit like thought process that was actually in the original react paper.