The Building Blocks of Agentic Systems with Harrison Chase - #698

0:00I think like at the end of the day what it all comes down to is communication. Like if there's some agent that's doing something, we need to communicate to it how we want it to behave. And in some cases, maybe a prompt is enough. But, you know, code is a very good communicator of what we want to happen as well.

0:31all right everyone welcome to another episode of the twimble ai podcast i am your host sam charrington today i'm joined by harrison chase harrison is co-founder and ceo of langchain before we get going be sure to take a moment to hit that subscribe button wherever you're listening to today's show. Harrison, welcome to the pod. Thanks for having me. It's great. It's great to be here. I'm a listener for a bunch of episodes, so honored to be here. I'm excited to chat with you and looking forward to digging into all things Langchain, as well as touching on topics like RAG and agents and evaluation and more.

1:08I'd love to have you start us out with a a little bit of your background and the kind of the Langchain story. Sure. So yeah, my background's in ML and MLOps. So I worked at a fintech company for a few years on the machine learning team there, did some time series stuff and then some NLP stuff, specifically Entity, Blinking. Kensho was the name of the company. Great, great company, very strong kind of like team to learn from and really grateful to a lot of the people there who helped me get started in the industry. then worked at robust intelligence which is an ml ops company doing testing and validation of machine learning models uh smaller earlier startup um and so it was there for a few years at some point knew knew i was going to leave wanted to go back to someplace small or start my own thing didn't didn't quite know what i wanted to do and this was in uh like september of 2022 so started going to a bunch of meetups hackathons talking to a bunch of people this was right after stable diffusion had kind of come out, but before chat GPT had still launched.

2:14But a lot of people were building on the underlying OpenAI APIs. And so got a chance to talk to a bunch of them, saw some patterns in terms of what they were building. I thought it would be really fun to just try to abstract out some of those patterns and put that in a Python package. And that became LinkChain. And a year and a half later, almost two years later, here we are. So that's happy to chat more about what we do at Langchain now, but that's a bit on my background in the start. No, that's awesome. And I think I pulled some stats from your website. You are at least as of the last website update as like 15 million monthly downloads, 100 ,000 apps powered by Langchain, 75 ,000 GitHub stars, 2 ,000 contributors.

3:01That is huge growth for such a short period of time. What do you think is the driver behind that? The rise in popularity of LLMs, basically. Like, right time, right place. I think we launched Lanchain nearly, like, almost nearly exactly a month before ChatGPT. It was kind of the first of its kind in terms of being these kind of, like, orchestration middleware frameworks. A lot of people want to build with LLMs. We try to make that as easy as possible. and I think the yeah maybe the numbers that's that's most interesting is like the number of contributors you know I think two thousand's a lot I think it's yeah it's probably a bit higher by now but I think that's largely due to just the fact of you know where we sit in the ecosystem so we really say as kind of like the glue that connects all these components together and there are like 80 different LLM providers that we integrate with 100 different vector store providers you know a hundred different kind of like document loaders like there's all these integration points and that's that's really where we get a lot of the community contributions from and i'm extremely grateful to everyone in the community who added those because there is such a long tail of integrations and so i'm very thankful for people who try to bring their tool of choice into the into the lang chain ecosystem absolutely so lang chain was the original product that was that abstraction layer that you talked about, but the product family has since evolved to include LangGraph, LangGraph Cloud, and LangSmith.

4:34Can you kind of talk about the portfolio as it sits today and what each of the products does for users? Yeah, absolutely. And I'll maybe go chronologically. So we launched with LangChain, the open source kind of like package. And again, that was a side project kind of like launched before there was a company. There was no strategy. There was no real thought kind of like behind it. As soon as we thought about seriously forming a company around the package, one of the things that we started working on was LangSmith. So we've been working on this basically from the beginning as well. And the idea is we noticed that one of the biggest pain points for people when they were building applications, whether they were using LangChain or not, was basically bridging the gap from prototype to production.

5:21And so I think there's a lot of factors that go into that, obviously. But one of the big things was just understanding what exactly is going on in your app, testing to see how well it's doing, and then also making sure that you don't introduce regressions. When you do start to get some scale, kind of like monitoring that, having some sort of like annotations that get turned into some data that you can use for evaluation or fine-tuning or few-shotting or something. And so that's really what we built LangSmith for. It works with or without Langchain. The two big parts are observability and then testing and evaluation.

5:55But we also have a prompt hub, a human annotation queue, monitoring charts and things like that. and so we launched that initially a little over a year ago was in private beta for a while we launched in ga a few months ago and yeah seeing i'd seen pretty good adoption there about 20 percent of our usage comes from non-link chain users i think that's that's a fun fact that i always kind of like like to track um and and so that's kind of like langsmith and you know could talk could talk for hours about LangSmith, there are other things that we're working on. Notably, recently, LangGraph we've been spending a bunch of time on.

6:36And so the way that I think about LangGraph is LangGraph is purely orchestration. So I mentioned before that LangChain had a lot of integrations, and that's one of the big benefits of using a framework like LangChain. Another benefit of using a framework, especially as the types of applications you're starting to build become more and more complex is the orchestration framework. And so Langchain has an underlying orchestration framework, but Langraph is basically a version that's much better suited towards agentic applications. So specifically ones that involve lots of looping, lots of branching based on what an LLM decides to do, and generally some form of persistence in memory.

7:17And so all of these are built into LangGraph. LangGraph is very low level, very controllable, has a built-in persistence layer. And so we launched this as an open source project at the start of the year. Saw really good interest in it, especially among people who were trying to bridge that gap from prototype to production. So there's a little bit of a learning curve. It's not the easiest thing to pull off the shelf and get started with. LangChain's easier for that. But if you really want to do serious things, I think Langraph is the place to turn to. And then you mentioned Langraph Cloud. That's something very new.

7:54That's like a few weeks old. That's basically a hosted runtime for Langraph. So you can kind of think of Langraph as being a framework like Airflow, but for agents or something like that. And then Langraph Cloud is infrastructure for deploying that. So the way that I like to describe that is like, if you think of the assistance API that OpenAI launched, There's a lot in there besides just a model, right? There's like persistence of all the chat messages. You can create different assistants. You can create threads. They have a concept of background runs. Exactly, yeah. And so the idea behind Lengraph Cloud is like, okay, define your agent with Lengraph.

8:30That's like the cognitive architecture, the logic of the agent, but then deploy it to Lengraph Cloud and get this infrastructure. That is really handy. And OpenAI did a great job with a lot of stuff there. And so get all of that, but for your specific kind of like agent and your specific cognitive architecture. I'd love to dig into agents and the way you think about that as an opportunity for users and developers, as well as kind of an opportunity for Langchain the business. How would you describe kind of the state of the world in terms of agents? What are folks doing today? How capable are the technologies?

9:12you know I see both a lot of promise but a lot of frustration like there's kind of a huge gap in capability in order to make all the the promise real and I'm wondering what you're seeing if you're seeing similar or are there some sweet spots in terms of applications where folks are really putting this agentic idea to use there are definitely some sweet spots and so so maybe be talking about like in my mind a quick history of kind of like agents so i think like you know actually before chat gpt came out there was a great paper by chenu on react so reasoning and acting and basically the core kind of like um cognitive architecture for a pretty generic type of agent um and so that was in like october i think of 2022 we included that in lang chain in like november um and uh you know the whole space kind of took off when chat gpt took off but this idea of like using an agent uh and i can maybe talk about you know i think agents also not the best word because i don't know if there's a concrete kind of like agreed upon definition and it means a bunch of different things a bunch of different people but for the purposes of kind of like it's yeah for the purposes of this like um you know let's think about something simplistic that's running kind of like in a loop and that's exactly what kind of like took off with auto gpt So AutoGPT, like March of 2023, you know, fastest growing GitHub project in history or something like that.

10:38And what they did is they basically ran an LLM in a loop. They gave it a bunch of tools that were really powerful. So like write to file system, search the web. And they basically gave it really ambitious tasks as well. Like, you know, grow my Twitter following or something like that. um and i think that idea of having this kind of like autonomous system that just did something for you was really interesting to people and that's where i think the peak of or that's when i think interesting agents really started taking off um i'd say it probably hit a peak in like summer of 2023 and then i think people as you mentioned kind of like started noticing a lot of the flaws of agents mainly that they weren't really reliable um they didn't really you know do what you asked them to do.

11:24They weren't capable of doing really complex tasks. And then secondary, they would take a while and were really slow. But I think the biggest thing is they just didn't really work. And so I'd say like largely between like, you know, the summer, late summer of 2023 and the end of the year, there's a bit of kind of like decrease and increase in skepticism of agents. And then I think starting in the beginning of this year in 2024, there have been more kind of like agentic systems that have been shipped. So I think like, you know, Ramp shipped an agentic system, Notion shipped something, Elastic shipped one, we wrote a case study with them, Klarna shipped something agentic-like, right?

12:08Can you describe one or two of those just for context? Yeah, absolutely. So the one that I'm kind of like most intimately familiar with is the Elastic one because we work pretty closely with them. So that is basically an assistant. It's an assistant for some of their logs, basically. So you could go in, you can debug things, you can take actions, it can answer questions, it can do a little bit of like rag research, but like not like a simple chatbot. It can like dive a little bit deeper into the logs and explains things. I think another one that I'll note is Ramp. So Ramp had a really cool one as well where basically you'd go on their website and you could ask their agent, like, hey, how do I, I don't know, file this expense report?

12:52And it wouldn't just chat back. It would actually like show you the button to click. And so it was in the browser. And I thought that was really cool because it wasn't really chat-based. And so I think those are two good examples. And I think some common things between all of these are that they're relatively focused, kind of like applications. You're not asking it to increase your Twitter following, right? It's much more narrow, like, hey, what's going on in this log or something like that? And so they're very focused. And then if you dig a little bit into the cognitive architecture of what's going on behind the scenes, for a lot of them, they're not really just running in a loop.

13:34They're a little bit more bounded. Maybe they have some check after the fact that like is verifying that what it does is correct. Maybe they have some explicit like classification step at the start to determine whether it even needs to use kind of like an LLM or not. But there's basically this like really custom, what we're kind of like calling cognitive architecture, which, and we didn't come up with that term. That's very kind of like, I think in 1960 was the first time or somewhere around there that people started using that term. But basically, when we talk about it, we use it to describe how your system or your application thinks.

14:12And we're generally talking about systems that are using LLMs to do this reasoning and then interacting with external things. Anyways, the point is that you don't just like, for all these applications, they're not letting it run unbounded. They have lots of little checks and little classification steps that really keep it focused on a specific workflow. And I think that's the commonality that a lot of these agents has is they're very like workflow based. And so I'd say like two of the big industries where we see them being successful. One is customer support. And then another is in kind of like data enrichment.

14:44So talking about customer support, you know, there's generally a pretty, there can be a lot of edge cases, but there's generally one, a nice kind of like job to be done. There's existing channels of communication. There's generally some like reference material that like 50 % of the questions can be answered by. and then like maybe there's uh you know customer specific information that answers another 30 and then the great thing about customer support is that you have this like built-in concept of escalation very naturally where you can just escalate them to a human and so it's a very kind of like it's it's a great place and i think that's where we saw a lot of uh agents first kind of like being used um and then a second category that i think has been coming online more and more recently is this idea of like data enrichment so we we did this internally for you know we launched Langraf Cloud.

15:35We got a bunch of people signed up for Langraf Cloud. We want to learn a little bit more about them and their company. We don't want to ask them about it, though. That's a lot to fill out in a form. So they say, hey, my name's Jim. I'm from company XYZ. We'll use a search engine. We use Tavilli for our case, but there's a bunch of other ones out there. And we'll go basically go use an agent to search the web and fill in different pieces of information, like where the company is located how big are they have they raised funding things like that um and i think this is this is a uh we see this being used a bunch in uh sales and and kind of like in enriching sales data because that's a very valuable place to do it um but i think in general this concept of data enrichment is a second like really good use case for these agents and it's again very like workflow based like you know if you think about how you or i would do this it's a pretty reasonable workflow of like, I'd go to Google, I'd search something and maybe like, look at the first result.

16:33Okay. Maybe that has it great. Then I'll fill in the information. Maybe it doesn't have it. Then I'll go back. I'll go check the second result. Okay. Maybe I need to redo the search. I type in it. Like there's this general like workflow that we can kind of describe. And I think that's the key for like reliable agents, like workflows that you can kind of like reasonably describe, whether it's through prompting or whether it's through the cognitive architecture of your whole system. Now, just digging into that a little bit more deeply, we've had data enrichment for a while. Entire large companies have been built on scraping the web for this kind of data.

17:07From the perspective of this cognitive architecture that you've described, dig a little bit deeper into what makes these systems agentic. Is it an LLM making decisions about when to continue? You kind of suggested that. Is it, you know, but you've also said classification, like classifiers, like imagining, you know, once you start going down a path of having your LLMs doing classifiers, it's only a matter of time before you pull those out and do something cheaper. Like, you know, what is like the core agentic aspects of this particular example? Yeah. And so I really liked what you said as well, using the word kind of like agentic rather than kind of like asking, like, you know, what's an agent in this case?

17:54I'm stealing this take from Andrew Ng, but he basically said, like, you know, it's really hard to find what an agent is, but there's the spectrum of agenticness. And rather than talking about whether something's an agent or not, we should be talking about like how agentic it is. And really, in what ways? I don't know that saying this is a seven agentic versus a 10 agentic really does justice to all the different ways that a system could be agentic. And for that matter, I'm kind of coming around on this whole line of thinking. Like my initial perspective on like what an agent was, was, you know, the thing that we've all kind of wanted like this avatar, for lack of a better term, you know, that's also super overloaded.

18:37But like this thing that lives, you know, in the web that like does stuff for me. Hey, I want to, you know, take a trip to New York City, like go figure it all out for me. and would kind of autonomously take some actions on my behalf based on things that it's seeing out on the internet.

19:02And as much as we get out of LLMs, we're still pretty far from, I think, that vision. But when I see some examples of systems where people are, you know, using LLMs to, you know, go fetch a lot of information, kind of frame out some decisions, interact with other downstream applications, reformatting information. I guess seems to be a good fit for what LLMs are capable for. And it does like if you squint the right way, it feels agentic. Yeah, I think like there's a lot of components that make up what it means to be agentic, I think the main one that I kind of look at is like, how much is the LLM deciding the control flow of the application, basically?

19:53And so like, you know, classification, yes, is, you know, it definitely, this is why I like the spectrum of agenticness rather than a concrete definition. Because when you do classification, you're kind of determining what happens, what's the control flow of the information. But like, yeah, one classification step isn't that agentic. but maybe you add you know if you have 10 different steps and some of them can loop back to each other okay maybe that's a little bit more interesting if you have kind of like tool calling now like then it's deciding what tools to call right off the bat and and and so that's uh and it's deciding the inputs to that as well and so i think like when i uh yeah one of the things i always try to understand when when talking with people about the systems they're building they're like yeah how much is the llm deciding kind of like what's happening and what the control flow of of the application is And so going back to your original question about talking about these data enrichment agents and like what maybe makes them different or why LLMs are good for it, I think there's a bunch of cases in data enrichment and in just things in general where there's some level of ambiguity or you hit some edge cases or you hit some kind of like errors.

21:00so like uh you know i think lms make it much easier for the ambiguity part like you can just describe kind of like in natural language what fields you maybe want it to enrich for and it can go do those right off the shelf you don't have to kind of like have uh done that before to like have some confidence that it will do it well and then when it goes tries to search like if it doesn't find something the first time it can decide that like what it found isn't good enough right like It can make that judgment. And then it can also decide, do I need to look for another link on the search? Or do I actually need to issue a new search?

21:35And then after it issues three new searches, it can decide, hey, there's really no good information here. Or it can say, hey, I actually need to issue a fourth because I think I'm narrowing in on it, right? So those types of decisions help cover some of the long tail of things. It makes it easier to just, I mean, this is slightly separate, but using LLMs makes it easier to just do data enrichment on kind of like arbitrary fields. And so, yeah, for those two reasons, I think tasks like data enrichment are good candidates for agentic things. Yeah, I wonder how you'd respond to this. Another argument that I sometimes make against like agents is, you know, there's often this feeling that, and this, I think this applies broadly to LLM-based applications, but agent stuff in particular that like we're not really building for the LLMs as they are today.

22:30We're building for like, you know, GPT-5 or like the version after that. Like, you know, the point is to, you know, learn and grab some land and kind of build some capability and hope that the LLMs catch up. But in the case of agents, like we've added, you know, tools into, you know, kind of the core LLMs to a large degree. At least that's the way most developers think of them. You know, we're adding, you know, super long context windows. Um, I guess the, like this argument is that like, okay, you're saying, you say that like you're, you know, the agent doesn't really work right now. It's kind of flaky because the LLMs aren't all that great with like cognitive stuff and reasoning, uh, that's going to come in the next version of, you know, GPTX, whatever.

23:22Um, but the next version of GPTX might not need the whole agent architecture in order to do the thing that you're trying to do. It might just be able to do it. Any reaction or thoughts of that just to help me tune my, you know, my kind of mental model of this? Yeah. I mean, a few quick thoughts, like one, like I think there will be tasks that like the they'll always be some aspect of agenticness that's needed. So like if I need to look up kind of like current information, I need to get that from somewhere. And like possibly this is handled under the hood by GPT-5, but that just means that under the hood GPT-5 is an agent itself.

24:02But if you need to connect to kind of like knowledge that's not in the training data, then like you need to get that somehow. And you need to do, you need to do some calls, get that reason over it, maybe do some more calls. Two, like I do think there are cases today where people are doing where people are using agents to just make like two calls to an LLM and there's nothing in the middle. Like there's no calls to external databases or there's no calls to external search engines. And I think those types of things absolutely will disappear as the models get better and better. Like that to me is a prime case of something that should just be a single prompt as opposed to tool calls.

24:40um and is an example of this like a research type of application where you're asking an llm to develop some idea and then pass it to an editor to give it some to give the original llm uh you know instructions for improving some writing or something yeah yeah i think i think that's a good example i think uh just rewriting things in a different tone so like having one llm generate an answer and then rewriting it in different tone, that can probably be one prompt in the future. I think an even more extreme example of this is just like chain of thought prompting, right? So like that used to be a prompting technique that was needed in order to get any reasonable answers.

25:20Now that's just RLHFed into the models. And I think that will happen more and more. And that wasn't really two calls, but it was, you know, a prompting technique that also became less relevant. And I think that absolutely will happen. I think you're alluding to something I've seen that I think of very similarly. And that's the idea that a lot of what I've seen folks trying to use agentic systems for is patching. Oh, maybe this is very similar to what I said previously, but it's like trying to get past shortcomings of LLMs. Like, I have this task. I'm going to decompose it into, you know, these three subtasks and send those out to individual quote unquote agents.

26:06But that's mostly because either the context window is too small or the LLMs aren't strong enough reasoners and they lose the point and they can't kind of hold multiple things in context at one time. Is that kind of what you're getting at? To some degree, yes. But I'll also argue against that. and so what I'll say is that I think like at the end of the day what it all comes down to is communication like we just like if there's some agent that's doing something we need to communicate to it how we want it to behave and in some cases maybe a prompt is enough but you know code is a very good communicator of what we want to happen as well and so if there's some system that breaks things up and whether that's like like right now like it absolutely is almost needed for these models to work But, you know, going forward, maybe that's easier for us to do.

26:58Maybe it's more deterministic and it's cheaper. And so we can just, you know, code is a form of communication as well. And so as we think of these systems as being agents, and we need to communicate to these systems how it should behave, like, is that really going to be all text? Or will there also be some code in there? Because code is great at communicating exactly what should happen in specific scenarios. And so I think the answer is it depends. Like, I think for some situations, yeah, there's probably some kind of like chaining right now where there's not a ton of sense in like breaking it up.

27:29And it's like more like you can communicate well enough to the LLM what it should do in one prompt. And it's just not like following it. And that's the shortcoming of the LLM. But I think in other situations, organizing in this modular, reusable way is an efficient form of communication. And that's around for the long term. um so uh you know a boring middle answer but i think it's no i appreciate that i appreciate that and by the way i also i also really liked what you said earlier about like you know uh we have to do this in the shortcoming but that's okay because like you know uh it's it's great to learn how these system works i totally agree with that like there's so much learning that happens from just building with the system today um i think uh there's a good podcast uh that the CEO of Klarna was on.

28:15And he was basically saying like, you know, a lot of our internal tooling we built and like, sure, maybe we could have bought some of it, the AI specific internal tooling. But like, we learned so much from that, that even if like, you know, we're like, okay, we don't really need this, we'll go buy it now. Like they accumulated all that knowledge that's going to be valuable in the future. And so I think I really liked that you said that, because I think there's so much truth to that. Beyond the limitations of LLMs today, What are the key challenges that folks are running into when they're trying to deploy these agentic systems?

28:52I mean, I think a lot of it still comes down to just communication. And so some of this is due to the LLMs not being good at understanding. But I think there's a bunch of situations where we see people building agents and they're like, why isn't this agent working? and they're just not providing it with the right context, whether this is through the prompt or through retrieval or through the tools that it has access to. I think we're still figuring out as an industry how to best communicate with these LLMs. And so there are definitely scenarios where they can be useful, but they're underperforming because the communication on our part isn't kind of good enough.

29:29So I'd say that's probably number one. I mean, I think after that, like cost and latency start to become somewhat of an issue. Models have been getting really cheap recently though. So that's less of an issue. Latency is still, well, it depends what scale you're operating at. I shouldn't say it's less of an issue, but it depends what scale you're operating at. Latency is maybe the one we hear more than costs, especially if you're doing chat, Like, if you're doing multiple calls behind the scenes, it starts to add up. And so I'd say, like, number one thing we talk to companies about is still performance.