LangChain and Agentic AI Engineering with Erick Friis

0:00Langchain is a popular open source framework to build applications that integrate LLMes with external data sources like APIs, databases, or custom knowledge bases. It's commonly used for chatbots, question-answering systems, and workflow automation. Its flexibility and extensibility have made it something of a standard for creating sophisticated AI-driven software. Eric Fries is a founding engineer at LangChain, and he leads their integrations and open-source efforts. Eric joins the podcast to talk about what inspired the creation of LangChain, agentic flows versus chained flows, emerging patterns of agentic AI design, and much more.

0:40This episode is hosted by Sean Falconer. Check the show notes for more information on Sean's work and where to find him.

0:58Eric, welcome to the show. Hey, Sean. Glad to have you here. I've used Langchain a bunch, so I'm excited to get into it and talk about it. Yeah, me too. So you were there from the beginning. So I just want to kind of go back to that point from what you can recall, like, what sort of for you and other people that were kind of part of that initial founding team, like what inspired you to create Langchain? Like, how did you sort of identify this need for a framework that chained together LLMs, created this abstraction layer? Yeah, definitely. And some important context there is that there's kind of a distinction between the starting of the open source project, which Harrison did in October 2022, and then kind of when the company formed around it, which was kind of the first half of 2023.

1:38So the open source projects kind of came out right place, right time, right before ChatGPT really launched and everyone kind of started building with these LLMs. And that was really Harrison's brainchild. He was kind of working with GPT-3, which I don't know if you interacted with some of the text completion models instead of the chat completion ones. but they were relatively challenging to work with. It was kind of string in, and then it just kind of continued the string output. And so there was a lot more manual steps you had to do as a user of those from kind of the concept of output parsing was kind of one of the big reasons that people use LinkedIn in the early days, all the way through, like, how do we turn this into some sort of agentic loop using some of the current research of the time?

2:17And so we've kind of continued that effort as the company, where we're kind of keeping up with the latest models and kind of integrations that people want to work with. That's primarily the area that I work in, as well as making kind of usable implementations, both for prototypes, as well as production applications of kind of the latest and greatest research. Like how do you build software with these new models? So back in the early days of like, you know, GPT-3, there was, I guess like their APIs were not necessarily the easiest thing to use, but they have as a whole, I think like the sort of the model foundation model companies have gotten a lot better in using their APIs directly?

2:54How does that kind of change LangChain's focus in terms of the value add that they're bringing above and beyond just having essentially a cleaner API into some of this stuff? Totally, yeah. So let's kind of compare October 2022 to now, where the landscape in October 2022 was you kind of used GPT-3, maybe you used one of the early cloud text completion models as well. And so there was, I think, three integrations in the original LangChain library, which were those LLMs. And at that point, you had to, as a user, handle all of your kind of message formatting, all of your output parsing into messages manually, where the kind of simplification of what these chat models are doing is it's still just a text completion model, but they're trained on very specific formats of alternating like human message, AI message, human message, AI message.

3:42And so that allows the API providers to actually guarantee that the next message is going to be an AI message in a more strict way than we can do as kind of just observers of the output of the model. And so with them handling more of that, our focus becomes a lot kind of further up the stack, if that makes sense. So our main focus right now is really on LandGraph. How do you orchestrate these kinds of agents as state machines, where the LLMs are clearly very powerful, but they're not quite powerful enough yet to build functional software where with just the simple React loop, where a React loop is just give an LLM access to all the tools that you want to give it access to, go and execute those tools, and then pipe that output back into the model.

4:26And so that's the simplest React loop. Doesn't work all that well, because as you increase the number of tools that you provide to the model, it starts calling the wrong tool. Sometimes it doesn't call it with relevant parameters to it and those kinds of things. And so what we've kind of been doubling down on is, give your LLM access to, and it depends a lot on the model, but give your LLM access to five tools or have some sort of flow where at different steps, it might have access to different ones. Let's use an email assistant as an example. You might classify an email either as recruiting inbound or from a recruiter or a candidate reaching out to you.

5:07And in those two different instances, you might want to give it access to different tools in terms of like, okay, I want to give it access to nothing and just respond like, hey, like write a draft of interested or not interested, depending on the background company or a candidate coming in, you might want to attach that to something like your recruit, like greenhouse or applicant tracking system in order to track that they emailed you. And so you can actually segment that request in this kind of like nice graph flow that we visualize in LineGraph Studio. Is that really about sort of having like an opinion, I guess, as like a framework for how people need to stitch together these agents.

5:43So rather than someone kind of like, you know, making the mistake of creating sort of this almost like a, you know, monolith agent, where it's going to have access to 1000 tools, you're saying, like, don't do that. If you use this opinion, essentially, and the flow of it will essentially force you to kind of break it up into modular components that make more sense. Precisely. And that actually is kind of a fun part of Langshade's evolution, where the first agent abstraction we came out with was the Langchain agent executor, which really just implemented that broad React loop. And to be clear, lots of people are using that and being very successful with it, as long as they're engineering the tools that it has access to in the right way, where you kind of have to have a limited number of tools, and you have to have really good prompting and descriptions for how to call it, such that the agents actually end up calling it.

6:31And it obviously performs better with kind of the latest and greatest larger models when you do that. And some of these line graph flows really enable you to use smaller models as well, both for cost savings, or maybe you want to run on hardware that isn't as powerful. Yeah. So let's stay on agents, but maybe before we go deeper on that topic, like, can you explain sort of like, what is different about like agentic versus sort of what we've maybe seen previously of sort of these fixed flow architectures through things like RAG and so on? Yeah, the distinction in my mind is really whether it's a feedforward application or a cyclic application.

7:07And so we kind of distinguish them as chains versus agents or graphs, if you're building them with Landgraf. And a chain always finishes, like it always just kind of goes through the steps, maybe for the rag case, it does a retrieval step, looks up some documents that it wants to paste into your prompt, passes that prompt to the LLM and generates some nice description that you might get out of a perplexity or something like that. like that. In the agentic version of RAG, you really do that retrieval step, generate that output. And then you might even like fact check that output and say, hey, is this factually accurate?

7:41Or you might do some other steps that kind of filter out that output. And if you don't like it, you can actually just bounce back to the beginning and say like, hey, regenerate this based on this feedback that our editor node or editor sub agent told it to do. How do you get into like essentially avoid a situation where you are running an endless loop reflection and planning and then like this cycle never actually finishes? Yeah, there's a few different strategies. So by default, Langraph has a recursion limit. So you can really think of this as the same problem you end up in when you like recursively call a function too many times in Python, or in any language where it'll kind of hit that stack limit.

8:17It's kind of the equivalent concept for a graph where it's really designed to hit that. And there's ways that you can kind of handle that case such that we can kind of gracefully exit when we hit that based on the artifacts that we've generated through all of those steps. But we've also seen a lot of people implement kind of just tracking in the state of a graph. Some important background on line graph is the model is this, like all these nodes and edges that you connect to each other, but all of the nodes operate on the same schema. And so we call that typed dictionary just the state of the graph.

8:50And so you can have a state field of like number of times I've fact-checked the answer. and it starts at zero and you just increment it each time. It's just a different way of writing a for loop. And when we hit three, we say like, okay, like we're kind of done fact-checking this. Now let's just respond to the user and say like, hey, I'm not completely sure if this is the right answer, but here's what we kind of ended up with. And so that caps the amount of time that your agent can be spent producing an answer. And in terms of like patterns of behavior, like what are some of the agentic patterns that people are using and that are supported in the box of line graph?

9:24Yeah, great question. So the first one that everyone starts with, well, I shouldn't say everyone, but many people start with is that React agent that I mentioned before. Because it's so simple, it's just two nodes. One of them is the LLM calling node and one of them is the kind of tool, we call it tool node, but it just executes the code associated with the tool and produces the output that can be passed back to the model. That is kind of the quickest dopamine hit when you're kind of getting started building agents where you can really build something that goes and sends an email for you or sends a Slack message to you based on some input that came in.

9:56But we very quickly see people start adding human in the loop type outputs where, okay, whenever I call my send email node, I really want to review that first. So whenever, before actually executing the send step, I'll interrupt is kind of the concept in line graph and show the user the draft that I've written. And they can choose to give some feedback that can be then edited and shown as a news draft or just hit send and send it away. So React is kind of where people get started. Human in the loop is one of the patterns that we see recurring and is now a first party concept in LandGraph. Other ones that people are building a lot with, we now have a concept of kind of a global state store.

10:38So whenever you start a conversation with an agent, we consider that a thread, kind of similar to how you would have alternating messages in ChatGPT. But you might want to have some sort of memory that's actually tracked between multiple interactions with your agent. And so that's where we get into kind of that global state store in the checkpointer. We've experimented with a lot of different versions of memory. And what we've kind of come to is a realization that kind of less is more, right? Like just being able to set and get keys, potentially with some added features around filtering by sessions or like other threads that are kind of associated with a certain one.

11:16those are useful abstractions, but maybe automatically editing and like trimming conversation histories and things like that, maybe not as useful. Yeah. I think another like simple pattern too is basic reflection. I mean, and you can even like, if you just go to chat GPT as a user, you can experiment with this where you say like, you know, write me an email that touches on these things and then copy that email output, paste it back in and say like, analyze this and improve it. And you'll get a better version and do that a couple of times. And essentially, that is the idea of like automated reflection agent.

11:45You're just doing it manually. And then in terms of like creating these abstractions, like given how fast everything is moving all the time, like is it hard to create these types of abstractions? Like how do you choose sort of the right abstraction when things are, I mean, you've been doing this since 2023, but now we're nearing 2025. But in the life of Gen AI, two years is really like 40 years or something like that. But so like things are moving so quickly, like how can you kind of choose the right abstraction and not get into a place where you end up with like having to make like a ton of, you know, breaking changes as you like learn new things and new things come out?

12:20Yeah, great question. And this is a constant struggle for us as I'm sure you as a LinkedIn user experience, as well as lots of the users listening to this podcast. We have gone through a lot of iterations where kind of the 2022 version of Langchain was really about these kind of all-encompassing, opaque chains where you would create and the simplest one was like an LLM chain class, which actually did a lot of magic under the hood. And it was really difficult to debug. And then in 2023, we really focused on the length chain expression language where you would kind of compose these chains as distinct steps.

12:56But a lot of the steps were still a little bit opaque, like you had to know that the JSON output parser would take a string in and output some sort of dictionary, and those kinds of things. And then this year, we've really gone towards LangGraph. And so each of those, even though we still support all the old things, from a user's perspective, every single time that changes, it can feel jarring, right? Because of the kind of front and center quick start is now something that I didn't learn when I first learned LangChain. And I think we've done a pretty good job of kind of announcing those and then still supporting the old models, because obviously, we have a lot of users operating on the LangChain expression language in particular.

13:33But I think the philosophy has really just become more and more bare bones, where everyone who comes to Langchain is either a Python developer or a JavaScript developer. As long as we're talking about the two packages that we maintain, there are some community-driven efforts in Go, Kotlin, there's a few other ones. But for the two main ones, everyone knows those two languages. And so the more just raw Python that we can let people write, the better, because it's things that they already understand and it's kind of no magic included there. And so with line graph, everything is really just a Python function.

14:08And the kind of main abstraction is the same as network X, if you'd used that before, where you're saying like, create my graph, graph.addNode, graph.addNode, and then connect these two nodes to each other with an edge or connect these two nodes with a conditional edge, which does those kinds of things. And sure, there's lots of bells and whistles on the side that you can use for interrupts if you throw a particular kind of error or kind of these checkpointer features where you're storing state or memory. But in order to get started, like seeing some lang graph code makes a lot more sense than seeing some lang chain expression language code if you've never seen it before.

14:46I don't know if you've used it, but it's a lot of kind of pipe operators. It looks a lot more like Bash than Python. And so that has been really the philosophy. That's kind of been the big change in my mind. Okay. And is it hard as well to, and when you're creating these abstractions, how do you think about how different models are going to have different limitations on them? And depending on if I switch the model suddenly from GPT-4 to a different version of GPT or quad or whatever, then the size of the context window could be impacted by that. Maybe other types of features could be affected by that.

15:19Definitely. And as the industry is involved, Actually, the constraints have changed a lot, I would say, where when I first joined Langechain, the main difference between models was the context window, right? You'd use, I'm going to forget the actual numbers, but I think like GPT-3-5-Turbo had like, I think a 4 ,000 token context window to start and maybe it came up to 16 ,000 later. But then a lot of the smaller models were like 1 ,024 tokens and so you could like barely fit in the messages that you wanted to send to them. and they terminated with these really jarring errors where it's like, okay, you exceeded the token window, sometimes in the middle of the output it was generating.

15:59And then you just get like these partial things that weren't that useful. And then nowadays, the main distinction I would call it is probably tool calling, where tool calling is easily the most important feature that Langchain and Langraph users are using out of the models, where it's really useful for providing some sort of tool calling and structuring output, where you provide a schema and the LLM generates all the fields that you kind of ask for, which is a really nice interface point between code and these LLMs. And different models perform very differently. And even with the same model, we can chat a little bit about this in a second, but like the open source LLM Align models for Meta, the tool calling performance is actually markedly different with different providers, depending on how they've implemented kind of parsing of those tool calls, which is kind of another fascinating thing that we're kind of working with right now, where it's kind of difficult to call that out on some of the provider pages in our documentation.

16:57But to answer your original question about how do we kind of manage a lot of that, the first step is really just documentation, right? We add notes to the provider pages where it's like, hey, this model has a checkmark for tool calling or an X for tool calling, or maybe even a warning sign in some cases where it's like, hey, this model says it has tool calling, but it never actually calls tools. and the next step is really building abstractions that make sense around those. And so now in the library, we have these bind tools and with structured output methods that if you just call them, they either work, right?

17:32They give you some sort of structured output or they throw a not implemented error of like, hey, this provider doesn't offer a tool calling and so you can't bind tools too. In some ways, like, and you're probably too young to remember this, but it feels like the early days of the web when it was a non-standardization around HTML and JavaScript and stuff like that. And you'd have to have these control loops essentially or control statements around like, okay, if it's this specific browser of this specific version, essentially this is the way that it needs to behave or this is the call that I can make.

18:02And it's just probably a byproduct of early days, things are moving quickly and everybody's trying to push things out to production and there's going to be essentially a non-standardization across all these things. And then when you bring in the open source models and they're going to be served by different people. There's going to be potentially different interpretations of how to respond to something like tools, for example, and people are going to have their own takes on those. Totally. And actually, if any of the listeners kind of want to dig into the lore here, if you look at some of the source code for the original Anthropic integration, and actually still like AWS Bedrock has one version of their integration that's kind of like this, where everything is still a text completion.

18:40So a lot of that message parsing logic actually happens in the Lankchain integration, which is kind of a crazy world that we lived in once upon a time where there was an if statement of like, in Bedrock in particular, right? It's like, if it's an anthropic model, parse it in this way. If it's a Cohere model, parse the output in this way because the message tokens that are actually outputted are different, which is, yeah, obviously kind of a speed to market type thing that you see across all these different providers. In terms of implementing agents with LangGraph, Can you walk through, what is that process?

19:14I want to get started with LangGraph. I want to build a basic agent. What do I need to do? Yeah, I'll actually start from the very beginning, where we actually have a lot of different media formats to get people started, because we've realized that some people like following video tutorials, some people like following written documentation where you can copy base code, and some people really like starting from a complete template, but they just extend themselves. So for those three, we have the LangChain Academy, which is academy.langchain.com, which is a video format for this. We have the Landgraf documentation, just Google that and there's a quick start.

19:47Or we have Landgraf Studio, which has kind of five templates to get you started, which is actually used in Academy if you end up doing that. So definitely, if you want to dive in further, would recommend going to one of the sources from my coworkers who have made much better content than I can describe right now. But for the kind of brief answer right now, it's all just Python. So you either open up a Jupyter notebook or you open up a text editor and you create a bunch of Python functions representing the different steps you want your graph to take. And funnily enough, the kind of graph interface for this tends to be more intuitive for people who've interacted with no-code type editors, which the whole orchestration of it actually reminds me a lot of LabVIEW or some of those kinds of old connect-a-bunch-of-edges between different nodes-for-robotics type things.

20:36And you define those operations, you connect them up and then you really just run it and see what the output is. One important step I left out is defining that schema. So by default, we recommend just storing like a message history. So the most simple agent actually doesn't even have any tools where it's just accumulating, like I send a message and it appends it to the messages state. And then the LLM sends a message and it appends that to the messages state. And you just go back and forth in an interaction. But then you could also store something of like number of turns of conversation and just increment that at each node.

21:12You could store which tools the LLM should have access to and then modify that over time. And you can really store any of that in there with the caveat that if you store anything that's not serializable, you won't be able to use some of the check pointer kind of hosted features, if that makes sense. And in terms of the workflow orchestration, that's all happening within my environment where I'm essentially hosting my code. Totally. Yeah. So actually, yeah, important distinction. Langraph is kind of mostly an open source project, but then we also have Langraph platform, which is kind of our hosting.

21:47If you've used like Next.js and Vercel, kind of a similar model where Langraph is all the orchestration. It knows how to execute everything. We have some open source versions of check pointers that allow you to kind of serialize that state and kind of fast forward and rewind through some of your execution using essentially database features for that. But then LangGraph platform is really about hosting everything as a REST API and also visualizing it. So we actually have some features in LangSmith, which is our kind of debugging and observability commercial product that lets you visualize your graph, interact with the state manually through these kinds of interrupts and things like that.

22:26and it overall just kind of makes it easier to build some of these things over time. And both of them have a generous free tier, but have to call out that they are not part of the open source offering. For the nodes in the graph, is memory shared across nodes? So the state memory is shared across all the nodes and it's identical across all the nodes. And that's kind of the whole reason for the abstraction. I think if that weren't the case, it would probably make sense to just write everything as raw Python. And to be clear, lots of developers still do that. But then the check pointer kind of global state that is stored across all threads as well.

23:03So you might send a single message, which ends up kicking off a sequence of like six nodes just from that one message before it returns something that's meant to be shown to the user, which is typically a message on the message history. But the execution of all those nodes does not affect like your conversation with it. if I was the one to start that thread, whereas that global state is accessible for both. So there's kind of a few layers of grouping of data. In terms of scaling these, I guess it's essentially on a developer who is going to be hosting this to take on the scale challenges that they might run into with running one of these agent flows, right?

23:47Scale in which regard? Well, in terms of if I build something in line graph and some agentic workflow, and I put it up on a server somewhere and someone starts hitting it, you know, essentially one, it's going to be hitting my hosting infrastructure, but the orchestration and workflow, if I'm using the open source version, that's also hosted within my environment. So I'm assuming it's on me to essentially meet whatever the scale requirements and also architect it for scale by, you know, potentially breaking this up as needed. I'll answer this in kind of two ways. So Landgraf itself is the one that's kind of helping execute those nodes just for a single request.

24:22And so if you have some architecture, if you, for example, use all the synchronous APIs instead of the async APIs, that's going to hog a thread for much longer than if you use something like Python async for that. And there's decisions in terms of how you implement your graph that developers will always have to take responsibility for because that's their graph code. But then once I have that packaged into essentially a fast API endpoint or something like that, that's when you kind of have a decision of whether you want to, or actually right before you have the fast API endpoint, you have a decision of whether you want to build that fast API endpoint yourself and kind of host that just in a Docker container on EC2 or some sort of hosting service.